Visibility, accessibility, and relevance through AI: the key to real-time data-driven decision-making (Guest blog by Adarga)
Data has the potential to deliver significant battlefield and organisational advantage to military decision-makers. However, properly realising that potential presents defence and national security organisations with an immense challenge: data will only ever be useful in decision-making if it is visible, accessible, and relevant.
“Unstructured data” – documents, reports, emails, audio, video, web-content, and news feeds – is arguably the most value-rich and untapped source of insight for the defence and national security community. More than doubling in size year on year[i], it comes from two primary sources: internal information (organisational material) and external Publicly Available Information, otherwise known as Open-Source Data (OSD). It’s also worth noting the sub-set of this which is Commercially Available Information (paid for/firewalled content).
‘We don’t know what we know’: the lost in-house knowledge dilemma
Keeping track of in-house information is tough. In the defence and national security sectors, the problem is tougher still given their size, legacy systems, the way in which operations, activities and responsibilities are sub-divided, the intricacies of supply-chains, the lack of consistency in governance and the complicated administrative and reporting structures. This was fully acknowledged in the Ministry of Defence’s (MOD) Data Strategy for Defence in 2021[ii].
As a result, internal knowledge is often deeply buried. It’s poorly signposted, fragmented, diffuse, unshared, siloed across departments and territories, managed by incompatible software, or it may have simply sunk without trace as people move on, or transfer, or simply forget that it was ever there in the first place. Thus, wheels are reinvented, work is duplicated, and critical intelligence can remain unseen. This is inefficient, costly, and potentially places military operations at risk.
In this sense (we repeatedly hear this first hand from our customers) much internal defence information fails the basic “visibility” and “accessibility” tests of data usefulness. This is equally true in global corporations. It can’t be seen, it can’t be found, so it doesn’t exist. Even if it could be seen and found, the “relevance” bar is equally hard to meet: how do we identify the information that is pertinent in a given situation?
Open-Source Data – the challenge and the opportunity
The second source of information is OSD. This information is accessible, but its sheer volume (the world generates such data at a rate of 2.5 quintillion bytes per day[iii]) renders it next to impossible to make visible anything useful, and even less possible to extract what is relevant.
OSD is universally acknowledged to have incalculable value in a national security context. Patrice Tibbs, Chief of Community Open Source at the CIA has observed it has “proven itself over and over again”: 80% of the US Defense Intelligence Agency’s (DIA’s) intelligence now originates from OSD (termed “Open Source Intelligence” or OSINT once analysed)[iv]. There are numerous highly current examples of how OSD is delivering decisive insights and advantage in conflict situations.
The challenge is that there is too much information and too little visibility. A message emphasised in Amy Zegart’s “Spies, Lies, and Algorithms: The History and Future of American Intelligence”. This is the fundamental problem for the analysts, researchers, and planners whose work enables leaders to make critical decisions. They have high level, sophisticated analytical skills, but around 80% of their time is dedicated to the basic job of locating, translating, summarising, and absorbing all the information that is circulating on a particular topic, leaving just 20% of their working week to do the thing they do best – analyse.
They can often only view a tiny and partial body of evidence gleaned from a limited range and volume of materials and sources. In a world in which crucial pieces of the intelligence jigsaw may reside in a foreign language in the least obvious open-source places, or in reconnaissance reports filed seven years ago, this is a major obstacle to achieving accurate insight. As a result – completely unavoidably – accuracy is compromised. This places defence and national security organisations at significant risk of missing vital indicators that could radically affect the decisions they take.
Natural Language Processing (NLP) – the game-changer
The game-changer in creating “Information Advantage” and “Decision Advantage” from vast pools of data lies in the sophisticated teaming of humans and Artificial Intelligence (AI). This is precisely what Adarga is concerned with.
While conventional technologies have been unable to address the unstructured data challenge, developments in Natural Language Processing (NLP) are changing the game. NLP is a branch of AI that focuses on extracting information from unstructured data in a format that machines can process and store. The unique teaming of proprietary NLP models with network science and machine learning, as in Adarga’s Knowledge Platform, is bringing information intelligence derived from OSD and/or in-house sources more readily into the hands of decision-makers.
By leveraging this technology, understanding moves from the fragmented to the dynamic. Vast volumes of unstructured data are processed at high-speed, 24/7, and presented in comprehensible formats. All content is translated for round-the-clock global monitoring. Video is transcribed for immediate analysis. People, places, and organisations are automatically linked to perform intricate network analysis. Global events are extracted, categorised and reported to enhance situational awareness. Complex and distant bodies of information are summarised and clustered into themes. Advanced filtering allows users to quickly compare narratives. The list goes on. Most importantly a “knowledge graph” stores every single link and relationship within the data. A human cannot hold millions of references in their head, but AI can.
Critically, this whole corpus of information becomes searchable by multiple, highly granular parameters. This can then be relayed back to users in the shape of summarised, referenced, audited, and fully sourced reports, and constantly refreshed, fully contextualised, interactive dynamic visualisations of networks and connections.
The benefits of deploying technology to do the heavy lifting of information extraction are immense. A recent application of Adarga’s technology for a defence customer, for example, “achieved in two minutes what would have taken human analysts weeks alone” according to the project lead. In a defence & national security context, this is a truly phenomenal and potentially life-saving gain.
Elsewhere, the European Leadership Network recounted being 20 x faster, with the ability to leverage 300% more sources, and identifying a range of unexpected insights to inform action when it came to reporting on geopolitical risk for government customers. Read more about these results here.
Unlocking information and decision advantage
We have reached an exciting inflection point: defence and national security have recognised the power of machine collation and interrogation of organisational intelligence and OSD; technology, like Adarga’s out-of-the-box platform, can be deployed in days to tackle the data visibility/accessibility/relevance challenge.
Together, we can locate, order, control, and exploit what we don’t know we know (internally) and what we would otherwise never know there is to know (externally). We can now make better decisions – at tactical, operational and strategic levels - by accessing and analysing previously unimaginable volumes of complex information from a previously unimaginable range and number of sources, and at previously unimaginable speeds. This, truly, makes sense of data in the way that decision makers so urgently require - in real time.
[i] IDC
Authors
