How to successfully create an object-centric, interconnected OSINT asset spanning foundational and current intelligence
Guest blog by Phil Smith, Chief Technology Officer at Janes #DefTechWeek
The story and learning lessons of Janes successful transformation from a legacy publisher into a modern data provider in under three years. Exploring the triumphs and challenges of building an assured platform-agnostic OSINT asset of 50 million highly interconnected intelligence objects that can be utilised within a broad range of low- and high-side systems. To achieve this Janes brought together technology, tradecraft, and data engineering disciplines to deliver a transformational change. And yes, it uses Graphs and AI …
How to successfully create an object-centric, interconnected OSINT asset spanning foundational and current intelligence.
In 2021 Janes started a transformational journey to create a state-of-the-art platform for the management of open-source intelligence, with the goal of creating an interconnected data asset that could be used across a range of low- and high-side systems.
Today, the Janes Single Intelligence Environment is fully operational: used by over 250 Janes analysts daily. It powers our new customer portal and feeds interconnected object-centric OSINT into customer high-side systems and third-party solutions such as ESRI, BAE GXP, and i2. The solution contains millions of intelligence objects and over 180 million relationships connecting them, aiding analysis and insight, and was recently awarded Data Project of the Year at the BCS UK IT Industry Awards.
Delivering the Single Intelligence Environment
To deliver this capability, Janes had to transform both its use of technology and its organisation. The journey was executed in three phases:
Phase 1: Let there be graph – The first step was to implement an RDF Graph to interconnect our existing data assets through an ontological model. This created an intelligence schema and data payload that could be loaded into customer systems via API.
Phase 2 (A): Committing to a core data platform – The ‘heavy lift’ was building a core object-centric data platform that could create, manage, and orchestrate Janes intelligence to deliver a ‘higher-quality’ output. This platform sources external information; processes it leveraging combined analyst/AI augmented workflows; creating analyst-led research, which is structured, categorised, and assembled into OSINT objects within a rich, tightly defined object schema.
Phase 2 (B): Transforming our technical capability – To achieve this we undertook a digital transformation, moving to a disciplined enterprise agile methodology that emphasised hybrid teaming of analyst, data, and technology staff; discovery and learning; and iterative development. Goal-driven rather than scope-driven delivery enabled us to learn through doing, resulting in optimised outcomes at each stage of the development.
Phase 3: A dynamic new portal – Our final phase is to create a new portal that fully leverages the interconnected asset to provide highly efficient analyst discovery, analysis, and presentation capabilities to users. With dynamic object-driven navigation and search, this portal showcases the power of the asset and enables users to find and interact with information in new ways.
Some learning lessons – critical success enablers
- We had to transform ourselves before we could transform our solutions. Specifically, Janes established new ways of working using hybrid teams, leveraging agile development techniques, focusing on an evolutionary learning mindset.
- To enable hybrid data/technology design processes we needed to break down departmental barriers between research, data, and technical teams.
- Focus on building the data management regime – data schema, inter- and intra-object validation rules, data classification etc. – is a challenge and needs to be matured before iterative code development commences.
- Use the right technology for each problem – we believe relational databases are not well suited to OSINT – we use a mix of graph databases, object databases, hierarchical databases, and serverless compute. We use each for the specific capability they are well suited to.
- We try to use proven generic rather than esoteric technologies where we can. We seek to overlay techniques and capabilities to maximise cost/performance outcomes.
- We use a variety of techniques to identify and interconnect data (rather than relying on one), leveraging analyst definition, ML/AI, and graph inference – in that order of importance.
- An evolutionary mindset – build something ‘good enough’ and then iteratively innovate to improve and expand. Our journey is not complete – rather this is our new way of working.
The outcomes
Today, Janes is a business operating within the Single Intelligence Environment, with all analysts using it daily to create research, using a simple, easy-to-use interface, with their research activities focused on their sphere of expertise. The platform then post-processes this high-quality research to automatically categorise, link to a defined object, interconnect, and then package into products for consumption by users. This delivers the following key outcomes:
- A highly leveragable, interconnected data asset that can be used by our customers in a wide variety of intelligence scenarios.
- A highly optimised internal multimedia research tool for use by our analysts that manages quality, lifecycle management, and traceability back to source.
- Out-of-the-box integrations to many leading intelligence platforms.
Janes believes it has managed to deliver something exceptional through disciplined execution in an unpredictable engineering challenge, through self-reflection and transformation, leveraging its strengths in tradecraft, data analysis, and technology. The current platform creates a base for long-term evolution, being both extensible in terms of function and expandable in terms of data coverage.
To find out more about Janes transformation visit www.janes.com
Defence Programme activities
The Defence programme works to help the UK’s defence technology sector align itself with the MOD. techUK members are able to navigate and better understand the UK Defence sector to successfully align their own investment and resources to take advantage of business opportunities. Visit the programme page here.
Upcoming events
Latest news and insights
Learn more and get involved
Defence updates
Sign-up to get the latest updates and opportunities from our Defence programme.
Meet the team
Fred Sugden
Associate Director, Defence and National Security, techUK
Fred is responsible for techUK's activities across the Defence and National Security sectors, working to provide members with access to key stakeholders across the Defence and National Security community. Before taking on the role of Associate Director for Defence and National Security, Fred joined techUK in 2018, working as the Programme Head for Defence at techUK, leading the organisation's engagement with the Ministry of Defence. Before joining techUK, he worked at ADS, the national trade association representing Aerospace, Defence, Security & Space companies in the UK.
Fred is responsible for techUK’s market engagement and policy development activities across the Defence and National Security sectors, working closely with various organisations within the Ministry of Defence, and across the wider National Security and Intelligence community. Fred works closely with many techUK member companies that have an interest in these sectors, and is responsible for the activities of techUK's senior Defence & Security Board. Working closely with techUK's Programme Head for Cyber Security, Fred oversees a broad range of activities for techUK members.
Outside of work, Fred's interests include football (a Watford FC fan) and skiing.
- Email:
- [email protected]
- Phone:
- 07985 234 170
Read lessmore
Raya Tsolova
Senior Programme Manager, techUK
Raya Tsolova is a Programme Manager at techUK.
Prior to joining techUK, Raya worked in Business Development for an expert network firm within the institutional investment space. Before this Raya spent a year in industry working for a tech start-up in London as part of their Growth team which included the formation and development of a 'Let's Talk Tech' podcast and involvement in London Tech Week.
Raya has a degree in Politics and International Relations (Bsc Hons) from the University of Bath where she focused primarily on national security and counter-terrorism policies, centreing research on female-led terrorism and specific approaches to justice there.
Outside of work, Raya's interests include baking, spin classes and true-crime Netflix shows!
- Email:
- [email protected]
- Phone:
- 07712630603
Read lessmore
Jeremy Wimble
Programme Manager, Defence, techUK
Jeremy manages techUK's defence programme, helping the UK's defence technology sector align itself with the Ministry of Defence - including Defence Digital, DE&S, innovation units and Frontline Commands - through a broad range of activities including private briefings and early market engagement events. It also supports the MOD as it procures new digital technologies.
Prior to joining techUK, from 2016-2024 Jeremy was International Security Programme Manager at the Royal United Services Institute (RUSI) coordinating research and impact activities for funders including the FCDO and US Department of Defense, as well as business development and strategy.
Jeremy has a MA in International Relations from the University of Birmingham and a BA (Hons) in Politics & Social Policy from Swansea University.
- Email:
- [email protected]
Read lessmore
Tracy Modha
Team Assistant - Markets, techUK
Tracy supports several areas at techUK, including Cyber Exchange, Cyber Security, Defence, Health and Social Care, Local Public Services, Nations and Regions and National Security.
Authors
Phil Smith
Chief Technology Officer, Janes
