Private sector OSINT, or Open Source Intelligence, refers to the collection, analysis, and dissemination of information from publicly available sources by private organizations or individuals.
Private Sector OSINT in the UK
There are huge opportunities for the private sector OSINT industry, but those opportunities seem to be clouded by a fog of differences in opinion on what OSINT is and what it can offer national security. That fog thickens further when companies attempt to meet national security needs through data science alone, placing the non-trivial task of fusing that insight with traditional intelligence flows at the feet of Government intelligencers. As those intelligencers drown in a sea of products vying for their attention, is it any wonder that vapourware and snake oil slip through the net?
In order to unlock the power of OSINT for national security, the sector must increase its understanding of existing government intelligence disciplines and processes. This will force more coherent private sector language around standards, shift the burden of translation away from Government, and remove the grey areas in which the cheats lurk.
Here be dragons
The opportunities for the sector are plain to see. Government and commercial entities seem to have woken up to the power of OSINT. Through both the idea that so called ‘big data’ can be used to power increasingly effective predictive models and, as General Hockenhull recently put it, that every citizen becoming a sensor gives the world an increasingly accurate view of any given situation.1 As a result, countless start-ups and established names alike are responding to an ever-increasing number of tenders claiming to be able to fulfil a variety of needs, from due diligence to counter disinformation, from AI-powered research to GEOINT.
So, it is fair to say that the demand for OSINT conducted by private sector practitioners using Publicly Available Information (PAI) is remarkably high right now. Yet, the sector itself contains such a diversity of perspectives that it would take a full week’s worth of conference to agree on whether the terms in the previous sentence have been used correctly.2 That incoherence matters.
It matters because the combination of a massive demand signal and a disjointed industry means that the British OSINT market is ripe for plunder by charlatans. While many in the market have genuinely innovative ideas (bringing together skilled investigators, collating and fusing disparate datasets, or coding powerful technology) some are simply repackaging unreliable data and freely available GitHub tools. Even worse, some are using ‘farms’ of workers in low-paying jurisdictions to essentially brute force Google.
Because of the incredibly high number of solutions on offer and the often varied language used to describe them, Government has an inordinately hard time trying to pick the innovators from the cheats. This threatens to kill the opportunity on the vine, as Government ability to use private sector OSINT is undermined by a lack of clarity over its quality and confidence.
The answer may lie (as it so often does) in lessons that we’ve learned before. OSINT designed to complement existing information flows should learn to mirror some of the processes and disciplines that govern those flows. This can be done by bringing those with that expertise into the fold. This will force a common language on standards to emerge in the sector and, by shifting the burden of formatting outputs to meet existing Government standards, make it significantly easier for Government to pick the wheat from the chaff.
Further translation required
The varied language surrounding OSINT standards is compounded by a further technical language barrier. A recent paper from RUSI highlights that the Government’s need for private sector OSINT is driven by an ever-increasing ‘data curve’.3 This is roughly akin to saying that the number of needles continues to increase, but so too does the size of the haystack. Despite Government instinct then, simply offering access to an increasing number of data sets will not meet the need. Government intelligencers need to be able to fuse the insights generated by PAI with their own understanding generated from secret intelligence.4
A common solution to this problem is to throw data scientists at it. The idea is that the role of a data scientist (loosely defined) is to use data to answer complex questions. Therefore, data plus data science should equal insight. That is true, but the type of insight it generates is complementary to intelligence analysis, it is not a replacement for it. Data science is often grounded in statistics and as a result requires predictive models with a much higher level of confidence and therefore much deeper inputs. Traditional intelligence analysis, however, is based on a much broader understanding of factors and is comfortable with far lower levels of confidence, as long as they are clearly expressed and recorded. In short, data science deals well with probability, but traditional intelligence analysis is far better at context and therefore better at predicting anomalies. The marriage of these two disciplines is therefore powerful, but is complicated by a fundamental language barrier; data science often produces models that are opaque to less mathematically-minded intelligencers and intelligencers use confidence judgments that frankly appal data scientists.
At the moment, this divide often sits right on the line between the private and public sectors, placing the burden of translating data science into intelligence squarely on Government analysts.5 For a variety of reasons, not least of which is the sheer variety and scale of solutions on offer, this is a tall order for Government personnel.
The next step therefore is for private sector OSINT companies (whether they are harvesting data, creating tech, or collating and fusing freely available data) to bridge the gap by having both data science and intelligence analysis capabilities in-house, shouldering the burden of translation themselves. This must include a deep understanding of existing Government requirements around intelligence insight production, including source validation, analytical rigour and standardisation of outputs. Because private sector specialists of this type would only have to understand one set of data and/or tooling (the one being presented by their company), the burden of translation is much more manageable.
Output and outlook
The production of finished intelligence products would undoubtedly be the easiest way for OSINT to translate into national security intelligence impact, but that may butt up against cultural barriers. Different forms of OSINT company can, however, employ their intelligencers differently. Those focused on producing tech or user interfaces should use professional intelligencers to create tradecraft that can refine the tech’s target use-cases into those compatible with Government processes. Those who are focused on dataset creation or collation can use their intelligencers to ensure the data’s usability, both in terms of how it is searched and how value is returned. They can also ensure standards around validation and confidence articulation are met, in order to give government users the confidence that they need.
Whatever the use, the focus should be on creating a class of private sector intelligencers who are capable of bridging the gap between the innovation that the private sector brings and the established safeguards that Government intelligence mechanisms require. In so doing, the murky grey areas in which the snake-oil salesmen lurk are removed and those with something truly worthwhile to offer to national security can step forward.
For more information on how you can harness OSINT for the growth benefit of your company, reach out to info@karveinternational.com and one of our expert advisors will be in touch.
Co-authored by Matthew Lawrence.
