The challenges of intelligence gathering: Lesson learned from Russia-Ukraine war
Intelligence gathering challenges and obstacles
Data gathering from any platform can overwhelm analysts with a surplus of information, preventing them from providing policymakers with timely situational awareness. The recent events taking place in Ukraine demonstrate the challenges caused by huge amounts of data arriving from different resources and the need to fuse them into a holistic intelligence panorama. On top of that, the amount of “disinformation/misinformation” regarding the events in Ukraine highlights the need to validate the intelligence information while relying on several sources.
While progress has been made in recent years in analyzing and monitoring large amounts of data, more must be done to enable analysts to gather and analyze intelligence effectively. Examples of this include smarter searches than currently possible by using operators; by use of Natural Language Processing (NLP) to find permutations of key words, enhanced use of Optical Character Recognition (OCR), image analysis, and more (see Callaghan, 2021).
Many agencies and bodies suffer from a technological challenge in disseminating and integrating relevant material (often siloed into specific departments or units) into existing intelligence channels and systems. Advanced technological systems, integrated entity-wide, can solve this problem but do so at the expense of security and ease of access. Additionally, analysts may not integrate these emerging technologies because of cultural aversion to change and uncertainty in the use of new methods.
This challenge only intensifies given the fact that today the need to have a competent analyst is not confined only to the security establishment or Law enforcement agencies. The private sector also has a vast use of Analysts as a key component in their ability to conduct due diligence, competitor analysis and to follow and assess a company’s performance.
The Analysts basic challenges and the much-needed response:
Analysts and AI (Human-machine interface)
In order to cope with the amount of information, Analysts must be skilled in gathering, analyzing and working with data. As data grows exponentially, there is an increasing need for a sophisticated data-driven system. AI can assist analysts in streamlining and making sense of this growing intelligence data. Today, researchers face the problem without a real response or solution technologically. They realize that the need for a solution is growing every day given the fact that these issues are only mounting.
Essential technologies for fusion of data
Multiple data types and sources. The analyst will maintain continual awareness of their target operational environment, with a system that will produce high-quality data across the information spectrum, from open-source to highly classified. Connect databases and various file types. Adding Smart links creates relationships between different entities and data sets, thus automatically connecting bank transactions, call logs, social media, vehicle registries, and more, with an emphasis on analyst-friendly interfaces, that will allow them to view their intelligence in compelling manipulable formats and integrate it into creative products for their consumers.
The intelligence cycles. The system/platform should integrate the collection, processing, and analysis stages of the “Intelligence cycle” allowing the analyst to do more in less time and focus on his main assignment – meaning to analyze and to invest less time in collecting (which should be done automatically).
Customizable dashboards and displays. Optimize intelligence flows, automate mundane but vital processing tasks, augment analysts’ sensemaking and critical thinking skills, and even perform certain types of analysis. Thus, predefined and customizable views of investigation status, statistics, and staff efficiency, are paramount in every intelligence fusion system.
Intelligence teams’ collaboration. The required system needs to enable analysts to coordinate and collaborate more effectively from start to finish in the analytic process, from sharing and developing datasets and algorithms to jointly authoring products, by collaboration, defining user permissions, and updating settings as it progresses.
Optimize flows and investigation summaries with swift, secure and actionable insights. AI and analytics tools can help optimize information flows and augment and enrich them to gain more insight from less data in a fraction of the time. AI tools could also be trained to scan, spot, and flag information the analysts designated as critical or anomalous and prioritize it in their queues, providing early indicators and warning of new developments for analysts to alert policymakers. Thus, tailored reports that automatically summarize case details, leads, locations, media, and targets, are essential in every investigation, and will transform this information into actionable insights.
Smart engine search. With AI-tailored queries, analysts should be able to leverage tools to search and sift across vast and various datasets. Analysts could ask smarter questions and search algorithms for a given intelligence question, casting wider and more efficient nets across datasets to piece together critical but often non explicit information (e.g., what is adversary X’s strategy for Y?). Analysts could leverage AI-enabled data mining, sentiment analysis, and geolocation tools to help monitor and predict disruptive events—from mass protests to pandemic outbreaks—by understanding what to ask from the relevant collection units and sources.
AI tools can assist analysts in how they search, organize, and begin making sense of relevant reporting. Technology can help analysts pose optimal questions, search the right datasets, and automate how data is collated and cataloged, thus maintaining a near real-time Detection Anomalies and Incremental Change analysis system.
Command and Control. The relevant system needs to be transparent to commanding officers and supervisors. These individuals must be able to track the progress of all investigations that they supervise. The system must therefore be able to generate simple and understandable reports for senior management. In addition, given that various investigations intersect with different parties, the system must be accessible to other units and agencies while remaining secure. It is important to state that despite all its advantages Artificial intelligence and associated technologies cannot replicate all the complexities of crafting strategic analysis made by a human factor.
Analysts need to return to the driver’s seat
From law enforcement to national security organizations, from banks to private-owned companies, the Analyst is the “center of gravity” for any intelligence process or investigation being conducted.
The Analyst’s role is to gather information from a variety of sources, including raw data material from relevant databases. The analyst examines and validates the accuracy of the data to ensure that it produces meaningful information.
If in the past, it was assumed that the machine can replace the Analysts, nowadays, given the complexity of investigations and the need to handle multiple sources, the ability to “find the needle in a haystack” is dependent more on the Analysts. Furthermore, the analyst’s ability to work with various research data analytics and handle modeling and predictive tools is becoming crucial for every Analyst.
First and foremost, the pace at which data is generated, whether by collection or publicly available information, is increasing exponentially and exceeds our collective ability to understand it or to find the most relevant data with which to make analytic judgments
The Analysts have to deal with the increase in threats with less staff and resources and doing so while operating several technological systems simultaneously. All of this needs to done even when the Analysts training is usually not based on unified tradecraft, and the time invested in this training is becoming shorter every year.
Considering the mounting changes that the “modern” Analyst is coping with, and in order to prepare the analytic community to meet the threats of the 21st Century, it is imperative to improve management of analytic efforts, deepen expertise and enhance the usability of existing information—all of which would improve the quality of intelligence. Specifically, there is a need to implement these steps:
- Leveraging the power of data analytics to provide the contextual understanding of events with strategic implications.
- Given the complexity of investigations, and the need to collaborate with multiple parties, there is a need to build a mechanism that will help Analysts present their views to all the relevant entities, and a common analytic work environment (a shared network, compatible tools, and a common filing system for products and work in progress).
- Reducing the workload of the Analyst with the help of automation processes that will be embedded within the collection steps taken by the Analysts.
- Improving the training for analysts and managers, and providing appropriate performance incentives.
- In addition to these activities, there is the need to invent new methods and strategies to capture, store and manage the raw data.
The best response to the mounting challenges of the Analyst, is to harness the technological and methodological solutions and to create and enhance to role of “Fusion Analyst”. A Fusion Analyst will know all collection sources and have the ability to use the findings from all of these sources and gain insight into products, thus creating a finished intelligence product. Building the relevant intelligence systems that allow this analyst to view the incoming intelligence material from all relevant intelligence sources, and adding the ability to task those relevant sources, will help the Fusion Analyst to “crack the case”. The Fusion Analyst will gather, interpret, and disseminate intelligence data, including geographical and operational information, regarding potential threats.
The article was co-authored with Daniel Cohen, head of the Policy and Tech Program at the Abba Eban Institute