The role of big data in private investigations and analysis

In November 2013, New Yorker magazine published an article about a certain Elliot Higgins . But his Twitter follower is known by the nickname Brown Moses . Being at that time an unemployed financier and administrator, Higgins wisely used the wide opportunities provided by a huge shaft of all kinds of information constantly disseminated on the Internet.

Not being a military or representative of any other power structure, possessing exclusively civilian professions, Higgins was baptized by journalists as “probably the best armament expert used during the war in Syria”. An extremely flattering epithet for a man who had nothing to do with weapons or intelligence. How did the unemployed white collar get such recognition from a respected publication?

Open source exploration

Higgins does not speak Arabic and has never been to the Middle East. However, this did not prevent him from mastering the vast amount of information about the parties to the Syrian conflict, and conducting in-depth analysis. And his working tool was just a computer with Internet access. All data Higgins collects sitting at home in Leicester, UK. Moreover, all these online investigations are just a hobby for him. General public Higgins became known in 2012, when he started his blog and published analytical information , according to which the Syrian army used cluster bombs, and also revealed a channel for the delivery of weapons from Iran to Syria. And all this he did only with the help of a laptop and attentive to details.
')


This work can be described as “ open source intelligence ”. Of course, the intelligence services of the whole world have been doing this for centuries, but now it is available to any man in the street. Higgins collects information from sources that are publicly available: photos, videos, posts in social networks. Collecting bit by bit and comparing the data, he painstakingly collects them into the overall picture. As a result, Higgins even began publishing in The Guardian and blogging in The New York Times , and his research is referenced by Human Rights Watch . The combination of talent for blogging and a strong desire to separate facts from fictions brought international recognition to the scout alone .



If we rise to a higher level, then this private success story demonstrates the incredible potential of "intelligence from open sources." We have entered an era when the volume and diversity of information in the network allows anyone to build a true picture of events and interrelations that were previously available only to the special services with their agent network and means of interception and listening. Now, any information entering the network, in the right hands, can become a powerful tool for analyzing secret and non-obvious phenomena.

The role of big data

The emergence of the open source intelligence phenomenon was made possible by a combination of two important factors.

First, the explosive growth of social networks provided access to content that is richly generated by users. If desired, and a certain luck, any inhabitant of the planet can get a huge audience around the world. With the increase in the number of users and the development of social tools, more and more people are “sharing” their thoughts, opinions and observations with the network. This process of generating information has led to the emergence of " digital communities " of enormous size and in unbelievable quantities.



Secondly, the development of big data has completely changed the scale of information available to each person for analysis. The term “big data” is often mistakenly described as “a collection of information that reaches the size of a decent database”. By some estimates, about 1,200 exabytes of information are currently stored on the network, and 90% of this amount was created in the last two years.

Social media is a good example of a big data generator. For example, Twitter users create about 500 million messages per day. Also about 90 million Tumblr posts are published daily. Every minute more than 100 hours of video is uploaded to Youtube. Social media is a real information universe.



This gigantic, most complex digital community has created new opportunities and approaches to analytics. Rushing activity, from daily maintenance of order on network resources to civil wars and the fight against terrorism, forms a new look at the processes taking place in the world. In this ocean of information, finding something necessary is much more difficult than the notorious tool for sewing in dried stems and leaves of grassy plants . But it is still possible.

The same Higgins views many hours of videos uploaded to Youtube, highlighting useful data and exposing them to cross-comparison with information gathered from other sites. But his approach is only one of many possible. He is like a carpenter from ancient times, working exclusively with an ax, in comparison with a team of builders. In the field of big data analysis, which is in the public domain, high-tech methods of analysis work using the latest advances in computer science and computing. And between the “manual” work of Higgins and the automated software and hardware complexes are various methods that combine independent research using free analytical tools .

The report of the research company Demos on the example of Twitter demonstrates the rich possibilities provided by social networks in police investigations. Published messages and a history of status changes can lead to evidence and evidence, as well as become a kind of scale measuring the passions between all parties involved in the incident. And in some cases, constant monitoring of the most active conversations can help in preventing unlawful actions. But here it is already easy to enter the slippery slope of the justification of total surveillance by the state.



It should be noted that automated analysis systems do not belong to the field of “intelligence from open sources” in its usual sense. Rather, it is already from the field of electronic intelligence. Probably for this reason, researchers specializing in the analysis of social media have come up with a separate term for their work: social media intelligence , SOCMINT (intelligence in social media). However, in essence, it is still intelligence from open sources in the sense that all the necessary information is drawn from publicly accessible places.

In fact, the development of technology has breathed new life into this phenomenon. It is possible that soon we will witness the heyday of the small business industry, built on a targeted analysis of publicly available big data.