FutuReView: the story of one R & D project

In this article, I want to briefly share my experience in developing a project and attract an additional audience to receive feedback.

More than a year ago, together with RIA Novosti and RVC, the FutuReView project, an experimental navigation tool in the described future, was born in the RIA Novosti Media Lab. In essence, this project is the visualization of a library of documents (foresight), describing various aspects of our civilization’s life in the future, about a century ahead. Different technologies, ideas about how cities, life support industries, health, communications, and so on will be arranged - all of this from different companies and research institutes.
')

Data

A special editorial team of RIA Novosti selected about 2 hundred foresight and entered them into a database in a special editorial system, accompanying with multiple metadata, including industry coverage, technological platforms, relevance time in a particular industry, description locality, etc. In addition to the editorial work, we “dissected” textual representations of original documents (mainly pdf and doc articles) through third-party services (such as Alchemy and the like) for keywords and extracted for each foresight a set of entities such as keywords, geolocation, key phrases , that is, some semantic "squeeze" of the original text.

The back-end in our project is built on the NodeJS + MongoDB bundle through our constantly evolving development - the Dataflo.ws framework. As an additional full-text module, we used Sphinx. Based on the server load, the filtered collection of documents is statistically processed.

Presentation concept

In the first approximation, that is, a year ago, we focused on the visual attractiveness of the project. The task was to present a multitude of documents (almost quoting) in a “non-standard non-list way”. The issue of visualization revolved around several ideas:

1. connection graph: metadata are selected according to which all foresight are grouped, links between these groups are found in the form of references to the same document, and a graph with a force field “repulsive nodes, rubber connections” is constructed from these data;
2. timeline: timestamps of each foresight, but there is one thing but - all documents start at about the same time, 2000s;
3. "Multiverse": a cloud of categories with "falling through" in the content.

As a result, we stopped at the "graph of connections", especially since there were already examples of work using d3js . In the first version, in addition to the example, we also used the redesigned fisheye distortion graph modifier (which essentially works as a vertex shader instead of z-coordinates, we used the scale factor), which is also in the examples on d3js . All this looked very attractive and for a few minutes you could hang out to play an animated molecule that is quite lively and physically transformed from one form to another.

Utility

After almost a year of independent project life, we decided to return to it and analyzed the shortcomings of the first version. The problem with the first version was that it did little to do except to attract attention, so we decided to make the project more utilitarian and user-oriented - a researcher and / or entrepreneur who could use our project for a more effective orientation or search. his information in the field of vision of the future. We decided to focus on the list of foresight, additional visual slices, views, as well as the creation / storage of user preferences and filtered collections. In addition, we added the functionality of creating a pdf-poster, which allows you to translate the vision of user interests on a printed basis.

Experience and pitfalls

1. d3js is a powerful svg-based data presentation platform, today no such project, in my opinion, can boast such a variety of ideas and power of possibilities.
2. As an html framework, we used bootstrap 3 , pleased with the customization and management features in JS.
3. I am very sorry that in the early stages I did not have time to master the backbone for the human mvc, because in our project there was a great start for refactoring of self-written controllers and optimization with repeated code fragments - he was lying!
4. Canvas & SVG. Rasterization svg - here we ate a lot of dust. We stumbled upon one eerie feature of working with canvas and svg, namely the impossibility of converting a canvas image with the svg (xml-valid) drawn into it using drawImage into “image / jpeg | png” data. As it turned out, a valid svg for drawing should contain xmlns-attributes in the head tag, the values ​​of which are references to w3c resources. When trying to call canvas.toDataURL (), a security-related ejection is thrown, saying “you are trying to export an image received from another domain”. The canvg project came to the rescue - it traverses the body on svg and neatly calls the canvas methods to draw certain nodes.
5. It is very convenient that today QR codes can be generated directly in the browser via the qrcode.js library

In general, the project for me personally is of great value in terms of gaining experience, value for users - only time will tell. About the server part (the experience of the implementation of mapReduce in a bunch of NodeJS + MongoDB) and the editorial system, a separate conversation, if the topic is interesting, perhaps you can write about it.