Map of Russian science: resonant or resonant project?

May 21, 2012 Dmitry Livanov is appointed Minister of Education and Science of the Russian Federation. In his first public speech, he voiced the intention of the Ministry of Education and Science (MES RF) to conduct a comprehensive audit of the research and development sector, including the institutes of the Russian Academy of Sciences , state scientific organizations and higher educational institutions. This statement can be called the birth of the "Map of Russian Science".

Unfortunately, behind the events around the RAS reform, this project was somehow lost and, in our opinion, did not receive due attention from the IT community. We offer you a small retrospective: the path of the project from concept to implementation.
')

The aimless path turns blue before me
Far way, streams dug,
And then - darkness; and hidden in this darkness
Soars the fate of the Actor fatal.

Alexander Blok, October 1899

Part 1: Competition

The project "Map of Russian Science" ( http://mapofscience.ru/ ) was officially announced in December 2012. On the eve of the Ministry of Education and Science of the Russian Federation, a competition was held to carry out scientific research work on the theme “Forming a system for evaluating and monitoring the results of research activities of organizations and scientists for regular assessment of the state of the scientific sphere”. The initial (maximum price) of the contract is 100 million rubles. The project was financed under the federal target program Research and Development in Priority Areas of the Scientific and Technological Complex of Russia for 2007-2013 ( Competition for 2012, Activity 2.1, Line 11, Lot 1 ).



The following organizations took part in the competition :

1. Institute of Systems Analysis of the Russian Academy of Sciences;
2. PricewaterhouseCoopers Russia B.V. (hereinafter - PwC);
3. Research Institute of Automatic Equipment them. Academician V.S. Semenikhin;
4. Moscow State University of Instrument Engineering and Computer Science;
5. Moscow State Technical University named after N. E. Bauman;
6. Moscow State University named after MV Lomonosov;
7. INEC-Information Technologies.

Baumanku was not allowed for a formal reason: outdated extract from the YESLF . It was negligence in the preparation of documents, or played by some other factor - it is hardly possible to find out.

The private consulting company PricewaterhouseCoopers Russia B.V. won the competition, offering a contract value of 90 million rubles. and the deadline for its implementation - 90 days.

It should be noted that Moscow State University proposed to develop a "Map of Science" about twice as cheap - for 50 million rubles, but received low points from the competition commission for quality and qualification, taking second place. It looks strange if we consider that there are useful developments in this area at the University: the Nauka-MGU information and analytical system has recently been launched, which is used to collect and analyze employee publication activity.

In the official press, the decision on the choice of PwC as a performer was explained by the desire of the customer in the person of the Ministry of Education and Science of the Russian Federation to conduct an “audit of Russian science” by external organizations that are not related to the scientific environment.

PwC spent 40 million on the purchase of data from the Web of Science (hereafter - WoS) from Thomson Reuters and 15 million on the deployment of technical infrastructure. Plus, the system requires, according to performers, 10-15 million rubles. per year for support.

Unfortunately, we could not find in public access the state contract with the winner of the competition, as well as the technical task. (Attention, question: is this not contrary to the competition law?) I would very much like to look at the amount of work that was announced on paper. Although from a formal point of view, this is no longer so important, since the project is designed as R & D: its result may be just a report, the implementation of at least the prototype is not at all necessary.

Part 2: what did you want to do?

“Our goal is to identify by name those scientists and those small research teams (that is, laboratories, research groups) that already work in Russia at a high international level. We will do this project with one simple goal - to understand where Russia today remains competitive, which areas of science today are promising for us, where we have a chance to make a breakthrough in the future. And, most importantly, support — it’s targeted — precisely those people, those scientists, those laboratories that deserve this support, ”said Dmitry Livanov in his interview to the television channel Enlightenment.



In a document prepared by PwC , the project itself is described by the following thesis: “The Map of Russian Science” should be the basis for making informed management decisions in the field of research activities of scientists and organizations ”; also set specific goals:

1. “Inventory” of the current state of Russian science;
2. quick access on demand to current and correct indicators of Russian science;
3. analytical tools for making sound management decisions;
4. identifying the most reputable experts and research teams for their targeted support;
5. comparing the level of development of science in Russia with other countries and identifying growth points;
6. ensuring transparency of management decisions.

The same document stated the three main pillars of the project: data coverage, data quality and functional .

The data coverage was planned to be provided through:

• international sources: publications, reports at international conferences, patents, publications;
• Russian sources: publications, patents, grants, R & D , publications;
• indicators of both fundamental and applied science.

The result should have been “a unique in coverage base, providing the most complete possible coverage of the results of research activities of Russian scientists.”

By data quality was meant:

• clearing the source data to avoid mismatching proper names;
• use of data adjustment mechanism by scientists and organizations themselves;
• use of unique identifiers of scientists and organizations.

As a result, it was expected to achieve "an unprecedented level of data accuracy, which will help to ensure the visibility of Russian science for the international community."

And, finally, the functional included:

• tools for comparing and identifying growth points in science;
• building reports on the set parameters for scientists, organizations, research areas;
• flexible data search and filtering;
• identification of informal groups of scientists.

This is an "improved set of analytical tools and their flexibility to ensure management tasks."

And now it will probably not be very interesting for non-professionals , but since Habr is a tech blog, we thought it necessary to show the work plan and the architecture of the system being built. Only three slides.

Part 3: what happened?

First of all, we offer Habr readers to evaluate for themselves whether the declared functionality corresponds to what has been implemented. The “Map of Russian Science” is available at this link http://mapofscience.ru/ . Can this be considered a role model? Is this project unique not only in Russia, but also in the world? Try to answer these questions yourself.

Not so long ago, the main page "Maps" has been updated. A red flashing block was added, telling us: “Attention! The system is in trial operation. " This was probably due to the large number of negative reviews about the project. If you look at the application for the contest , you can see that the deadline of this project is the end of 2013. Thus, something fundamentally correct in this version for “trial operation” seems unlikely, and we can proceed to the assessment of the entire project.



The goal of the project (if someone did not remember) was to “identify by name those scientists and those small research teams (that is, laboratories, research groups) that are already working at a high international level in Russia”. In our opinion, it is impossible to do this with the help of the proposed tools.

Next we turn to the pillars of the "Map of Science".

Data coverage

Rather, it did not work out what happened. The two main bases, the Russian Science Citation Index (RISC) and Web of Science (WoS), are in the range of 2007–2012, and also with a reservation for WoS. The data are simply irrelevant (upper plank of the year) and incorrect (lower plank of the year) for the stated primary purpose. And this is despite the fact that access to the WoS database (its parts related to Russian scientists) cost the state 40 million rubles (almost without the right to transfer data to the Ministry of Education and Science).

For the rest of the sources stated in the project, there is also, to put it mildly, some incompleteness. After a long search for leading Russian scientists, they were unable to find their books, monographs, and information about their participation in research and development and grants. It can be assumed that these data are either simply not provided in the science map, or could not be prepared.

Data quality

In our opinion, this task was key in the implementation of the “Map of Russian Science”, it was exactly the technological part , which was the main difficulty and should have taken the main forces and time. Simply put, the task of the whole "Map" was to reduce, clear and correctly link the data. Or, as follows from the transcript of the setup meeting of expert groups on the implementation of the “Map of Russian Science” project, a key block of work was “cleaning and integrating data from various sources”. And, unfortunately, this part did not work at all. The data were not compiled at all: we are offered either RISC or WoS. In fact, we are presented with a simple interface to these two bases, with not very distinct functionality. It so happened that it was precisely to the quality of the data that most of the complaints came from the scientific community. We tried to put them together (but surely they missed something, there are a lot of complaints):

1. the use of a classifier (rubricator) of scientific fields that is not applicable to current areas in Russian science;
2. arbitrary selection (grouping) of scientific institutions by rubrics;
3. lack of control over the level of random coincidences;
4. inconsistency of numerical indicators with real values ​​(by the number of scientific institutions, by the number of publications in WoS and RISC, by the number of patents, by citation index), errors in the transfer of data from operators to another database;
5. incorrect selection of "leading" institutions or researchers (top-5), built on the use of arbitrary features that have no connection with each other (either data on WoS, or RISC, or alphabetical order, or rubricator, etc.);
6. incorrect (incorrect) spelling of full name a researcher in both the Russian and English spelling systems;
7. incorrect (incorrect) affiliation of the researcher;
8. the lack of separation of namesakes and their correct relationship with the scientific direction and scientific institution;
9. lack of information about the departments of organizations (including the faculties of large universities, such as Moscow State University and St. Petersburg State University).

Functional

With the functionality, too, not everything is good. For example, here’s how the data correction mechanism is implemented: “Correction of technical errors noticed by users occurs through the provision of a paper version of the comments, certified by the seal of the organization in which the user works.” Meanwhile, the above transcript says: “The main principle of the project is minimization of efforts on the part of scientists. It is assumed that most of the information in personal accounts will be filled in automatically. ”

No one has yet been found who can clearly explain what information the so-called “heat map” carries. The only interesting feature, in our opinion, is the “minimize the map” link in the lower right, its functionality is at least uncommon and contains the lion’s share of self-irony.



We tried to register in the system to see how it looks from the inside. We were lucky to have an account about a month before the writing of this review, because at the moment the registration of new users for some reason does not work anymore (it seems all polymers have been lost).

To register, the scientist must provide the name, year of birth and email, and then go through the “verification” procedure. This can be done in two ways: by mail or through the so-called SPIN-code.



Verification by mail is carried out through the "manual circuit". To avoid this, we decided to master an innovative SPIN-code. Most likely, not every reader is familiar with this concept (there were those among us too), so we will decipher.

The SPIN code is the personal identification code of the author in SCIENCE INDEX, an information and analytical system based on the data of the Russian Science Citation Index (RISC).

We sent a request for a SPIN-code, filling in a huge form with several dozens of fields and classifiers for this on the RINTS website (in just 20 minutes), and successfully received the code after two weeks of waiting. Pleased with our achievements, we entered the SPIN-code in the scientist's profile, after which the “Map” informed us that this information required verification (not again!). Two weeks have passed since the entry, and the account has not yet been verified.

If you have the patience, then you got to your personal account.


There is nothing special to edit in your personal account, since it contains only the data that you entered during registration. The authors of the system assume that everything else a scientist will tell about himself, filling in a considerable number of fields. Note that in the Western systems (ResearchGate, Academia.edu, Google Scholar), after registration, the user receives an almost ready profile that the system has prepared for him, automatically collecting data from various sources. He can only confirm them and, if necessary, supplement.

It is doubtful that scientists in good faith will use a system in which only the registration takes more than 4 weeks. Obviously one thing - "minimizing the efforts of scientists" did not work.

The official unsatisfactory assessment of the project also corresponds to our conclusions. “ This is a mock-up, it is not even a pilot project ,” noted Lyudmila Ogorodova, Deputy Minister of Education and Science of the Russian Federation (mock-up for 90 million).

Part 4: Scientific Community Reaction

This will be the most concise part of our story. The reaction of the scientific community was sharply negative.

Part 5: the reasons for the failure

As follows from the official position of the Ministry of Education and Science of the Russian Federation and the numerous comments of the expert scientific community, the Science Map turned out to be unsatisfactory. Whether it corresponds to the objectives of the executed state contract, we will not argue in view of the lack of information about it. Another thing is important - how could this situation be avoided? In our opinion, the key point in this story is that all the data on which this public information system was built are not open.

And here we would like to touch on the very topical issue of open data in science. They are simply not there. But they would have been opened, perhaps there was no need for such a state order. The Science Map could be implemented by any professional developer interested in the subject of open data and science. Moreover, there would be several such “cards” with appropriate demand from the state and the scientific community.

Let's look at the list of alleged Russian sources for the "Map of Science":

1. articles in Russian and foreign journals ( NEB );
2. Russian and foreign patents ( FIPS );
3. grants ( FNBI RINKTSE , RFBR , RGNF );
4. reports on research and development ( CITS );
5. dissertations and abstracts (CITS);
6. book publishing (Russian Book Chamber);
7. information about scientific organizations and their divisions (including universities and their faculties).

The overwhelming majority of the above sources were generated from the state budget and it is not clear why these data are not public.

Part 6: How to fix the situation?

As stated E.G. Hat , a member of the Government Commission for the Coordination of Open Government Activities and, concurrently, Managing Director of PricewaterhouseCoopers Russia B.V., “ opening scientific data sets , creating electronic scientific directories, creating specialized scientific networks and services for collaborative work of researchers is global trends. In this regard, the creation of the IC "Maps of Russian Science" is of particular importance for the integration of Russian science into world science. "

Apparently, the current artist is unlikely to be able to refine the “Map” to a more or less working state. But he can, firstly, ensure the completeness of the data in accordance with the state contract (and not for the period 2007–2012). And secondly, to place the results of the work carried out on the “Map of Russian Science” as open data.

It would be a decent way out of this situation, given the fact that the lion’s share of those 90 million rubles. should have been spent working with data. It is necessary to implement an API, according to which anyone can access the results of the project and thereby create a competitive environment for the further development of the "Map".

With an anxious heart in your chest,
CyberLenink project team.