How open data is arranged in Russia
It is no secret that in Russia in recent years, on the one hand, there has been more open data, on the other - a lot of criticism has accumulated.
So it was no coincidence that the subject of open data is what I have been doing for a long time and I know about most of the initiatives in this area not by hearsay, but from my own experience of participation. Here I will try to do without links to my projects, so as not to violate the rules of the site, they will be easy to find by their names anyway. But I will try to give all links to official resources.
Therefore, instead of the preface, I will introduce myself.
')
My name is Ivan Begtin. For several years now I have been promoting the idea of ​​open data in Russia, I am heading a small non-profit organization (Infoculture) on whose behalf and together with our companions we organize competitions and promote openness of data among government agencies.
Among other things, I am a member of the Open Data Council of the Government Commission for the Coordination of the Open Government and a number of public and expert councils under the authorities and aware of many successful (and failed) government initiatives in this area.
Well, since I, including, in the comments on Habré, read a lot of questions asked explicitly or implicitly, I will answer the most frequent in this note, as well as the ones that will be in the comments.
What is happening in Russia with open data?
To answer this question it is necessary to separate two concepts from the very beginning. Open data - as an open form for presenting any database. Whether collected by activists, prepared by commercial companies or published by government bodies. Open data is a comprehensive term encompassing all possible sources of data, the main thing is that the data be available for free reuse and machine-readable (plus following the 8 principles of open data about which we have already written.
But open state data is a kind of open data that is produced from inside the state machine. This may be information, both about the state itself, and what government agencies collect when performing their functions.
Based on these definitions, we can say that what happens in Russia with open data as a whole is 100% dependent on ourselves, how we collect, publish, persuade and so on to publish data, and this is what happens. There are many projects that exist autonomously from the state - this is the GIsLab community, these are our OpenGovData and Hubofdata projects and almost all the crowd projects on MediaWiki can be attributed to open data initiatives because they provide an API for getting data to anyone.
But in terms of open state data , one cannot do without the state itself. In 2013, the Federal Law of the Russian Federation of June 7, 2013 N 112- “On Amendments to the Federal Law“ On Information, Information Technologies and Protection of Information ”and the Federal Law“ On Ensuring Access to Information on the Activities of Government Agencies ” was adopted. and local governments . "
This law contains a set of amendments to the federal law of February 9, 2009 N 8- “On ensuring access to information on the activities of state bodies and local self-government bodies” and to the federal law of July 27, 2006 N 149- “On information, information technology and data protection “that data should be published in open data formats.
Further, at the disposal of the Government of the Russian Federation dated July 10, 2013 N 1187-r, a list of the data that should be published in the first place was presented.
A complete list of laws, orders and resolutions is collected, for example, on the Moscow Open Data Portal in the “ Documents ” section. There are many documents, both federal and those adopted by the Moscow mayor's office.
2. Who in Russia is responsible for open government data?
One could say that the Government is entirely, but this is not quite so. Approval of the openness plans of departments, which include work on open data, is engaged in the Open Data Council. The Open Data Council is the name of the working group under the Government Commission for the Coordination of Open Government Activities.
I also enter this council and I can say that despite the loud name, the Council has practically no authority. Yes, and can not be in the mind of the Russian government structure in which all the tips are exclusively consulting form. In fact, the only resource of the council is the opportunity to convey this or that position to its chairman, Minister Abyzov.
On the other hand, since the law on open data was an amendment to 8-FZ, the Ministry of Economic Development and Trade develops the methodological support for data disclosure and the creation of a federal open data portal.
Here we must make a small digression. In 2012-2013, when the topic of open data in Russia was just acquiring the form of a state initiative, there was a departmental dispute between the Ministry of Communications and the Ministry of Economic Development about which department would oversee this topic. As a result, the Ministry of Economic Development and Trade won then, and in the future it was the main executor of the data openness efforts.
The result of this was a lot of consequences, the main of which was that the emphasis on data disclosure shifted from state information systems as the Ministry of Communications wanted, the official websites of government agencies were always asked about by the Ministry of Economic Development.
3. How much money are all these initiatives?
The topic of open data is almost completely absent in all government programs, including the Information Society program. Over the past few years, only the Government of Moscow, which created the Moscow open data portal and even advertised it on the streets, made substantial investments in making open data accessible.
At the federal level, nothing like this. All members of the open data council work without any compensation for their efforts.
And the costs of creating a federal open data portal are small.
In fact, all the council meetings that were held and attended by representatives of government agencies during this time came down to convincing government agencies to spend efforts to open data. In some cases, it brought results, and in some all the conversations were divided into counter-requests "... and you help us to allocate a budget for XXX million rubles, and we will open everything ...". The most vivid example of such behavior there was Rosstat, a similar situation was with the CEC of Russia and many other departments.
Similarly with all subjects of the federation and municipalities. None of them received money for the introduction of the practice of publishing open data, all efforts were to convince them to spend their own money on these initiatives. Some subjects in this advanced, and some frankly scored and simply sabotaged revealing not the data, but their simulation.
4. Does anyone in Russia use open data?
In short, yes.
Yes, data is used and more than active. They are used by activists and business, used for commercial projects, public and personal, used to improve existing products, and to create new ones.
There are several areas and data sets that are more than required. First of all it is such data as:
- Base of state and municipal procurement;
- Operational data of the Central Bank of the Russian Federation on currencies and open data on banks;
- FIAS base (federal address information system);
- Data on satellite constellation GLONASS
and much more.
There are several dozens of projects for tracking new state orders (Procurement360, Bikotendder), for analyzing contractors (Spark.Interfax, Kontur.Focus, Kommersant.Karoteka) that use data directly. Aggregating data about companies, contracts, purchases, licenses and other. By mixing open data with the ones they get from commercial sources, they create commercial products. As a result, existing commercial products exist and develop on the basis of open data.
On the basis of the FIAS base and before it, on the basis of the KLADR base, a significant number of services operate throughout the country. This base is usually used not to create new products, but to improve the convenience of address verification in numerous commercial information systems.
Data of the Central Bank are used by almost all sites showing official exchange rates and all sites for analyzing banks and the banking system as a whole.
In addition to this data, there are many others - much more highly specialized and which are also actively used, but most often their users do not advertise themselves and are little known.
5. And yet, why are so few examples of those who use data?
The main reason is that the main users of the data are commercial companies that are not at all interested, neither to disclose their income, nor to tell about their business model. And in general, most of them do not want to spend on this time. All this is mixed with their fear that having told about how and how much they earn from this or that data, problems will immediately begin and they will have to pay for the data that they receive for free. The fear is not so unjustified, but also not so unjustified.
For this reason, all those examples that I call are examples of which I know and those examples are in plain sight.
6. Still, there is very little useful data. Everything else is either meaningless data sets, or with a disgusting description
There are really few useful data. And I will say more, the most useful and demanded data were published even before the term “open data” appeared in the form of a state initiative.
The Central Bank of the Russian Federation has been publishing data for over a decade as an API through SOAP services. The FIAS base appeared in the form of open data much earlier. And data on government contracts are published in machine-readable form since 2008.
Everything is very simple. There is the data that state agencies publish in response to a public or corporate request and the data that they publish as a result of a dispute .
Requirements that have been formulated in the law on open data and the disposal of the government for most government agencies - this is another annoying fly. They do not understand why they do it, they do not understand who needs it, they do not understand the benefits of opening data, and they do not understand how they benefit from it. As a result, we get a lot of examples of “bad data” when data is published in sections on government websites without description, without charts, with empty data files or other features. Most often, such data are published by representatives of the press service of the authorities and they do it all on the principle of "formal closure" , an empty publication that allows you to tick the results.
7. Why is everything bad?
I can not say that everything is very bad. Russia has not the last places in data openness ratings like the Open Data Index and the Open Data Barometer. But the reasons to rejoice, really small, and the reasons for the fact that everything goes so hard set.
One of the reasons is that in open data in Russia there are very few clearly expressing their interest of consumers.
The fact is that in the world the main consumers of open data are:
- non-profit organizations and activists (civil society);
- journalists;
- commercial companies;
- universities.
We have our own problems with each consumer in Russia.
There are almost no independent non-profit organizations now. All that lived on foreign grants now, or closed, or rebuilt under state grants, or dramatically curtail their activities. Our non-profit organization exists solely for the reason that all the founders are people engaged in business and we have had and have the opportunity to spend a little money on its support and that we have done the largest projects with the support of the Kudrin Foundation. The only, perhaps, which of the funds in Russia supports public projects on open data. There are virtually no other possibilities. Government grant operators do not allocate funds for such projects, it is very difficult to find commercial sponsors for projects on open data, but foreign grants are now taboo.
Almost all journalism in Russia is very far from a neutral presentation of information. But even the pro-government publications do not interact with government agencies regarding the openness of their data in order to use them in their work. Alas, for all these years, the direction of data journalism has not gained any noticeable outlines in our country and so far journalists have very little impact on the openness of the state in certain issues. Piercing economic analysts, researchers of demography, talkative sociologists - all of them on the pages of the media do not form a request to the authorities for openness.
Commercial companies are simple here - very few of them. Commercial companies using open data can be divided into two categories. The first group is those who live at the expense of government orders and they need the data to show their competencies in order to receive more orders. And the second group living in the information market is extremely small and does not want to interact with state bodies without urgent need. Mainly for reasons of deep distrust of the state machine.
Finally, higher education . Alas, the request from the universities to ensure that the data are available for research, to enable students to use relevant data in their work, this request is simply not there. If in the world universities form their own databases on the areas of research and the work of their faculties, then in Russia, if there is one, then in very limited forms.
8. Will it be better?
Before the introduction of sanctions, the law on foreign agents and other deterioration of foreign and domestic policy, I would say that definitely yes. When it all started, Russia was in the G8, signed the G8 Open Data Chapter, and claimed to participate in the Open Government Partnership. Now the development of the theme of open data occurs with high resistance of the state apparatus. The most popular data are revealed with great difficulty. Detailed educational, criminal and other municipal statistics is extremely difficult to access. Government geodata still open hard. But much more is possible. This is a climatic, transport and other data demanded in practice.
9. Why do I personally participate in all this?
There are many reasons. The main thing is that I got involved open data back in 2009 with anger that this topic is developing all over the world, and there is nothing in Russia. The other is that the device of the “state”, not only the Russian one, is my old hobby. While the greatest efforts came and come not only to convincing officials of open data, but to independently collect data from various sources and turn it into open data.
- I attach a couple of polls to the note and, as promised, I am ready to answer all the questions on the merits in the comments.
So it was no coincidence that the subject of open data is what I have been doing for a long time and I know about most of the initiatives in this area not by hearsay, but from my own experience of participation. Here I will try to do without links to my projects, so as not to violate the rules of the site, they will be easy to find by their names anyway. But I will try to give all links to official resources.
Therefore, instead of the preface, I will introduce myself.
')
My name is Ivan Begtin. For several years now I have been promoting the idea of ​​open data in Russia, I am heading a small non-profit organization (Infoculture) on whose behalf and together with our companions we organize competitions and promote openness of data among government agencies.
Among other things, I am a member of the Open Data Council of the Government Commission for the Coordination of the Open Government and a number of public and expert councils under the authorities and aware of many successful (and failed) government initiatives in this area.
Well, since I, including, in the comments on Habré, read a lot of questions asked explicitly or implicitly, I will answer the most frequent in this note, as well as the ones that will be in the comments.
What is happening in Russia with open data?
To answer this question it is necessary to separate two concepts from the very beginning. Open data - as an open form for presenting any database. Whether collected by activists, prepared by commercial companies or published by government bodies. Open data is a comprehensive term encompassing all possible sources of data, the main thing is that the data be available for free reuse and machine-readable (plus following the 8 principles of open data about which we have already written.
But open state data is a kind of open data that is produced from inside the state machine. This may be information, both about the state itself, and what government agencies collect when performing their functions.
Based on these definitions, we can say that what happens in Russia with open data as a whole is 100% dependent on ourselves, how we collect, publish, persuade and so on to publish data, and this is what happens. There are many projects that exist autonomously from the state - this is the GIsLab community, these are our OpenGovData and Hubofdata projects and almost all the crowd projects on MediaWiki can be attributed to open data initiatives because they provide an API for getting data to anyone.
But in terms of open state data , one cannot do without the state itself. In 2013, the Federal Law of the Russian Federation of June 7, 2013 N 112- “On Amendments to the Federal Law“ On Information, Information Technologies and Protection of Information ”and the Federal Law“ On Ensuring Access to Information on the Activities of Government Agencies ” was adopted. and local governments . "
This law contains a set of amendments to the federal law of February 9, 2009 N 8- “On ensuring access to information on the activities of state bodies and local self-government bodies” and to the federal law of July 27, 2006 N 149- “On information, information technology and data protection “that data should be published in open data formats.
Further, at the disposal of the Government of the Russian Federation dated July 10, 2013 N 1187-r, a list of the data that should be published in the first place was presented.
A complete list of laws, orders and resolutions is collected, for example, on the Moscow Open Data Portal in the “ Documents ” section. There are many documents, both federal and those adopted by the Moscow mayor's office.
2. Who in Russia is responsible for open government data?
One could say that the Government is entirely, but this is not quite so. Approval of the openness plans of departments, which include work on open data, is engaged in the Open Data Council. The Open Data Council is the name of the working group under the Government Commission for the Coordination of Open Government Activities.
I also enter this council and I can say that despite the loud name, the Council has practically no authority. Yes, and can not be in the mind of the Russian government structure in which all the tips are exclusively consulting form. In fact, the only resource of the council is the opportunity to convey this or that position to its chairman, Minister Abyzov.
On the other hand, since the law on open data was an amendment to 8-FZ, the Ministry of Economic Development and Trade develops the methodological support for data disclosure and the creation of a federal open data portal.
Here we must make a small digression. In 2012-2013, when the topic of open data in Russia was just acquiring the form of a state initiative, there was a departmental dispute between the Ministry of Communications and the Ministry of Economic Development about which department would oversee this topic. As a result, the Ministry of Economic Development and Trade won then, and in the future it was the main executor of the data openness efforts.
The result of this was a lot of consequences, the main of which was that the emphasis on data disclosure shifted from state information systems as the Ministry of Communications wanted, the official websites of government agencies were always asked about by the Ministry of Economic Development.
3. How much money are all these initiatives?
The topic of open data is almost completely absent in all government programs, including the Information Society program. Over the past few years, only the Government of Moscow, which created the Moscow open data portal and even advertised it on the streets, made substantial investments in making open data accessible.
At the federal level, nothing like this. All members of the open data council work without any compensation for their efforts.
And the costs of creating a federal open data portal are small.
In fact, all the council meetings that were held and attended by representatives of government agencies during this time came down to convincing government agencies to spend efforts to open data. In some cases, it brought results, and in some all the conversations were divided into counter-requests "... and you help us to allocate a budget for XXX million rubles, and we will open everything ...". The most vivid example of such behavior there was Rosstat, a similar situation was with the CEC of Russia and many other departments.
Similarly with all subjects of the federation and municipalities. None of them received money for the introduction of the practice of publishing open data, all efforts were to convince them to spend their own money on these initiatives. Some subjects in this advanced, and some frankly scored and simply sabotaged revealing not the data, but their simulation.
4. Does anyone in Russia use open data?
In short, yes.
Yes, data is used and more than active. They are used by activists and business, used for commercial projects, public and personal, used to improve existing products, and to create new ones.
There are several areas and data sets that are more than required. First of all it is such data as:
- Base of state and municipal procurement;
- Operational data of the Central Bank of the Russian Federation on currencies and open data on banks;
- FIAS base (federal address information system);
- Data on satellite constellation GLONASS
and much more.
There are several dozens of projects for tracking new state orders (Procurement360, Bikotendder), for analyzing contractors (Spark.Interfax, Kontur.Focus, Kommersant.Karoteka) that use data directly. Aggregating data about companies, contracts, purchases, licenses and other. By mixing open data with the ones they get from commercial sources, they create commercial products. As a result, existing commercial products exist and develop on the basis of open data.
On the basis of the FIAS base and before it, on the basis of the KLADR base, a significant number of services operate throughout the country. This base is usually used not to create new products, but to improve the convenience of address verification in numerous commercial information systems.
Data of the Central Bank are used by almost all sites showing official exchange rates and all sites for analyzing banks and the banking system as a whole.
In addition to this data, there are many others - much more highly specialized and which are also actively used, but most often their users do not advertise themselves and are little known.
5. And yet, why are so few examples of those who use data?
The main reason is that the main users of the data are commercial companies that are not at all interested, neither to disclose their income, nor to tell about their business model. And in general, most of them do not want to spend on this time. All this is mixed with their fear that having told about how and how much they earn from this or that data, problems will immediately begin and they will have to pay for the data that they receive for free. The fear is not so unjustified, but also not so unjustified.
For this reason, all those examples that I call are examples of which I know and those examples are in plain sight.
6. Still, there is very little useful data. Everything else is either meaningless data sets, or with a disgusting description
There are really few useful data. And I will say more, the most useful and demanded data were published even before the term “open data” appeared in the form of a state initiative.
The Central Bank of the Russian Federation has been publishing data for over a decade as an API through SOAP services. The FIAS base appeared in the form of open data much earlier. And data on government contracts are published in machine-readable form since 2008.
Everything is very simple. There is the data that state agencies publish in response to a public or corporate request and the data that they publish as a result of a dispute .
Requirements that have been formulated in the law on open data and the disposal of the government for most government agencies - this is another annoying fly. They do not understand why they do it, they do not understand who needs it, they do not understand the benefits of opening data, and they do not understand how they benefit from it. As a result, we get a lot of examples of “bad data” when data is published in sections on government websites without description, without charts, with empty data files or other features. Most often, such data are published by representatives of the press service of the authorities and they do it all on the principle of "formal closure" , an empty publication that allows you to tick the results.
7. Why is everything bad?
I can not say that everything is very bad. Russia has not the last places in data openness ratings like the Open Data Index and the Open Data Barometer. But the reasons to rejoice, really small, and the reasons for the fact that everything goes so hard set.
One of the reasons is that in open data in Russia there are very few clearly expressing their interest of consumers.
The fact is that in the world the main consumers of open data are:
- non-profit organizations and activists (civil society);
- journalists;
- commercial companies;
- universities.
We have our own problems with each consumer in Russia.
There are almost no independent non-profit organizations now. All that lived on foreign grants now, or closed, or rebuilt under state grants, or dramatically curtail their activities. Our non-profit organization exists solely for the reason that all the founders are people engaged in business and we have had and have the opportunity to spend a little money on its support and that we have done the largest projects with the support of the Kudrin Foundation. The only, perhaps, which of the funds in Russia supports public projects on open data. There are virtually no other possibilities. Government grant operators do not allocate funds for such projects, it is very difficult to find commercial sponsors for projects on open data, but foreign grants are now taboo.
Almost all journalism in Russia is very far from a neutral presentation of information. But even the pro-government publications do not interact with government agencies regarding the openness of their data in order to use them in their work. Alas, for all these years, the direction of data journalism has not gained any noticeable outlines in our country and so far journalists have very little impact on the openness of the state in certain issues. Piercing economic analysts, researchers of demography, talkative sociologists - all of them on the pages of the media do not form a request to the authorities for openness.
Commercial companies are simple here - very few of them. Commercial companies using open data can be divided into two categories. The first group is those who live at the expense of government orders and they need the data to show their competencies in order to receive more orders. And the second group living in the information market is extremely small and does not want to interact with state bodies without urgent need. Mainly for reasons of deep distrust of the state machine.
Finally, higher education . Alas, the request from the universities to ensure that the data are available for research, to enable students to use relevant data in their work, this request is simply not there. If in the world universities form their own databases on the areas of research and the work of their faculties, then in Russia, if there is one, then in very limited forms.
8. Will it be better?
Before the introduction of sanctions, the law on foreign agents and other deterioration of foreign and domestic policy, I would say that definitely yes. When it all started, Russia was in the G8, signed the G8 Open Data Chapter, and claimed to participate in the Open Government Partnership. Now the development of the theme of open data occurs with high resistance of the state apparatus. The most popular data are revealed with great difficulty. Detailed educational, criminal and other municipal statistics is extremely difficult to access. Government geodata still open hard. But much more is possible. This is a climatic, transport and other data demanded in practice.
9. Why do I personally participate in all this?
There are many reasons. The main thing is that I got involved open data back in 2009 with anger that this topic is developing all over the world, and there is nothing in Russia. The other is that the device of the “state”, not only the Russian one, is my old hobby. While the greatest efforts came and come not only to convincing officials of open data, but to independently collect data from various sources and turn it into open data.
- I attach a couple of polls to the note and, as promised, I am ready to answer all the questions on the merits in the comments.
Source: https://habr.com/ru/post/248801/
All Articles