How to make utilities so that it was GIS
In the country , at the address http://dom.gosuslugi.ru , a single information resource in the housing and utilities sector, the state information system for housing and communal services (GIS utilities), was created and put into commercial operation. The goal and the super task of the system is to finally make the utilities fully transparent and provide the public with ways to strengthen public control in such an important area. The article briefly describes some of the GIS utilities available to all residents of our country today, as well as a review of the main architectural solutions and technologies that made it possible to meet the time requirements and provide the necessary operational characteristics of the system.
A source
I was lucky to work on a project to create a state information system for housing and communal services (hereinafter referred to as GIS housing and communal services) as an architect and head of system development from 2014, that is, from the very beginning. In this post I will try to talk about what it is and how the key decision-making processes looked from inside when developing a large information system. Very large information system.
At once I want to say that this post was conceived as an introductory one, which would allow a bird's-eye view of some aspects of the implementation of the GIS utilities and identify the most interesting topics for further coverage. In subsequent posts, we plan to focus in more detail on specific issues of the implementation of GIS utilities.
')
Over the past 5 years, my life has largely gone online, and this fact allows me to save a large number of forces, freeing up a lot of time. For example, I like to be able to buy a ticket to the theater or to the movies, without getting up from the couch. I like to choose a seat on the plane and book a hotel without leaving home. I like the fact that my bank has no offices, and 99% of all my banking operations I perform via the Internet. I like a couple of button presses in the smartphone to order a taxi and not get caught in traffic (well, almost) thanks to the online navigators. I have a feeling that Internet technologies have finally reached the level at which they began to bring real people real benefit.
A source
In many industries, technology has long been transferred to online, but in the housing and utilities sector, things are not quite so, or rather, not at all. Many residents through the portal of public services can already perform basic operations (such as paying bills or entering meter readings). As a Moscow resident, I, of course, cannot complain, but I would like more and better. For example, I pay part of the services on the website of my management company, electricity — on the Mosenergo website, overhaul fees — on the public services portal. It so happened that in one “wonderful” moment I forgot about Mosenergo, and in my apartment I almost turned off the light. I had to give up everything and urgently go to the Mosenergo office in order to eliminate the resulting debt.
Judging by the reviews of relatives, friends and acquaintances, in many regions things are much worse. My parents live in Astrakhan and resolving issues in the housing and utilities sector also causes them considerable inconvenience. For example, they still spend their time each month to personally transfer meter readings to the management company. Their other headache is that there is no single resource where you could see all the information about invoices and payment facts. Providers of various services periodically lose information about payments, arrears arise, and in order to figure it out, they have to come to the office to a specific service provider along with a pile of payment receipts.
I think that everyone can give their own examples of problems in the housing and utilities sector, which cause trouble and make you spend your time.
Not everyone knows, but our country is moving towards a bright future, when these problems are resolved and we can solve many issues in the housing and utilities sector via the Web or mobile phone, just as it can now be done in the banking sector or other advanced industries.
In accordance with the Federal Law of 21.07.2014 No. 209-FZ and the Housing Code of the Russian Federation in Russia, a state information system of housing and communal services (GIS housing and communal services) is being created. The state customer of GIS utilities is the Ministry of Communications and Mass Media of Russia. The implementation of the system and the creation of a regulatory framework are carried out by the Ministry of Communications and Mass Media of Russia together with the Ministry of Construction of Russia. Financing of the creation of the system is carried out at the expense of the Operator of the system (FSUE “Russian Post”). The development of GIS utilities "from scratch" involved the team LANIT.
Since July 2016, the system has been in industrial operation at http://dom.gosuslugi.ru . The system continues to increase its functionality, and also rapidly filled with data.
In the GIS housing and public utilities throughout the country, data are entered by organizations (management companies - management companies, resource-supplying organizations - RSO, etc.), by houses (multi-family and residential). The figure below shows a map of the implementation of the GIS utilities (see https://dom.gosuslugi.ru/#!/implementation-map - corrected 13.08.2018), where the successful and lagging regions are highlighted.
Map of the introduction of GIS utilities
For example, there is a introduction card for one of the advanced regions - the Murmansk region.
I found a house in Astrakhan, in which I lived a long time ago and was able to view basic information on it, as well as information about related organizations (CC, RNO, local governments).
House and related organizations on the map
I also studied information about the management organization that serves this house. I got acquainted with the license of this organization. It turned out that she has 40 apartment buildings under her management, and over the past year she has “fallen” several times on fines.
Management Information
Every person can already do the same thing on their own. In the following articles we will try to tell about it in more detail and describe what else interesting and useful can be done with the help of GIS utilities.
Each resident of the country on the GIS utilities portal has a personal personal account, which can be accessed through the account of the portal of public services. More than 40 million users have already been registered on state services, and all these people without any additional registration can now enter their personal offices in the GIS utilities. If, by some strange coincidence, one of the users of Habr has not yet been registered on public services, now is the time to spend a little time and do it! I guarantee that the time spent (in fact it takes very little) is more than offset by the resulting "buns".
In the personal account, everyone can see information about all the real estate owned by him. This information is automatically pulled up from the Rosreestr database. Also, through a personal account, you can ask a question or send an appeal to the authorities or to your management company. In addition, in the GIS utilities you can see charges for utilities, the volume of utility services rendered, you can transfer meter readings, you can pay for utility services without unnecessary intermediaries and additional third-party sites. These functions are available now (provided that your Criminal Code has entered the necessary data, see below) and their number is constantly increasing. We will try to announce the most interesting and useful on Habré with a detailed story "how, why and for what."
I recommend to see the story of Deputy Minister of Communications and Mass Media Mikhail Evrayev at CNews forum about the possibilities of the GIS utilities.
All the listed facilities for users are only the tip of the iceberg of the GIS utilities. In order for all this to work, it is necessary that management companies, resource-supplying organizations, local governments, regulatory bodies and all other all participants in long business processes of the housing and utilities infrastructure promptly contribute quality information to the system. And with this we usually have the biggest problems. The duty seems to be as it is, but in good faith it is not always possible for many to fulfill it.
A source
Therefore, for the information providers, the state at the legislative level provides for administrative responsibility, as well as the most understandable and effective economic motivation: starting from January 1, 2018 (originally from January 1, 2017, but then the deadlines were postponed) in the case “if no information has been posted in the system about the amount of payment to be made by the consumer for residential premises and utilities, or information is posted that does not match the payment document submitted to the consumer on paper, the payment Document considered not submitted in accordance with the requirements of the Russian legislation "(see para. 209-FZ of June 21, 2014, and a bill to postpone the timing of liability ). Thus, until an account appears in the personal account of each user of the GIS utilities service, the service will not be considered rendered. Remember that the accounts in the GIS utilities are the only correct!
And now, finally, let's talk about the technical side.
In principle, GIS utilities could be implemented in a dozen different ways. How it ultimately looks is primarily determined by the requirements of the State customer and the Operator. Some requirements directly entail specific consequences, while others allow various variations, which we, as a developer, evaluated based on our experience, capabilities and advantages of our team. Below I will give the basic requirements that were imposed on the GIS utilities at the initial stage of the project.
One of the most important features of the project is the need to implement very large functionality in a very short time in the face of constant changes and refinements. The fact is that the “waterfall” approach, in which the federal law is written for one or two years, the bylaws are adopted, then the software is developed for two or three years, the hardware and software complex is deployed, the system passes the experimental one and then is put into commercial operation to the fact that the system would have gone into industrial capacity in five to seven years. In addition, it is possible that during the time when the regulatory framework would be formed and the system designed for it, the housing and utilities sector has already changed significantly and many things have become irrelevant.
According to the Deputy Minister of Communications and Mass Communications, Mikhail Evrayev, “GIS utilities are created in record time - in 2.5 years ... the work on creating the system went in parallel: the formation of a regulatory legal framework, the development of software, the organization of organizational measures for the implementation of GIS utilities in the regions RF ".
A source
For us as developers of GIS utilities - this means the need to develop and implement software GIS utilities in small iterations, taking into account the accumulated experience, the changes and clarifications.
In practice, unfortunately, for such a large-scale project, as I cheerfully wrote - with iterations and taking into account experience and clarifications, it does not work ... We had to substantially redraw the management scheme, organizational structure, release processes and much more. But this is a whole separate topic, which we will try to discuss another time.
The following key project milestones can be identified:
The project to create GIS utilities started in 2014 in the context of difficult relations with the West. Therefore, one of the most important requirements of the State Customer and the Operator was the complete abandonment of the use of unique equipment and software of Western vendors in favor of using open architecture and open source software.
For our development team, which at that time has the greatest experience of working precisely on the technologies of world vendors, this has become a serious challenge. Serious work has been done on the search and analysis of the entire open source market in order to select the technologies that best fit in terms of functionality.
GIS housing and communal services serves all participants of the housing and utilities market.
The main target parameters include (these are not real numbers, but targets that the system must withstand):
The main users of the system are residents of Russia, but the use of the system is, of course, not limited to the territory of our country. The main activity of users falls on the period from 7:00 to 18:00 local time. In order that each inhabitant of our country could use the GIS housing and communal services at a convenient time for him, the functioning of the system is provided in 24x7 mode.
If any works are impossible without stopping the system, then they are carried out during the period of maintenance work, which are mainly appointed in agreement with the Operator for the period from 21:00 to 00:00 Moscow time.
In general, at the initial stage it is difficult to predict what load will be applied to the system. If you overestimate the potential load, you can choose technologies and architecture that will be unnecessarily difficult to develop and operate. This may entail extremely high labor costs and time to create a system.
Residents of our country, as well as employees of the Criminal Code, the RSO and authorities at all levels have the opportunity to work with GIS utilities through a web browser. In addition, there is a separate mobile client that allows you to perform the basic functions of the system for the residents (it has not yet started production).
In a number of regions, at the time of project initiation, local regional utilities systems were already introduced and used, in which the processes in this region were already automated. The introduction of the GIS housing and communal services should not break the existing regional practice and destroy their achievements, therefore the GIS housing and communal services in such regions should work through regional systems and integration interfaces.
Many resource-supplying organizations and management companies are small organizations without developed IT services, so the GIS utilities had to focus on the work of such organizations through the Web client and data upload / download using XLS files. However, there are large enterprises with advanced automation tools that serve hundreds of homes and thousands of households. For them, GIS utilities provides advanced integration interfaces for interacting with such organizations.
In fact, it turns out that we do not have a main interface - at a minimum, the web interface and the integration interface can be considered equivalent in importance. What does this mean in terms of system architecture? It seems to hint that it would be better to separate the general business logic from the logic of mapping to the web or access via integration services. On the one hand, it will increase the complexity of deploying GIS utilities, but it will minimize the duplication of logic and potential errors.
Despite the fact that in GIS utilities, emphasis is placed on storing data in a structured form and a general movement towards transferring processes into electronic form, for example, concluding contracts, voting, etc., many functions now imply (or allow) loading unstructured processes. or semi-structured documents - scans, text documents in various formats, photos, etc. The volumes of documents uploaded to the GIS housing and public utilities can be tens of terabytes per year.
Storing and making documents available in such volumes is not a trivial task, if you take into account the following aspects:
For the work of GIS housing and communal services, data from various federal resources and departments are needed, as well as interaction with credit organizations.
The main contractors of the system:
GIS Housing and Public Utilities interacts with federal resources and credit organizations through the system of interdepartmental electronic interaction ( SMEV ).
Interaction with the above information bases and systems can easily be called separate large projects that deserve a separate post or even a series of posts.
As a result of the development of many options and alternatives of architecture, we have come to solutions that are best suited for the implementation of GIS utilities.
A source
GIS housing and public utilities is decomposed into several blocks, each of which includes a set of components that are closely related to each other in business and / or perform an independent technical function.
A source
Select the following main blocks, including components related to:
This decomposition allows you to provide a more manageable project structure under tight deadlines, high intensity of changes and has far-reaching consequences, since the organizational chart of the project is also built around it and since it significantly affects the scaling of the system and many other aspects.
It was decided not to use decomposition (sharding) on ​​a territorial basis, for example, to make a set of regional nodes, a federal node, perhaps, to deploy them in different geographically remote data centers, etc.
The division of the system on a territorial basis for GIS utilities recognized as inappropriate, since all the advantages are outweighed by the disadvantages:
Functional subsystems use a single file storage for loading and storing documents, which is a specialized solution that provides efficient storage of large amounts of data.
Functional components include several levels:
The layered structure is used to support different interfaces: user interface through the browser, integration, mobile, and potentially other interfaces.
The following technologies are used to develop functional components.
For services:
For web modules:
For client interface:
For integration interface:
The specified set of technologies is used as standard for most functional components. In general, at the initial stage, we did research on the possibility of using NoSQL technologies to implement functional components, since It is believed that these technologies are sharpened to solve the problem of database scaling (see the small overview report ). However, as a result, we came to the conclusion that this would be unjustified. The decomposition of the system into functional components, capabilities of the equipment and the PostgreSQL relational DBMS allows us to achieve the required level of scalability in a more efficient way and with less risk. Taking into account the statistics of the industrial operation of the system, we can say that this decision was correct.
Of course, other technologies are used for specific tasks. For example, to implement file storage, a NoSQL key-value database is used, since this makes it possible not to use expensive storage systems for storing large volumes of documents, but to use cost-efficient servers with internal disks, as well as to ensure the ability to effectively scale to hundreds of TB of data.
If in the future we are faced with restrictions in terms of scaling PosgtreSQL for any functional component, then we will be able to reconsider its implementation, including using NoSQL database.
It was decided to abandon the Spring Framework (with rare exceptions) for simplification in view of the fact that all necessary services are provided by JavaEE.
The application of GIS utilities software requires a set of servers (virtual or physical) with characteristics expressed in certain parameters: the number of processors, the amount of RAM per processor, IOPS requirements for disk memory, the operating system used, etc. However, in reality These indicators will be achieved (the equipment of which vendor will be used, which virtualization system will be used, etc.) is not so important.
On the other hand, during the operation of the GIS utilities, the Operator faces a difficult question, how to provide the GIS utilities with the necessary equipment so that it, on the one hand, provides the necessary performance and reliability, and on the other hand, that it does not stand idle. The question is complicated by the fact that the amount of technical capacity required for GIS utilities at some point in time depends on many factors: from the time to which, according to the legislation, the necessary data must be entered into the system, from the demand for this or that functions by residents, etc. The value and quantity of these factors is changing all the time.
A source
Therefore, the so-called “cloud” approach is used for GIS utilities: the power of the PAC and how it will be increased are formulated in terms of virtual machines and the unit resources required for them in a period of time. Engineering units that are engaged in the design of the PAC, think through in advance how to “shift” these requirements to specific specifications of equipment.
In more detail about the device "clouds" of the software and hardware complex, we propose to tell in a separate post.
GIS utilities functioning and developing. The efforts of all those who make this system were recently noted at the CNEWS forum - our system was recognized as the best IT project in the nomination “Modern Country” of the CNews AWARDS 2016 Award.
This was a review article. In it I was able to touch superficially only some of the most important issues, but it still turned out to be rather long. In the following articles we can focus more on some specific aspects of the implementation of GIS utilities, which are now left behind. For example, it is an organizational structure and project management scheme, release management solutions, information security solutions, configuration and monitoring issues, ensuring system availability, filling with reference data, decisions on interaction with the Federal Registration Service, credit organizations, decisions on access rights and delegation of rights. and many others.
Write in the comments which topics are more interesting for you about GIS utilities, we will try to tell about them first of all.
A sourceIntroduction
I was lucky to work on a project to create a state information system for housing and communal services (hereinafter referred to as GIS housing and communal services) as an architect and head of system development from 2014, that is, from the very beginning. In this post I will try to talk about what it is and how the key decision-making processes looked from inside when developing a large information system. Very large information system.
At once I want to say that this post was conceived as an introductory one, which would allow a bird's-eye view of some aspects of the implementation of the GIS utilities and identify the most interesting topics for further coverage. In subsequent posts, we plan to focus in more detail on specific issues of the implementation of GIS utilities.
')
Over the past 5 years, my life has largely gone online, and this fact allows me to save a large number of forces, freeing up a lot of time. For example, I like to be able to buy a ticket to the theater or to the movies, without getting up from the couch. I like to choose a seat on the plane and book a hotel without leaving home. I like the fact that my bank has no offices, and 99% of all my banking operations I perform via the Internet. I like a couple of button presses in the smartphone to order a taxi and not get caught in traffic (well, almost) thanks to the online navigators. I have a feeling that Internet technologies have finally reached the level at which they began to bring real people real benefit.
A source
In many industries, technology has long been transferred to online, but in the housing and utilities sector, things are not quite so, or rather, not at all. Many residents through the portal of public services can already perform basic operations (such as paying bills or entering meter readings). As a Moscow resident, I, of course, cannot complain, but I would like more and better. For example, I pay part of the services on the website of my management company, electricity — on the Mosenergo website, overhaul fees — on the public services portal. It so happened that in one “wonderful” moment I forgot about Mosenergo, and in my apartment I almost turned off the light. I had to give up everything and urgently go to the Mosenergo office in order to eliminate the resulting debt.
Judging by the reviews of relatives, friends and acquaintances, in many regions things are much worse. My parents live in Astrakhan and resolving issues in the housing and utilities sector also causes them considerable inconvenience. For example, they still spend their time each month to personally transfer meter readings to the management company. Their other headache is that there is no single resource where you could see all the information about invoices and payment facts. Providers of various services periodically lose information about payments, arrears arise, and in order to figure it out, they have to come to the office to a specific service provider along with a pile of payment receipts.
I think that everyone can give their own examples of problems in the housing and utilities sector, which cause trouble and make you spend your time.
Patience, calm! Now they will appear
Not everyone knows, but our country is moving towards a bright future, when these problems are resolved and we can solve many issues in the housing and utilities sector via the Web or mobile phone, just as it can now be done in the banking sector or other advanced industries.
In accordance with the Federal Law of 21.07.2014 No. 209-FZ and the Housing Code of the Russian Federation in Russia, a state information system of housing and communal services (GIS housing and communal services) is being created. The state customer of GIS utilities is the Ministry of Communications and Mass Media of Russia. The implementation of the system and the creation of a regulatory framework are carried out by the Ministry of Communications and Mass Media of Russia together with the Ministry of Construction of Russia. Financing of the creation of the system is carried out at the expense of the Operator of the system (FSUE “Russian Post”). The development of GIS utilities "from scratch" involved the team LANIT.
Since July 2016, the system has been in industrial operation at http://dom.gosuslugi.ru . The system continues to increase its functionality, and also rapidly filled with data.
What's inside
In the GIS housing and public utilities throughout the country, data are entered by organizations (management companies - management companies, resource-supplying organizations - RSO, etc.), by houses (multi-family and residential). The figure below shows a map of the implementation of the GIS utilities (see https://dom.gosuslugi.ru/#!/implementation-map - corrected 13.08.2018), where the successful and lagging regions are highlighted.
Map of the introduction of GIS utilities
For example, there is a introduction card for one of the advanced regions - the Murmansk region.
I found a house in Astrakhan, in which I lived a long time ago and was able to view basic information on it, as well as information about related organizations (CC, RNO, local governments).
House and related organizations on the map
I also studied information about the management organization that serves this house. I got acquainted with the license of this organization. It turned out that she has 40 apartment buildings under her management, and over the past year she has “fallen” several times on fines.
Management Information
Every person can already do the same thing on their own. In the following articles we will try to tell about it in more detail and describe what else interesting and useful can be done with the help of GIS utilities.
Each resident of the country on the GIS utilities portal has a personal personal account, which can be accessed through the account of the portal of public services. More than 40 million users have already been registered on state services, and all these people without any additional registration can now enter their personal offices in the GIS utilities. If, by some strange coincidence, one of the users of Habr has not yet been registered on public services, now is the time to spend a little time and do it! I guarantee that the time spent (in fact it takes very little) is more than offset by the resulting "buns".
In the personal account, everyone can see information about all the real estate owned by him. This information is automatically pulled up from the Rosreestr database. Also, through a personal account, you can ask a question or send an appeal to the authorities or to your management company. In addition, in the GIS utilities you can see charges for utilities, the volume of utility services rendered, you can transfer meter readings, you can pay for utility services without unnecessary intermediaries and additional third-party sites. These functions are available now (provided that your Criminal Code has entered the necessary data, see below) and their number is constantly increasing. We will try to announce the most interesting and useful on Habré with a detailed story "how, why and for what."
I recommend to see the story of Deputy Minister of Communications and Mass Media Mikhail Evrayev at CNews forum about the possibilities of the GIS utilities.
And you're so cold
All the listed facilities for users are only the tip of the iceberg of the GIS utilities. In order for all this to work, it is necessary that management companies, resource-supplying organizations, local governments, regulatory bodies and all other all participants in long business processes of the housing and utilities infrastructure promptly contribute quality information to the system. And with this we usually have the biggest problems. The duty seems to be as it is, but in good faith it is not always possible for many to fulfill it.
A sourceTherefore, for the information providers, the state at the legislative level provides for administrative responsibility, as well as the most understandable and effective economic motivation: starting from January 1, 2018 (originally from January 1, 2017, but then the deadlines were postponed) in the case “if no information has been posted in the system about the amount of payment to be made by the consumer for residential premises and utilities, or information is posted that does not match the payment document submitted to the consumer on paper, the payment Document considered not submitted in accordance with the requirements of the Russian legislation "(see para. 209-FZ of June 21, 2014, and a bill to postpone the timing of liability ). Thus, until an account appears in the personal account of each user of the GIS utilities service, the service will not be considered rendered. Remember that the accounts in the GIS utilities are the only correct!
And now, finally, let's talk about the technical side.
What do you need
In principle, GIS utilities could be implemented in a dozen different ways. How it ultimately looks is primarily determined by the requirements of the State customer and the Operator. Some requirements directly entail specific consequences, while others allow various variations, which we, as a developer, evaluated based on our experience, capabilities and advantages of our team. Below I will give the basic requirements that were imposed on the GIS utilities at the initial stage of the project.
More functionality in a short time.
One of the most important features of the project is the need to implement very large functionality in a very short time in the face of constant changes and refinements. The fact is that the “waterfall” approach, in which the federal law is written for one or two years, the bylaws are adopted, then the software is developed for two or three years, the hardware and software complex is deployed, the system passes the experimental one and then is put into commercial operation to the fact that the system would have gone into industrial capacity in five to seven years. In addition, it is possible that during the time when the regulatory framework would be formed and the system designed for it, the housing and utilities sector has already changed significantly and many things have become irrelevant.
According to the Deputy Minister of Communications and Mass Communications, Mikhail Evrayev, “GIS utilities are created in record time - in 2.5 years ... the work on creating the system went in parallel: the formation of a regulatory legal framework, the development of software, the organization of organizational measures for the implementation of GIS utilities in the regions RF ".
A source
For us as developers of GIS utilities - this means the need to develop and implement software GIS utilities in small iterations, taking into account the accumulated experience, the changes and clarifications.
In practice, unfortunately, for such a large-scale project, as I cheerfully wrote - with iterations and taking into account experience and clarifications, it does not work ... We had to substantially redraw the management scheme, organizational structure, release processes and much more. But this is a whole separate topic, which we will try to discuss another time.
The following key project milestones can be identified:
- June 21, 2014 - Federal Law N 209- “On the State Information System for Housing and Communal Services” dated June 21, 2014;
- July 2014 - the start of software development;
- February 2015 - the opportunity to post information on 209-FZ from federal executive bodies and local governments was provided;
- May 2015 - commercialization of the register of licenses;
- July 2016 - entering into commercial operation with all the functionality provided for by 209-FZ and by-laws;
- January 1, 2018 - the entry into force of economic motivation, which I have already mentioned above.
The most common hardware and software
The project to create GIS utilities started in 2014 in the context of difficult relations with the West. Therefore, one of the most important requirements of the State Customer and the Operator was the complete abandonment of the use of unique equipment and software of Western vendors in favor of using open architecture and open source software.
For our development team, which at that time has the greatest experience of working precisely on the technologies of world vendors, this has become a serious challenge. Serious work has been done on the search and analysis of the entire open source market in order to select the technologies that best fit in terms of functionality.
Scaling all over Russia, 24x7 work
GIS housing and communal services serves all participants of the housing and utilities market.
The main target parameters include (these are not real numbers, but targets that the system must withstand):
- 50 million households in Russia;
- 2 million apartment buildings;
- 20 million residential buildings;
- 200 thousand managing organizations (UK / TSZH / ZhSK) and resource-supplying organizations (RNO);
- 25 thousand of local governments (CHI);
- 1 thousand authorities of the subjects of the Russian Federation (including the State Housing Inspection and others).
The main users of the system are residents of Russia, but the use of the system is, of course, not limited to the territory of our country. The main activity of users falls on the period from 7:00 to 18:00 local time. In order that each inhabitant of our country could use the GIS housing and communal services at a convenient time for him, the functioning of the system is provided in 24x7 mode.
If any works are impossible without stopping the system, then they are carried out during the period of maintenance work, which are mainly appointed in agreement with the Operator for the period from 21:00 to 00:00 Moscow time.
In general, at the initial stage it is difficult to predict what load will be applied to the system. If you overestimate the potential load, you can choose technologies and architecture that will be unnecessarily difficult to develop and operate. This may entail extremely high labor costs and time to create a system.
Support for different clients to access the system
Residents of our country, as well as employees of the Criminal Code, the RSO and authorities at all levels have the opportunity to work with GIS utilities through a web browser. In addition, there is a separate mobile client that allows you to perform the basic functions of the system for the residents (it has not yet started production).
In a number of regions, at the time of project initiation, local regional utilities systems were already introduced and used, in which the processes in this region were already automated. The introduction of the GIS housing and communal services should not break the existing regional practice and destroy their achievements, therefore the GIS housing and communal services in such regions should work through regional systems and integration interfaces.
Many resource-supplying organizations and management companies are small organizations without developed IT services, so the GIS utilities had to focus on the work of such organizations through the Web client and data upload / download using XLS files. However, there are large enterprises with advanced automation tools that serve hundreds of homes and thousands of households. For them, GIS utilities provides advanced integration interfaces for interacting with such organizations.
In fact, it turns out that we do not have a main interface - at a minimum, the web interface and the integration interface can be considered equivalent in importance. What does this mean in terms of system architecture? It seems to hint that it would be better to separate the general business logic from the logic of mapping to the web or access via integration services. On the one hand, it will increase the complexity of deploying GIS utilities, but it will minimize the duplication of logic and potential errors.
Loading and storing documents
Despite the fact that in GIS utilities, emphasis is placed on storing data in a structured form and a general movement towards transferring processes into electronic form, for example, concluding contracts, voting, etc., many functions now imply (or allow) loading unstructured processes. or semi-structured documents - scans, text documents in various formats, photos, etc. The volumes of documents uploaded to the GIS housing and public utilities can be tens of terabytes per year.
Storing and making documents available in such volumes is not a trivial task, if you take into account the following aspects:
- integrity assurance (documents must be located by links from subsystems, if they have successfully loaded);
- ensuring reliability (disks / nodes can fail, but documents should not be lost);
- according to these documents (where possible) a full-text search should be carried out with the support of the morphology of the Russian language;
- a constant anti-virus scan should be performed;
- the cost of storing such volumes of documents should be minimal.
Interaction with adjacent systems
For the work of GIS housing and communal services, data from various federal resources and departments are needed, as well as interaction with credit organizations.
The main contractors of the system:
- Unified Identification and Authentication System (ESIA) - provides authentication and identification of users,
- IS Rosreestr - the data of the State Real Estate Cadastre to obtain information about real estate objects, the data of the Unified State Register of Rights (USRR), which are needed to provide access to functions and data;
- IS of the Federal Tax Service - data from the Unified State Register of Legal Entities (USRLE) / EGRIP,
- IS of the Prosecutor General's Office - information on inspections of legal entities and individual entrepreneurs,
- IC of credit organizations - information about bills, information about facts of payment, etc.
GIS Housing and Public Utilities interacts with federal resources and credit organizations through the system of interdepartmental electronic interaction ( SMEV ).
Interaction with the above information bases and systems can easily be called separate large projects that deserve a separate post or even a series of posts.
Basic architectural solutions
As a result of the development of many options and alternatives of architecture, we have come to solutions that are best suited for the implementation of GIS utilities.
A source Functional decomposition
GIS housing and public utilities is decomposed into several blocks, each of which includes a set of components that are closely related to each other in business and / or perform an independent technical function.
A source
Select the following main blocks, including components related to:
- the publicly accessible part of GIS utilities - such as the main page, news and events, legislation, a public forum, public analytics and ratings, a single search on publicly available materials, etc.,
- the work of the state housing inspectorate - such as the register of licenses, the register of administrative offenses, the register of information on how to manage apartment buildings and others;
- the work of management companies
- the work of resource supplying companies,
- payments, personal accounts,
- regulatory information, registration of users and organizations, access rights, logging of actions, etc.,
- storing and loading documents (so-called file storage),
- system management and monitoring, etc.
This decomposition allows you to provide a more manageable project structure under tight deadlines, high intensity of changes and has far-reaching consequences, since the organizational chart of the project is also built around it and since it significantly affects the scaling of the system and many other aspects.
It was decided not to use decomposition (sharding) on ​​a territorial basis, for example, to make a set of regional nodes, a federal node, perhaps, to deploy them in different geographically remote data centers, etc.
The division of the system on a territorial basis for GIS utilities recognized as inappropriate, since all the advantages are outweighed by the disadvantages:
- the implemented functionality does not have a strict territorial reference, for example, a management company can manage houses in different regions, just as a resource-supplying company can serve houses without a reference to any kind of rigid territorial division, therefore the implementation of regional nodes will entail complication, duplication and potential problems with integrity;
- the deployment and support of the system is complicated, additional operations are added in case of changes in the territorial division, for example, if we decide to merge regional nodes or, on the contrary, divide them;
- If you use remote data centers, this will entail a significant duplication and less efficient use of infrastructure, as well as the emergence of additional nuances associated with data channels between data centers and security;
- Complicated data analysis and consolidated reporting.
Functional subsystems use a single file storage for loading and storing documents, which is a specialized solution that provides efficient storage of large amounts of data.
Development technologies
Functional components include several levels:
- services that implement basic functional logic and manage data,
- web modules that provide a REST interface for the interface,
- client interface that runs in the client-side browser
- integration interface that provides access to the functionality and data of information systems with which the GIS utilities interact.
The layered structure is used to support different interfaces: user interface through the browser, integration, mobile, and potentially other interfaces.
The following technologies are used to develop functional components.
For services:
- JBoss EAP application server,
- JavaEE Beans per EJB 3.1 specification,
- embedded JPA Provider (Hibernate 4),
- Liquibase for creating and updating the database (see the small overview report ),
- JUnit / Mockito for Unit Testing,
- relational DBMS PostgreSQL,
- SphinxSearch search service using a proprietary control and monitoring console
- JBoss A-MQ messaging management service;
For web modules:
- JBoss EAP application server,
- JavaEE web modules,
- authentication and authorization with Spring Security,
- JAX-RS services (implementation of JBoss RESTEasy);
For client interface:
- client-side rendering / single page application approach using Angular JS,
- Bower and Grunt for assembly;
For integration interface:
- JBoss EAP application server,
- JAX-RS services (implementation of JBoss RESTEasy),
- JAX-WS services (SOAP version 1.1),
- implementation of services based on Spring Integration, Camel, CXF, JAXB,
- processing Excel files with Apache POI,
- JBoss A-MQ messaging service.
The specified set of technologies is used as standard for most functional components. In general, at the initial stage, we did research on the possibility of using NoSQL technologies to implement functional components, since It is believed that these technologies are sharpened to solve the problem of database scaling (see the small overview report ). However, as a result, we came to the conclusion that this would be unjustified. The decomposition of the system into functional components, capabilities of the equipment and the PostgreSQL relational DBMS allows us to achieve the required level of scalability in a more efficient way and with less risk. Taking into account the statistics of the industrial operation of the system, we can say that this decision was correct.
Of course, other technologies are used for specific tasks. For example, to implement file storage, a NoSQL key-value database is used, since this makes it possible not to use expensive storage systems for storing large volumes of documents, but to use cost-efficient servers with internal disks, as well as to ensure the ability to effectively scale to hundreds of TB of data.
If in the future we are faced with restrictions in terms of scaling PosgtreSQL for any functional component, then we will be able to reconsider its implementation, including using NoSQL database.
It was decided to abandon the Spring Framework (with rare exceptions) for simplification in view of the fact that all necessary services are provided by JavaEE.
Hardware and software system
The application of GIS utilities software requires a set of servers (virtual or physical) with characteristics expressed in certain parameters: the number of processors, the amount of RAM per processor, IOPS requirements for disk memory, the operating system used, etc. However, in reality These indicators will be achieved (the equipment of which vendor will be used, which virtualization system will be used, etc.) is not so important.
On the other hand, during the operation of the GIS utilities, the Operator faces a difficult question, how to provide the GIS utilities with the necessary equipment so that it, on the one hand, provides the necessary performance and reliability, and on the other hand, that it does not stand idle. The question is complicated by the fact that the amount of technical capacity required for GIS utilities at some point in time depends on many factors: from the time to which, according to the legislation, the necessary data must be entered into the system, from the demand for this or that functions by residents, etc. The value and quantity of these factors is changing all the time.
A source
Therefore, the so-called “cloud” approach is used for GIS utilities: the power of the PAC and how it will be increased are formulated in terms of virtual machines and the unit resources required for them in a period of time. Engineering units that are engaged in the design of the PAC, think through in advance how to “shift” these requirements to specific specifications of equipment.
In more detail about the device "clouds" of the software and hardware complex, we propose to tell in a separate post.
Conclusion
GIS utilities functioning and developing. The efforts of all those who make this system were recently noted at the CNEWS forum - our system was recognized as the best IT project in the nomination “Modern Country” of the CNews AWARDS 2016 Award.
This was a review article. In it I was able to touch superficially only some of the most important issues, but it still turned out to be rather long. In the following articles we can focus more on some specific aspects of the implementation of GIS utilities, which are now left behind. For example, it is an organizational structure and project management scheme, release management solutions, information security solutions, configuration and monitoring issues, ensuring system availability, filling with reference data, decisions on interaction with the Federal Registration Service, credit organizations, decisions on access rights and delegation of rights. and many others.
Write in the comments which topics are more interesting for you about GIS utilities, we will try to tell about them first of all.
Source: https://habr.com/ru/post/321476/
All Articles