The experience of the withdrawal of the program implementation of the social project "Recall everyone" in the open source
Last year, our team was lucky to take part in the development of software for a socially significant project - a system for the input of archive data of the military commissariats of the Khabarovsk Territory during World War II. In short, when he was called, where he went and other related information - to digitize and give the opportunity to search for these data to anyone. The Far Eastern Center for Social Technologies in 2018, with the support of the Presidential Grants Foundation, implemented the project “Recall everyone”. And we, have developed an application on our open source product IONDV. Framework. The final application is now available under the GPLv3 license.
We share our decision and project experience.
Spoiler, some technical details.
')
During the development, we created a web application with the corporate name War archive - for storing, grouping and displaying archive documents about the Great Patriotic War. We actively worked from January to March 2018 and in the process corrected the framework's bugs - as we had the first experience with so many scans.
The result of the project “Recall each” was a data registry with 25k records (now 35k). In April of this year, the Far Eastern Center for Social Technologies published the source code of the application under the GPLv3 license on GitHub . In the repository you will find metadata, design templates and special utilities - everything that forms the basis of the application for IONDV. Framework. The framework itself is distributed under the Apache 2.0 license and is also available on GitHub , along with ready-made modules.
Also deployed a demo application . Test query for the search "Ivanov Ivan." You can see back office by reference . Login - demo, password - ion-demo. By the way, you can get a finished docker image .
The goal of the project “To Remember Everyone” is to preserve the memory of people of the times of the Second World War, providing free access to documents of those years. Maria Stepko, director of the Far Eastern Center for Social Technologies, once said: "Saving and analyzing the data about the past is a task necessary for modeling the future."
The project volunteers digitized 10 thousand pages of documents and entered data into the register of Far Easterners participating in the Great Patriotic War. In the database you can find information about the rank of the conscripted, about the part, date, cause of death and place of burial. In addition, a digitized source of the record is available, indicating the number of the inventory, case, sheet in accordance with the numbering of the state archive. Sample document scan.
We are very grateful for this opportunity and for the feedback about our work:
Yesterday we completed the transfer of the project to Yandex. Cloud and made available at the new address dvarchive.ru .
At the same time we conducted load testing, which we would also like to share.
Server resources are minimal, as it is paid for by the fundraising company: 2 CPUs with 20% priority and 2Gb of memory.
A typical reaction to a small load looks like this - 2% of the CPU load and 36% of the memory load.
We tested user requests without static files, only requests for data and we liked the result.
It turned out that the application in the docker container processes 400 requests per second (without static). And at the same time it loads only one processor, which is typical when only one instance of node.js is started. The second handles the remaining tasks (DBMS, nginx, monitoring).
Clustering is not even needed. For the regime of the usual load of a social project there is a large margin, despite the minimum of resources.
For us, this was the first experience of participating in work on a project related to the memory of the Great Patriotic War, about the Far East.
Moreover, we are proud that this result, developed in the Far East - a region that is not very active in the field of IT, can be used by anyone involved in such projects, especially in the open-source model.
We share our decision and project experience.
Spoiler, some technical details.
')
IONDV. Framework
IONDV. Framework is an open source framework for node.js for creating high-level web applications based on metadata, which does not require serious programming skills.
The basis of the application functionality is the data registry - the Register module. This is a key module designed specifically for working with data based on metadata structures - including project management, programs, events, etc. Also, the project uses a portal module to display arbitrary data templates - it implements the front of the archive files registry.
For the DBMS, MongoDb is used - it stores the application settings, metadata, and the data itself.
The basis of the application functionality is the data registry - the Register module. This is a key module designed specifically for working with data based on metadata structures - including project management, programs, events, etc. Also, the project uses a portal module to display arbitrary data templates - it implements the front of the archive files registry.
For the DBMS, MongoDb is used - it stores the application settings, metadata, and the data itself.
During the development, we created a web application with the corporate name War archive - for storing, grouping and displaying archive documents about the Great Patriotic War. We actively worked from January to March 2018 and in the process corrected the framework's bugs - as we had the first experience with so many scans.
The result of the project “Recall each” was a data registry with 25k records (now 35k). In April of this year, the Far Eastern Center for Social Technologies published the source code of the application under the GPLv3 license on GitHub . In the repository you will find metadata, design templates and special utilities - everything that forms the basis of the application for IONDV. Framework. The framework itself is distributed under the Apache 2.0 license and is also available on GitHub , along with ready-made modules.
Also deployed a demo application . Test query for the search "Ivanov Ivan." You can see back office by reference . Login - demo, password - ion-demo. By the way, you can get a finished docker image .
More about the project
The goal of the project “To Remember Everyone” is to preserve the memory of people of the times of the Second World War, providing free access to documents of those years. Maria Stepko, director of the Far Eastern Center for Social Technologies, once said: "Saving and analyzing the data about the past is a task necessary for modeling the future."
The project volunteers digitized 10 thousand pages of documents and entered data into the register of Far Easterners participating in the Great Patriotic War. In the database you can find information about the rank of the conscripted, about the part, date, cause of death and place of burial. In addition, a digitized source of the record is available, indicating the number of the inventory, case, sheet in accordance with the numbering of the state archive. Sample document scan.
We are very grateful for this opportunity and for the feedback about our work:
“The technical solution created by the Far Eastern development company IONDV goes into free distribution. The system is suitable for NGOs, initiative citizens, museums, archives. It will help save resources and do a good deed for people. Programmers in our region know what “free software” is. Together, we will make this software more functional.
Result of deployment and testing
Yesterday we completed the transfer of the project to Yandex. Cloud and made available at the new address dvarchive.ru .
At the same time we conducted load testing, which we would also like to share.
Server resources are minimal, as it is paid for by the fundraising company: 2 CPUs with 20% priority and 2Gb of memory.
A typical reaction to a small load looks like this - 2% of the CPU load and 36% of the memory load.
We tested user requests without static files, only requests for data and we liked the result.
It turned out that the application in the docker container processes 400 requests per second (without static). And at the same time it loads only one processor, which is typical when only one instance of node.js is started. The second handles the remaining tasks (DBMS, nginx, monitoring).
Clustering is not even needed. For the regime of the usual load of a social project there is a large margin, despite the minimum of resources.
Total
For us, this was the first experience of participating in work on a project related to the memory of the Great Patriotic War, about the Far East.
Moreover, we are proud that this result, developed in the Far East - a region that is not very active in the field of IT, can be used by anyone involved in such projects, especially in the open-source model.
Source: https://habr.com/ru/post/451092/
All Articles