ATOL Online: a look from the data center
We have already talked about how we conceived and implemented the ATOL Online service , designed to work with electronic cash registers. This time we would like to show its internal structure.
ATOL Online is a cloud-based solution, and most of its infrastructure is located in a dedicated data center. The infrastructure of the data center implies a multi-level security, high-quality and stable power supply of equipment from two independent feeders, a reliable fiber-optic connection with telecom operators, air-conditioning and fire-fighting systems.
')
But due to the specifics of the functioning of the service during its deployment, it was decided to add to the standard features provided by the infrastructure of the data center, their own solutions that provide both additional security and convenience.
All the equipment involved in the ATOL Online is located on the territory of the Tier 3 data processing center in a specially designated area that is closed around the perimeter by roller shutters and equipped with an access control system. Only the engineer on duty of the company and, in the event of an emergency at the facility, the data center engineer has the right to enter it.
To exclude unauthorized operations, visitors are registered in the work area and video monitoring of their actions is organized, with the recordings from the camera being kept in the monitoring system.
In the zone specially allocated for the ATOL equipment, several dozens of racks are placed, each of which has 12 assemblies with 48 cash register modules each.
The build logic of the assembly is simplified to the maximum, which allowed minimizing the number of points of failure. Simply put, each cashier is inherently an independent network element that can be replaced "physically" by simply switching two connectors.
In addition, to improve the reliability and resiliency of the system, its network equipment has been duplicated at all levels. For example, assemblies with cash registers are connected to a stack of two switches of the rack level, each of which in turn is attached to the logical stack of core switches. Thus, the malfunction of one of the ports or the interruption of one of the connecting lines at any of the levels will not lead to a fatal accident and a malfunction of the service.
It is important that the equipment of one vendor, Cisco, which is considered the manufacturer of one of the most reliable solutions on the network equipment market, is involved in the ATOL Online infrastructure. With this approach, problems that sometimes arise when building systems on multi-vendor platforms are eliminated.
And finally, Cisco produces part of its equipment in Russia, and ATOL has agreements with the company to reserve production reserves in case of a rapid increase in the number of customers and the need for rapid expansion of the service. This shortens the delivery time of equipment and the time it takes to commission it.
Duplication of nodes and automation is, of course, good, but there are situations when human supervision of equipment is not only desirable, but also necessary.
Therefore, in addition to the usual shift on duty, servicing the equipment of the data center itself, an additional workplace has been deployed on duty by an engineer on duty who is directly responsible for the functioning of the ATOL Online service infrastructure, where experts are on duty around the clock and in shifts.
Workplace duty shift is equipped with two large screens. They display the most diverse information coming from the monitoring system. Here events are tracked on all hardware nodes and software components. In particular, network switches are monitored, starting from devices at the assembly level and ending at the kernel-level switches - that they are available, their ports are in the uplink state and have a calculated load.
Also, in the form of graphs and tables, the general parameters of the service are displayed: the number of successful and erroneous checks per second, the time of penetration of test checks, events from network equipment, servers, software, cash registers, etc.
Given the specifics of the equipment, none of the existing monitoring systems in its basic version could not meet our needs, so we took up deep customization of Zabbix. To monitor the status of cash registers, we wrote our own services that collect data from each cash desk on more than 50 metrics, such as the number of documents sent through the cash register, the state of the communication channel with the CRF, the cash register response time for driver commands, etc. Analyzing the data obtained in this way, we can see, among other things, integration errors, for example, when an incorrect TIN is given in the document or the discount is incorrectly calculated. In this case, the check will not be broken, and the technical support service will inform the client in a timely manner so that he, in turn, corrects this error and continues to work with the cash register.
In addition to the engineers on duty, other employees receive alerts about the events of the monitoring system: technical support service reports about integration problems, information that the archive of the fiscal accumulator will soon overflow, the letter is sent to customer service. All this allows us to improve the quality of service and help customers to deal with possible problems encountered when working with an online cashier.
In case of an engineer’s absence from the working area, alerts from the monitoring system are duplicated on his mobile phone, including a call for the highest priority events.
Depending on the type of failure, various alert schemes are possible. For example, if an electronic check was not processed by the cashier due to its failure, an online store and an engineer responsible for the performance of the ATOL Online service will receive a message about the accident.
Another option is that the client did not appreciate the scale of his business and ordered few cash desks, as a result of which they were overloaded. The system monitors such situations and reports them to the client so that he can change something on his side or order additional cash registers.
It is also possible that the cash register seems to be fiscalized, the client starts to work and generates electronic checks, and the operator of the fiscal data does not receive them.
The reasons may be different: the client submitted incorrect data, or did not do it at all, or the operator has not yet registered the drive, or the fiscal data operator is simply unavailable due to some circumstances.
It also happens that the client forgets to activate the code issued by us in the LC of the CRF, which is why the cash desk in the CRF is considered unpaid and the CRF does not accept checks from it. In this case, they are accumulated in the fiscal drive, and there is a delay in notifying users by SMS or e-mail from the operator of the fiscal data about the passage of the payment and the issuance of an electronic check on it.
When working correctly, all the above procedures are carried out almost in real time. Monitoring allows you to identify problems encountered in sending checks and promptly inform both clients and the operator of fiscal data.
From July 1, 2018, all without exception should finally go to the online ticket office, and we expect a sharp increase in demand for our online service closer to this date. Especially since small businesses are just starting to move. Therefore, we are preparing a "cart" in the winter. However, this is not the main concern. So, for example, we are now actively preparing for the transition to the new expanded protocol of the fiscal data format FFD 1.05, which should come into effect in the beginning of 2019 and is intended to replace the current one. It provides for the addition of new fields to electronic checks that are required to display additional information.
Such a composition of the check should be supported both by the service and the cash registers, as well as by the fiscal data operators. To do this, you will need to modify the service software, the cash register firmware and replace the fiscal drives installed in them.
Some of our clients already need electronic checks with an expanded set of fields, and we are actively cooperating with them.
Also, in the future, in order to improve the disaster resilience of the ATOL Online, it is planned to use two geographically independent data centers. And these are not expansion costs, but direct conditions for working with some large online retailers. We have already prepared the software for such a transition, and as soon as the appropriate site is selected, we will begin to deploy additional infrastructure.
ATOL Online is a cloud-based solution, and most of its infrastructure is located in a dedicated data center. The infrastructure of the data center implies a multi-level security, high-quality and stable power supply of equipment from two independent feeders, a reliable fiber-optic connection with telecom operators, air-conditioning and fire-fighting systems.
')
But due to the specifics of the functioning of the service during its deployment, it was decided to add to the standard features provided by the infrastructure of the data center, their own solutions that provide both additional security and convenience.
Data Center in Data Center
All the equipment involved in the ATOL Online is located on the territory of the Tier 3 data processing center in a specially designated area that is closed around the perimeter by roller shutters and equipped with an access control system. Only the engineer on duty of the company and, in the event of an emergency at the facility, the data center engineer has the right to enter it.
To exclude unauthorized operations, visitors are registered in the work area and video monitoring of their actions is organized, with the recordings from the camera being kept in the monitoring system.
In the zone specially allocated for the ATOL equipment, several dozens of racks are placed, each of which has 12 assemblies with 48 cash register modules each.
The build logic of the assembly is simplified to the maximum, which allowed minimizing the number of points of failure. Simply put, each cashier is inherently an independent network element that can be replaced "physically" by simply switching two connectors.
Double Link Reservation
In addition, to improve the reliability and resiliency of the system, its network equipment has been duplicated at all levels. For example, assemblies with cash registers are connected to a stack of two switches of the rack level, each of which in turn is attached to the logical stack of core switches. Thus, the malfunction of one of the ports or the interruption of one of the connecting lines at any of the levels will not lead to a fatal accident and a malfunction of the service.
It is important that the equipment of one vendor, Cisco, which is considered the manufacturer of one of the most reliable solutions on the network equipment market, is involved in the ATOL Online infrastructure. With this approach, problems that sometimes arise when building systems on multi-vendor platforms are eliminated.
And finally, Cisco produces part of its equipment in Russia, and ATOL has agreements with the company to reserve production reserves in case of a rapid increase in the number of customers and the need for rapid expansion of the service. This shortens the delivery time of equipment and the time it takes to commission it.
Trust but verify: pitfalls
Duplication of nodes and automation is, of course, good, but there are situations when human supervision of equipment is not only desirable, but also necessary.
Therefore, in addition to the usual shift on duty, servicing the equipment of the data center itself, an additional workplace has been deployed on duty by an engineer on duty who is directly responsible for the functioning of the ATOL Online service infrastructure, where experts are on duty around the clock and in shifts.
Workplace duty shift is equipped with two large screens. They display the most diverse information coming from the monitoring system. Here events are tracked on all hardware nodes and software components. In particular, network switches are monitored, starting from devices at the assembly level and ending at the kernel-level switches - that they are available, their ports are in the uplink state and have a calculated load.
Also, in the form of graphs and tables, the general parameters of the service are displayed: the number of successful and erroneous checks per second, the time of penetration of test checks, events from network equipment, servers, software, cash registers, etc.
Given the specifics of the equipment, none of the existing monitoring systems in its basic version could not meet our needs, so we took up deep customization of Zabbix. To monitor the status of cash registers, we wrote our own services that collect data from each cash desk on more than 50 metrics, such as the number of documents sent through the cash register, the state of the communication channel with the CRF, the cash register response time for driver commands, etc. Analyzing the data obtained in this way, we can see, among other things, integration errors, for example, when an incorrect TIN is given in the document or the discount is incorrectly calculated. In this case, the check will not be broken, and the technical support service will inform the client in a timely manner so that he, in turn, corrects this error and continues to work with the cash register.
In addition to the engineers on duty, other employees receive alerts about the events of the monitoring system: technical support service reports about integration problems, information that the archive of the fiscal accumulator will soon overflow, the letter is sent to customer service. All this allows us to improve the quality of service and help customers to deal with possible problems encountered when working with an online cashier.
In case of an engineer’s absence from the working area, alerts from the monitoring system are duplicated on his mobile phone, including a call for the highest priority events.
Depending on the type of failure, various alert schemes are possible. For example, if an electronic check was not processed by the cashier due to its failure, an online store and an engineer responsible for the performance of the ATOL Online service will receive a message about the accident.
Another option is that the client did not appreciate the scale of his business and ordered few cash desks, as a result of which they were overloaded. The system monitors such situations and reports them to the client so that he can change something on his side or order additional cash registers.
It is also possible that the cash register seems to be fiscalized, the client starts to work and generates electronic checks, and the operator of the fiscal data does not receive them.
The reasons may be different: the client submitted incorrect data, or did not do it at all, or the operator has not yet registered the drive, or the fiscal data operator is simply unavailable due to some circumstances.
It also happens that the client forgets to activate the code issued by us in the LC of the CRF, which is why the cash desk in the CRF is considered unpaid and the CRF does not accept checks from it. In this case, they are accumulated in the fiscal drive, and there is a delay in notifying users by SMS or e-mail from the operator of the fiscal data about the passage of the payment and the issuance of an electronic check on it.
When working correctly, all the above procedures are carried out almost in real time. Monitoring allows you to identify problems encountered in sending checks and promptly inform both clients and the operator of fiscal data.
What is the benefit of online service?
A little help under the spoiler
Here are the main points on which the online service is more convenient and profitable than independent work with cash registers and fiscal drives.
Cashier is a very special service element that requires the most careful attention and constant monitoring, even if it is 100% operational.
This is due both to the features of the device itself and to the specifics of its service.
The online cashier module was specially developed by us for this service and is a regular cashier, but without a printing device.
It installs a fiscal drive that is owned by the client.
The fiscal drive has two parameters that limit its lifetime. This is its capacity - 170 or 250 thousand electronic checks, depending on the version of the drive, and a period of 13 months, after which, in accordance with regulatory acts, it must in any case be replaced with a new one.
The last operation is not as simple as it may seem from the side, and the replacement of the FN is always a simple cash register, since it is required to exclude it from the service, after which it is necessary to re-register and re-install the equipment for tax accounting and with the operator of the fiscal data. All this may take up to two days of working time.
To minimize delays on our side, we have provided a reserve of fiscal drives directly to the data center, ready to connect to this customer, which, in case of replacement, receives new data for registration with the tax authorities.
But if with a fixed term of 13 months everything is more or less clear, then with the capacity of the drives is not so simple. A small online store can be enough and one cashier, and a large company with a large turnover will score with electronic checks similar to FN in one day. Therefore, when connecting the service, we offer the client to use a special calculator, which allows, based on the load data, to calculate the amount of the minimum and optimal equipment for it.
Of course, the calculator gives only an approximate calculation and does not take into account, for example, the option when the client's business begins to grow rapidly and generate a more intensive flow of electronic checks than previously thought. However, our monitoring system monitors trends and implements a predictive forecast of the filling rate of all fiscal drives, taking into account not only their capacity, but also the level of load on the cash desk at the client.
A warning of possible overflow of the FN and the need to replace it is issued 60 days before the predicted date. After that, negotiations with the client begin regarding the acquisition of a new drive by it. In the case of a positive decision, the engineer, not allowing the equipment to be blocked, establishes an additional FN, and the client sends data on its fiscalization to provide them to the tax service.
Such cooperation relieves our clients from other problems. First, they can not store the used FN independently, but leave it under our responsibility in a protected cell in the data center.
Secondly, the user is not required to independently search for and purchase drives, which, moreover, are more expensive when purchased on the open market. New FNs will cost less, and the client will not have to visit us personally to replace them.
Thirdly, thanks to our advanced integration with some CRFs, registration of cash desks in their structure occurs almost automatically, due to which the client eliminates the need for additional actions.
In addition, the operators of fiscal data, with whom we have partnership agreements, the cost of the contract is included in the cost of renting the cash register, which allows you to carry out all operations in one window and is much cheaper than doing everything yourself.
Thus, users do not need to deploy the appropriate infrastructure themselves, purchase cash registers with a drive and integrate them into their system. At the same time, questions on expanding the set of cash desks are removed, as well as the need for monitoring the performance and filling of the FN.
Cashier is a very special service element that requires the most careful attention and constant monitoring, even if it is 100% operational.
This is due both to the features of the device itself and to the specifics of its service.
The online cashier module was specially developed by us for this service and is a regular cashier, but without a printing device.
It installs a fiscal drive that is owned by the client.
The fiscal drive has two parameters that limit its lifetime. This is its capacity - 170 or 250 thousand electronic checks, depending on the version of the drive, and a period of 13 months, after which, in accordance with regulatory acts, it must in any case be replaced with a new one.
The last operation is not as simple as it may seem from the side, and the replacement of the FN is always a simple cash register, since it is required to exclude it from the service, after which it is necessary to re-register and re-install the equipment for tax accounting and with the operator of the fiscal data. All this may take up to two days of working time.
To minimize delays on our side, we have provided a reserve of fiscal drives directly to the data center, ready to connect to this customer, which, in case of replacement, receives new data for registration with the tax authorities.
But if with a fixed term of 13 months everything is more or less clear, then with the capacity of the drives is not so simple. A small online store can be enough and one cashier, and a large company with a large turnover will score with electronic checks similar to FN in one day. Therefore, when connecting the service, we offer the client to use a special calculator, which allows, based on the load data, to calculate the amount of the minimum and optimal equipment for it.
Of course, the calculator gives only an approximate calculation and does not take into account, for example, the option when the client's business begins to grow rapidly and generate a more intensive flow of electronic checks than previously thought. However, our monitoring system monitors trends and implements a predictive forecast of the filling rate of all fiscal drives, taking into account not only their capacity, but also the level of load on the cash desk at the client.
A warning of possible overflow of the FN and the need to replace it is issued 60 days before the predicted date. After that, negotiations with the client begin regarding the acquisition of a new drive by it. In the case of a positive decision, the engineer, not allowing the equipment to be blocked, establishes an additional FN, and the client sends data on its fiscalization to provide them to the tax service.
Such cooperation relieves our clients from other problems. First, they can not store the used FN independently, but leave it under our responsibility in a protected cell in the data center.
Secondly, the user is not required to independently search for and purchase drives, which, moreover, are more expensive when purchased on the open market. New FNs will cost less, and the client will not have to visit us personally to replace them.
Thirdly, thanks to our advanced integration with some CRFs, registration of cash desks in their structure occurs almost automatically, due to which the client eliminates the need for additional actions.
In addition, the operators of fiscal data, with whom we have partnership agreements, the cost of the contract is included in the cost of renting the cash register, which allows you to carry out all operations in one window and is much cheaper than doing everything yourself.
Thus, users do not need to deploy the appropriate infrastructure themselves, purchase cash registers with a drive and integrate them into their system. At the same time, questions on expanding the set of cash desks are removed, as well as the need for monitoring the performance and filling of the FN.
It will be hot in summer
From July 1, 2018, all without exception should finally go to the online ticket office, and we expect a sharp increase in demand for our online service closer to this date. Especially since small businesses are just starting to move. Therefore, we are preparing a "cart" in the winter. However, this is not the main concern. So, for example, we are now actively preparing for the transition to the new expanded protocol of the fiscal data format FFD 1.05, which should come into effect in the beginning of 2019 and is intended to replace the current one. It provides for the addition of new fields to electronic checks that are required to display additional information.
Such a composition of the check should be supported both by the service and the cash registers, as well as by the fiscal data operators. To do this, you will need to modify the service software, the cash register firmware and replace the fiscal drives installed in them.
Some of our clients already need electronic checks with an expanded set of fields, and we are actively cooperating with them.
Also, in the future, in order to improve the disaster resilience of the ATOL Online, it is planned to use two geographically independent data centers. And these are not expansion costs, but direct conditions for working with some large online retailers. We have already prepared the software for such a transition, and as soon as the appropriate site is selected, we will begin to deploy additional infrastructure.
Source: https://habr.com/ru/post/344700/
All Articles