Traffic Inspector: full reboot
Hi, Habr! Almost 12 years ago, we, the company Smart-Soft , released the first version of its flagship product, Traffic Inspector . During this time, it has evolved from a regular proxy server to a universal security gateway, combining proxy, email, antispam, web servers with Phishing Blocker service modules, RBL, RAS Dialer, NetPolice content filtering plug-in, advanced CCC system certification and FSTEC, and the possibility of accounting for different types of traffic (IMAP, SMTP, POP, NAT and proxy).
As with any serious product, over the course of evolution, Traffic Inspector was flawed. Today we summarize the work on the mistakes that has been carried out by specialists of our company over the past couple of years.
The central component of Traffic Inspector is a network driver responsible for the direct removal of traffic data and its transfer to the program service. The driver code is executed in kernel mode, and any errors are fraught with "falling into the blue screen", a forced reboot and loss of all unsaved data. For customers, it is important that a program of this class (Internet gateway) can be operated on the principle of “set-set-up-forget”. The code and error messages were analyzed, the problem of low driver stability was successfully solved in early 2014 - “blue screens” are now a rare phenomenon.
')
The rest of the screenshots of the application
Alexander Yamashkin, head of development department, about driver work:
It is known that the lower the level of the reference model (OSI) on which protection is implemented, the more transparent it is for applications. However, by reducing this level, the set of available mechanisms decreases and the “cost of error” is higher. For a guaranteed analysis of all packets passing through the network interfaces, NDIS was chosen for the link level, but the cost of the error was a BSOD. We spent a lot of time analyzing errors in order to say with a large degree of confidence that at the moment driver problems are possible only in the event of a conflict with third-party drivers.
64-bit processors appeared in 2003. At the same time, Traffic Inspector remained a 32-bit in the Smart-Soft for a long ten years, which was often the cause of dissatisfaction among the system administrators. 32-bit platforms and software have a number of significant limitations: support for no more than 4 TB of hard disk space, support for no more than 3 GB of RAM and no more than 2 GB for the user space of the application. Traffic Inspector proxy server (implemented as part of the service) is designed for active use of available memory. The 32-bit version of the service often quickly “consumed” limited memory resources, rested against the memory ceiling, the program could not open more than 4096 threads in the system, the system hung under load and the proxy started to slow down a lot. The most important improvement of the program is the release of the 64-bit version of the service, released at the end of 2013. In the new version there are both x86 and x64 versions and the user himself determines which one to install. You can forget about past memory problems, and now the proxy can easily handle up to 1000 simultaneous HTTP requests.
Alexander Yamashkin on the implementation of work on 64-bit platforms:
Now there are both 32-bit and 64-bit versions of the program, and the user himself determines which one to install (at the moment 70-80% of installations fall on the 64-bit one). Using the latter removes the limitations imposed by the operating system on the amount of memory in 2 GB, which can be allocated to the process, and therefore there are no errors that occur when it is impossible to allocate the next memory block to the process.
The Traffic Inspector web portal provides the generation of network operation reports and other aspects of network statistics. Many customers acquired Traffic Inspector for this very purpose - statistical data collection and reporting. One could understand their disappointment when the process of forming even a simple report could drag on for a long time. This problem was overcome by translating the web portal to PHP and related technologies.
Alexander Yamashkin about the translation of the portal in PHP:
Prior to version 3.0.1, the portal was written in C # c using the .Net Framework. In addition to changing the interpreter, a report preparation service was implemented, into which all the reporting logic was moved. The portal now has only one function - the GUI.
In addition to speeding up the work of the portal, database distribution and data storage for a long period of time were introduced:
If you need to store data for a large interval, we recommend the use of third-party DBMS. But even with them from time to time there were problems. Imagine the complexity of servicing a 1.5 TB database in MS SQL, while 90% of this data is not used. Therefore, in 3.0.2, the database dividing mechanism was implemented, when, at the end of the period (month / quarter / half year / year - specified by the administrator), the database, including SQLite, is disconnected from operational work and is used only when requesting reports for the specified period.
Many customers who first encountered the Traffic Inspector program often talked about the great complexity in its development and the incomprehensible interface. Recently, a number of efforts have been made to improve the usability of the program.
Significantly changed the interface Traffic Inspector. Now use the various elements of the program has become easier and more convenient. In the new structure of the console tree, the elements are more logical and convenient. The content of the pages for the main nodes of the console tree has been completely redesigned (Objects, Traffic Accounting, Users and Groups, Rules, Services, Settings). As a result, the administrator of the Traffic Inspector program receives simple and convenient control over all the program's features, comprehensive information about its settings and links for solving basic administrative tasks. The new system of adding users and a number of wizards make the initial setup of the program as easy as possible.
Alexander Yamashkin about the new user addition system:
The system, fixing access to the Internet from a user not registered in the program, can either automatically create an account based on the specified parameters, or create a request for the administrator, who can only confirm the creation of a new account and assign the necessary access rules.
The latest changes in the program will be commented by the head of the development department Alexander Yamashkin:
The main improvements were related to the reports and the new registration system, the remaining ones were rather specific. For example:
This will tell Alexander Yamashkin and Vaidas Damoshevichyus, CTO and partner of the company Smart-Soft.
Alexander Yamashkin:
I really want to make the process of creating rules for access to information resources as simple and understandable as a sophisticated and novice administrator.
Vaidas Damoshevichus:
Now we are actively exploring technological solutions for classifications of packages at the seventh level (Layer 7) - we are looking for what we could integrate into the Traffic Inspector. After successful integration to our users, we will present a new functionality. When it will be? Then, when we ourselves will be satisfied with the results of integration, since we set high quality indicators for ourselves. We also did not forget about the reports - we want to improve them and make not just reports, but a set of analytical tools that will help system administrators to monitor and monitor network anomalies.
The company Smart-Soft took into account the shortcomings of its software and carried out work to eliminate. The program moved to the 64-bit platform, significantly increasing the performance of individual components. Numerous corrections and additions touched most aspects of the program. The new Traffic Inspector is a multifunctional Internet gateway of a new generation with modern functionality, but at the same time an interface that every user can understand.
PS: If there is something that we have not said, suggestions for correction, we will be glad to hear your comments.
As with any serious product, over the course of evolution, Traffic Inspector was flawed. Today we summarize the work on the mistakes that has been carried out by specialists of our company over the past couple of years.
Low stability Traffic Inspector
The central component of Traffic Inspector is a network driver responsible for the direct removal of traffic data and its transfer to the program service. The driver code is executed in kernel mode, and any errors are fraught with "falling into the blue screen", a forced reboot and loss of all unsaved data. For customers, it is important that a program of this class (Internet gateway) can be operated on the principle of “set-set-up-forget”. The code and error messages were analyzed, the problem of low driver stability was successfully solved in early 2014 - “blue screens” are now a rare phenomenon.
')
The rest of the screenshots of the application
Alexander Yamashkin, head of development department, about driver work:
It is known that the lower the level of the reference model (OSI) on which protection is implemented, the more transparent it is for applications. However, by reducing this level, the set of available mechanisms decreases and the “cost of error” is higher. For a guaranteed analysis of all packets passing through the network interfaces, NDIS was chosen for the link level, but the cost of the error was a BSOD. We spent a lot of time analyzing errors in order to say with a large degree of confidence that at the moment driver problems are possible only in the event of a conflict with third-party drivers.Poor Traffic Inspector Performance
64-bit processors appeared in 2003. At the same time, Traffic Inspector remained a 32-bit in the Smart-Soft for a long ten years, which was often the cause of dissatisfaction among the system administrators. 32-bit platforms and software have a number of significant limitations: support for no more than 4 TB of hard disk space, support for no more than 3 GB of RAM and no more than 2 GB for the user space of the application. Traffic Inspector proxy server (implemented as part of the service) is designed for active use of available memory. The 32-bit version of the service often quickly “consumed” limited memory resources, rested against the memory ceiling, the program could not open more than 4096 threads in the system, the system hung under load and the proxy started to slow down a lot. The most important improvement of the program is the release of the 64-bit version of the service, released at the end of 2013. In the new version there are both x86 and x64 versions and the user himself determines which one to install. You can forget about past memory problems, and now the proxy can easily handle up to 1000 simultaneous HTTP requests.
Alexander Yamashkin on the implementation of work on 64-bit platforms:
Now there are both 32-bit and 64-bit versions of the program, and the user himself determines which one to install (at the moment 70-80% of installations fall on the 64-bit one). Using the latter removes the limitations imposed by the operating system on the amount of memory in 2 GB, which can be allocated to the process, and therefore there are no errors that occur when it is impossible to allocate the next memory block to the process.
Slow web portal and reporting in Traffic Inspector
The Traffic Inspector web portal provides the generation of network operation reports and other aspects of network statistics. Many customers acquired Traffic Inspector for this very purpose - statistical data collection and reporting. One could understand their disappointment when the process of forming even a simple report could drag on for a long time. This problem was overcome by translating the web portal to PHP and related technologies.
Alexander Yamashkin about the translation of the portal in PHP:
Prior to version 3.0.1, the portal was written in C # c using the .Net Framework. In addition to changing the interpreter, a report preparation service was implemented, into which all the reporting logic was moved. The portal now has only one function - the GUI.
In addition to speeding up the work of the portal, database distribution and data storage for a long period of time were introduced:
If you need to store data for a large interval, we recommend the use of third-party DBMS. But even with them from time to time there were problems. Imagine the complexity of servicing a 1.5 TB database in MS SQL, while 90% of this data is not used. Therefore, in 3.0.2, the database dividing mechanism was implemented, when, at the end of the period (month / quarter / half year / year - specified by the administrator), the database, including SQLite, is disconnected from operational work and is used only when requesting reports for the specified period.
Ease of use Traffic Inspector
Many customers who first encountered the Traffic Inspector program often talked about the great complexity in its development and the incomprehensible interface. Recently, a number of efforts have been made to improve the usability of the program.
Significantly changed the interface Traffic Inspector. Now use the various elements of the program has become easier and more convenient. In the new structure of the console tree, the elements are more logical and convenient. The content of the pages for the main nodes of the console tree has been completely redesigned (Objects, Traffic Accounting, Users and Groups, Rules, Services, Settings). As a result, the administrator of the Traffic Inspector program receives simple and convenient control over all the program's features, comprehensive information about its settings and links for solving basic administrative tasks. The new system of adding users and a number of wizards make the initial setup of the program as easy as possible.
Alexander Yamashkin about the new user addition system:
The system, fixing access to the Internet from a user not registered in the program, can either automatically create an account based on the specified parameters, or create a request for the administrator, who can only confirm the creation of a new account and assign the necessary access rules.
New to Traffic Inspector 3.0.2
The latest changes in the program will be commented by the head of the development department Alexander Yamashkin:
The main improvements were related to the reports and the new registration system, the remaining ones were rather specific. For example:
- Implemented the ability to view requests blocked by access rules. This will allow both to understand the reason for blocking, and to see which users tried to access prohibited resources.
- Display of extended information when a resource is blocked, which is included in the Unified Register of Domain Names containing information, the distribution of which in the Russian Federation is prohibited.
- Work was done to minimize the installer, by eliminating the signatures of the anti-virus database and the library of anti-virus modules. This data will be loaded when the modules are first used.
- Produced work on optimizing the speed of building reports. Not only were all requests processed, but the data hashing system was implemented in the report generation service.
Future plans to change Traffic Inspector
This will tell Alexander Yamashkin and Vaidas Damoshevichyus, CTO and partner of the company Smart-Soft.
Alexander Yamashkin:
I really want to make the process of creating rules for access to information resources as simple and understandable as a sophisticated and novice administrator.
Vaidas Damoshevichus:
Now we are actively exploring technological solutions for classifications of packages at the seventh level (Layer 7) - we are looking for what we could integrate into the Traffic Inspector. After successful integration to our users, we will present a new functionality. When it will be? Then, when we ourselves will be satisfied with the results of integration, since we set high quality indicators for ourselves. We also did not forget about the reports - we want to improve them and make not just reports, but a set of analytical tools that will help system administrators to monitor and monitor network anomalies.Conclusion
The company Smart-Soft took into account the shortcomings of its software and carried out work to eliminate. The program moved to the 64-bit platform, significantly increasing the performance of individual components. Numerous corrections and additions touched most aspects of the program. The new Traffic Inspector is a multifunctional Internet gateway of a new generation with modern functionality, but at the same time an interface that every user can understand.
PS: If there is something that we have not said, suggestions for correction, we will be glad to hear your comments.
Source: https://habr.com/ru/post/254741/
All Articles