Megatons of waste paper with a flick of the wrist
Good day. In this article we will tell about how print is arranged in our platform .
For understanding, it will be useful to tell about the history of this function. Initially, the decision that was called "in the forehead." A screw spooler was used and printing was done through standard procedures. Actually, this decision did not last long, exactly until the moment when it was necessary to implement printing from the application server, and not from the client application.
Just in case, I will give the scheme from the first article:
As you can see, the application servers were in the data center. And the offices were connected to a single network. As soon as we implemented printing from the application server there were several problems:
In fact, the investigation quickly showed that the reasons for the high load on the channels, as everyone probably guessed, were the transfer of the rendered printed form directly to the printer.
I had to tinker with cause number 2, but it turned out that people also write printer drivers! It was they (the drivers) that hung up under high load and pulled the entire application server behind them. It was possible to remove it from the catatonic state only by its restart.
Reasons 3 and 5 were found in the print spooler Windows.
We managed to figure out problem 4 by carefully examining the expectations of the sessions. The reasons, as it now seems, lie on the surface, but then we were young ... It turned out that because of printing from the application server, the access time to the printer had greatly increased - now we were printing via Internet channels, and not over the local network. All this time, the transaction continued to block the objects of the system, which came across other server calls and gave us funny moments. It would be possible, of course, to fix the transaction before printing off, but this is wrong from our point of view. We apologize, but about this in another article about transaction management.
According to acquaintances, I know that such problems were not only with us.
Then we decided to do what is shown on the diagram as a print server.
')
We need to make a digression and tell us what we actually typed.
For printing in the platform implemented such a thing as "Printing forms". Sorry, we could not think of a more original name. A printed form is a template for a report and a script that prepares a data plate based on some algorithm specified by the developer. Yes, sometimes there are cases when it cannot be formulated as a single query. It is necessary to clarify - not as a table in a database, but as an object of the “type” DataTable. In quotes - because it uses its own class for storing tables, less memory-consuming and more compact serializable.
If we talk about the details, we use a report from DevExpress XtraReports.
Both the script and the report template are available for editing at run-time through the GUI of the main client application.
The script, I remind you, is always executed on the application server. The overwhelming part of the papers printed by the trade organization are all kinds of invoices, invoices, invoices, sales receipts, set sheets in the warehouse, and so on and so forth. At the same time, the pattern does not change, only the data changes!
Thus, the first thing we did was to submit to the print server the task of rendering and sending a rendered form to the printer. The application server only prepared the data and sent it to the print server. Saving on traffic - from (on average) 2MB per document up to 40Kb, the print server caches templates locally.
By consistently increasing the functionality of the print server, we were able to get rid of these problems.
So, what we have now.
As a rule, there is no need for time-guaranteed delivery of the printing form to the printer (i.e. print out right now). And, on the other hand, the script that sends the document to print has already made some changes in the database and it is likely that another transaction might run into these locks. In this case, you need to fix the changes as soon as possible. For this, the application server can accept a print job to process it asynchronously. In a separate transaction, run the data preparation script and send it to the appropriate print server. Synchronous printing implies that the data preparation script will be executed in the current transaction.
Illustrating scheme:
The PrintBuilder and PrintSender procedures, running in separate transactions (and streams) within the application server, process the queues of incoming jobs. Technically, these are threads within the application server that perform an infinite loop. The number of these threads is configured by the administrator depending on the load. PrintBuilder processes the jobs received asynchronously, runs the scripts to prepare the printed form data and sends them to the next queue to PrintSender. Synchronous print jobs are also included in the same queue.
In turn, PrintSender pulls jobs from the queue and sends them to the print server.
As you can see, with this approach, the blocking time is drastically reduced in Transaction 1, and the shorter the blocking time, the higher the number of transactions that the server can handle!
Guaranteed printing on the printer documents in a given sequence. Regardless of other tasks, all documents in the package will be printed inseparably. We needed the opportunity when implementing the printing of documents for drivers ( an example of application in the practice of real business). This is a task when you need to print the Itinerary (a document with a list of all addresses and a map) and then accounting documents for each order. Everything was good until the drivers were over 100 and the Windows spooler did not get confused. The situation worsened if someone else tried to print other documents on the same printer. These documents for some reason were sometimes among the documents for the driver. I had to abandon the spooler and make my own.
The document sent to print will be guaranteed to be sent to the printer. If the printer is unavailable, or the office with the printer server is unavailable, the system will wait until the connection is restored and send SMS and email to administrators.
Here is a diagram illustrating the print server operation:
In this diagram, PrintWorker is a separate process launched for each connected printer. If the printer driver freezes, the process stops responding, and when the timeout expires, the print server kills the process and restarts.
A side effect is taking into account who, when, what, and how much printed, well, users no longer have direct access to printers, which greatly reduced paper consumption for printing dissertations. Whether this is a cool thing - decide for yourself.
Printers that can be printed from the system are listed in the system settings and are connected only to the print server. This greatly simplifies the work of administrators - no need to connect the printer to each node of the cluster of application servers, and on each client machine, install drivers there and do other shamanism. In addition, listing the printers in the system makes it easy to implement a UI to select a printer for regular users. For example on which printer to print paper in stock.
Thus, each printer has its own identifier, which is saved when the printer is changed, and the code for printing looks like this:
In this example, the printout for the specified document is printed to the printer specified in the employee.
They promised to tell you about the queue. We tell.
In the print subsystem, as can be seen, the queue mechanism is actively used. Fortunately, in Oracle Database it is possible to organize them within the database, without resorting to special individual services. They are implemented quite simply using the design
The main thing in this procedure is to declare the cursor and pull out one record. Thus, without unnecessary locks and very conveniently, you can organize parallel processing of various queues.
For example, the problem of recalculating prices for 400,000 goods is solved in the same way.
Anticipating the question of why Oracle Advanced Queuing was not used, we answer:
Hopefully, we managed to roughly open up the implementation of the printing subsystem in our platform and push on ideas for solving problems in your applications. Next, we will try to uncover the topic of managing transactions and related problems.
A bit of history
For understanding, it will be useful to tell about the history of this function. Initially, the decision that was called "in the forehead." A screw spooler was used and printing was done through standard procedures. Actually, this decision did not last long, exactly until the moment when it was necessary to implement printing from the application server, and not from the client application.
Just in case, I will give the scheme from the first article:
As you can see, the application servers were in the data center. And the offices were connected to a single network. As soon as we implemented printing from the application server there were several problems:
- Loading of communication channels has grown very significantly. In some offices up to 100%
- The application server started to hang.
- Sometimes a piece of paper sent to print was simply not printed.
- System performance has dropped (users started complaining that they had to wait for the completion of operations for several minutes)
- Reversing the order of documents when printing
In fact, the investigation quickly showed that the reasons for the high load on the channels, as everyone probably guessed, were the transfer of the rendered printed form directly to the printer.
I had to tinker with cause number 2, but it turned out that people also write printer drivers! It was they (the drivers) that hung up under high load and pulled the entire application server behind them. It was possible to remove it from the catatonic state only by its restart.
Reasons 3 and 5 were found in the print spooler Windows.
We managed to figure out problem 4 by carefully examining the expectations of the sessions. The reasons, as it now seems, lie on the surface, but then we were young ... It turned out that because of printing from the application server, the access time to the printer had greatly increased - now we were printing via Internet channels, and not over the local network. All this time, the transaction continued to block the objects of the system, which came across other server calls and gave us funny moments. It would be possible, of course, to fix the transaction before printing off, but this is wrong from our point of view. We apologize, but about this in another article about transaction management.
According to acquaintances, I know that such problems were not only with us.
Then we decided to do what is shown on the diagram as a print server.
')
Printed forms
We need to make a digression and tell us what we actually typed.
For printing in the platform implemented such a thing as "Printing forms". Sorry, we could not think of a more original name. A printed form is a template for a report and a script that prepares a data plate based on some algorithm specified by the developer. Yes, sometimes there are cases when it cannot be formulated as a single query. It is necessary to clarify - not as a table in a database, but as an object of the “type” DataTable. In quotes - because it uses its own class for storing tables, less memory-consuming and more compact serializable.
If we talk about the details, we use a report from DevExpress XtraReports.
Both the script and the report template are available for editing at run-time through the GUI of the main client application.
The script, I remind you, is always executed on the application server. The overwhelming part of the papers printed by the trade organization are all kinds of invoices, invoices, invoices, sales receipts, set sheets in the warehouse, and so on and so forth. At the same time, the pattern does not change, only the data changes!
Thus, the first thing we did was to submit to the print server the task of rendering and sending a rendered form to the printer. The application server only prepared the data and sent it to the print server. Saving on traffic - from (on average) 2MB per document up to 40Kb, the print server caches templates locally.
Device and features
By consistently increasing the functionality of the print server, we were able to get rid of these problems.
So, what we have now.
Synchronous and asynchronous printing.
As a rule, there is no need for time-guaranteed delivery of the printing form to the printer (i.e. print out right now). And, on the other hand, the script that sends the document to print has already made some changes in the database and it is likely that another transaction might run into these locks. In this case, you need to fix the changes as soon as possible. For this, the application server can accept a print job to process it asynchronously. In a separate transaction, run the data preparation script and send it to the appropriate print server. Synchronous printing implies that the data preparation script will be executed in the current transaction.
Illustrating scheme:
The PrintBuilder and PrintSender procedures, running in separate transactions (and streams) within the application server, process the queues of incoming jobs. Technically, these are threads within the application server that perform an infinite loop. The number of these threads is configured by the administrator depending on the load. PrintBuilder processes the jobs received asynchronously, runs the scripts to prepare the printed form data and sends them to the next queue to PrintSender. Synchronous print jobs are also included in the same queue.
In turn, PrintSender pulls jobs from the queue and sends them to the print server.
As you can see, with this approach, the blocking time is drastically reduced in Transaction 1, and the shorter the blocking time, the higher the number of transactions that the server can handle!
Batch printing
Guaranteed printing on the printer documents in a given sequence. Regardless of other tasks, all documents in the package will be printed inseparably. We needed the opportunity when implementing the printing of documents for drivers ( an example of application in the practice of real business). This is a task when you need to print the Itinerary (a document with a list of all addresses and a map) and then accounting documents for each order. Everything was good until the drivers were over 100 and the Windows spooler did not get confused. The situation worsened if someone else tried to print other documents on the same printer. These documents for some reason were sometimes among the documents for the driver. I had to abandon the spooler and make my own.
Guaranteed delivery
The document sent to print will be guaranteed to be sent to the printer. If the printer is unavailable, or the office with the printer server is unavailable, the system will wait until the connection is restored and send SMS and email to administrators.
Here is a diagram illustrating the print server operation:
In this diagram, PrintWorker is a separate process launched for each connected printer. If the printer driver freezes, the process stops responding, and when the timeout expires, the print server kills the process and restarts.
Print accounting
A side effect is taking into account who, when, what, and how much printed, well, users no longer have direct access to printers, which greatly reduced paper consumption for printing dissertations. Whether this is a cool thing - decide for yourself.
Printer virtualization
Printers that can be printed from the system are listed in the system settings and are connected only to the print server. This greatly simplifies the work of administrators - no need to connect the printer to each node of the cluster of application servers, and on each client machine, install drivers there and do other shamanism. In addition, listing the printers in the system makes it easy to implement a UI to select a printer for regular users. For example on which printer to print paper in stock.
Thus, each printer has its own identifier, which is saved when the printer is changed, and the code for printing looks like this:
In this example, the printout for the specified document is printed to the printer specified in the employee.
RDBMS Queues
They promised to tell you about the queue. We tell.
In the print subsystem, as can be seen, the queue mechanism is actively used. Fortunately, in Oracle Database it is possible to organize them within the database, without resorting to special individual services. They are implemented quite simply using the design
select .. for update skip locked The main thing in this procedure is to declare the cursor and pull out one record. Thus, without unnecessary locks and very conveniently, you can organize parallel processing of various queues.
For example, the problem of recalculating prices for 400,000 goods is solved in the same way.
Anticipating the question of why Oracle Advanced Queuing was not used, we answer:
- We use Managed ODP.NET, but it still does not support AQ.
- Using your queues allows you to store data in a structured form, and implement built-in mechanisms for managing these queues:
Conclusion
Hopefully, we managed to roughly open up the implementation of the printing subsystem in our platform and push on ideas for solving problems in your applications. Next, we will try to uncover the topic of managing transactions and related problems.
Source: https://habr.com/ru/post/242595/
All Articles