Web applications integration with commodity accounting systems - text of the report from the conference DUMP-IT.RU
On May 30, we gave a number of reports in different sections of the DUMP conference, which was held in Yekaterinburg. Initially, we were asked to touch on the holivar topic “why 1C-Bitrix”, shit :-), but we decided to limit ourselves to the problems of customizing boxes when creating sites synchronized with commodity accounting systems (hereinafter referred to as STU)
So, that's what the report was about.
Artsofte most often faces the challenges of integrating an online store with 1C \ Scala \ SuperMAG and online payment systems. We also introduce tracking systems for order status, but are not limited to them. The pool of integrable solutions may also include more complex systems, such as online banking on the customer side, integration with billing systems, etc ...
')
We deal exclusively with custom development on symfony (PHP) and .NET, and often clients who already have all kinds of experience come to us.
But what if an alternative solution is used as one of the links: a CMS or a custom system on the one hand and an alternative commodity accounting software from the other side?
In this case, things are not so rosy. If the commodity accounting system or CMS does not have synchronization tools, or these tools are incompatible, it is necessary to refine one or both links, which can turn into a significant cost for the customer both in time and in cost. These costs may even exceed the cost of the remaining work, or force the customer to switch to another solution (meaning the solution for commodity accounting or the store)
One way or another, “finished” or taken straight out of the box, such a solution will eventually exhaust its limits on the standard functionality. The box will be full. Dead end.
The scenarios described above, in addition to the problems of integration as such, can also have “side effects” for the customer.
Our seo-division was engaged in the promotion of the site detsad.ru, developed by another company. This site is an uncomplicated online store with a three-level directory hierarchy and configured synchronization with 1C-Accounting, by manually downloading the CSV file through the CMS.
The path to the catalog page (in this case the link with the maximum nesting level is presented) looks like this: www.detsad.ru/catalogue/200/3598 , where the last two numbers are the number of the category and product, respectively.
The problem was that these numbers correspond to the identifiers in 1C, which tend to change. As a result of synchronization, the old links to products became incorrect and, as a result, the site lost positions in search engines. As a result, we rewrote the parser.
Here is another example. Let's say my friends and I decided to have a beer. We love Hougarden, and decided to order a box. We love to buy at the Zvezdny supermarket, but we are too lazy to go to it. Therefore, we decided to order a beer in the online store "Star". Once on the homepage of the online store, choose: "Beer", "San InBev", "SB" ... what the hell is this? What else is “San InBev” ?! What kind of "SB", "RC", "PB"? Reinforced concrete or what?
I directly imagine how I enter the store and look at the huge signboard: “Beer”. I wrap myself up in the “San InBev” section, I am looking for the “SB” shelving, and on it ... Oh, no, it’s already the beer that got sick. I'll go buy water better. Before her not so deep.
As a result, the client understands that he needs a new solution, which will not only fully take into account business logic, but also improve along with his business.
Based on our experience, we can with a high degree of probability assume that during the integration of STU and Web frontend, the developer simply copies the directory structure from the STU.
The result is a low conversion.
When we start a custom integration project, in most cases we have to deal with the fact that companies, as a rule, do not have an established IT strategy. Not described the rules and policies for the development, implementation, use of information systems, principles and methods of integration. Starting work on a project, you have to develop these documents from scratch, to formalize business processes. Work on such projects usually begins with the formalization of the task, which we call the “Black Box Model”.
The scheme of this model is as follows:
At the entrance there is usually a development task containing the requirements of the customer in general, as well as general data on the commodity accounting systems used. The developed Web application and STU are black boxes, the internal structure of which, if known, is usually in the most general terms. The first step in working on a project is a decomposition of a web application, which allows you to understand the requirements for its own architecture, and on the basis of these requirements, build the logic of interaction with STU.
But why not start the implementation right away with a detailed analysis and not immediately come to a custom solution?
In most cases, the answer is simple. Offering the customer their services, the companies estimate the required amount of work differently, and when the offers differ in price up to 30 times (the real case from our practice with the B2B store shop.nag.ru ), the offers from the lower price segment look most attractive to the customer but at the same time they hide a trick.
Being one of the key factors, the low cost of implementing the system does not mean that we saved, and when choosing a solution we have to evaluate in a broader aspect, for example, calculate the cost of owning an integrated solution, but it will include:
The last item often becomes the basis of the project budget.
The absence in the commercial offer of mention of the cost of maintenance creates a rosy picture for the customer and the fact is that a packaged solution will be really inexpensive in terms of implementation.
Online store on Bitrix or VirtueMart with a template design takes place in a couple of hours, not counting the content. After that, the unloading is also quickly adjusted. But what to do if the necessary information block is not in the system or the description of the data exchange format does not contain information about the balances in warehouses? “Finishing” large and, especially, commercial CMS is very difficult. And therefore it is very expensive.
So how does “make friends” STU with a web application? To answer this question, first consider the possible integration models.
Let us turn to the experience of our Western colleagues. It is worth noting that in the Western market the number of different products and solutions is much greater than ours, and the opportunities for interaction are often very fragmented. In addition, many systems require much more complex and deep integration than unloading commodity items from the ERP system. To ensure flexible and reliable integration in these cases, so-called. Middleware, an intermediate layer that provides data exchange between disparate subsystems, such as eCommerce, ERP, Web-services, etc. Combining many formats and data transfer protocols, providing the possibility of converting one format to another, middleware also provides a high degree of reliability and flexibility of an integrated system, since changing one of its components will only entail a change in the rules of interaction with it. Sometimes, speaking of integration Middleware, they introduce the concept of an integration bus, which provides a single channel of data exchange for all nodes of the system, regardless of their origin and geographical location.
A typical interaction diagram looks like the diagram below:
Architecture of Pervasive Data Integrator Middleware interaction with applications.
The arrows indicate the direction of data transmission and the protocols used for various tasks. We list the most popular ones.
Data transfer using these protocols is usually provided by HTTP / S and FTP / S transports.
Having the necessary tools, it is often sufficient to describe the data structures and select the most appropriate exchange protocol.
But we still have a low level of automation of business processes in mind, on the one hand, the task is narrowed, and on the other hand, it becomes more complicated. Firstly, the market for Web integration in Russia, according to experts, is underdeveloped, which means there is simply no need for Middleware. Western products also often do not suit us due to the tradition of development in Russia, when each company strives to have its own solution. But on the other hand, a small number of vendors narrows the set of possible cases and often reduces the task of integration to the need to integrate the commodity accounting system with an online storefront or online store. We divided the interactions of the Web application and the product system, depending on the complexity of the solution, into 3 levels in the areas of data exchange and data sets transmitted to both sides.
Read more about them here.
To ensure interactions at all three levels, we use the so-called. interface layer This layer allows you to upload the necessary data for synchronization in the required format that meets the requirements of business logic. However, we do not use bloated XML standards like CommerceML, but adapt protocols to the needs of a particular model.
When designing interactions for web applications and STU, we usually work in close connection with specialists on this STU.
This allows us to receive data in a convenient format for processing and not limit ourselves to the built-in capabilities of exporting STU data. An extension (upload) is connected to the STU, which provides data for processing on the Web application side.
In the case of two-way exchange, this extension also assumes the functions of data transfer to the STU. The web application contains the same module that performs the parsing and updating of the database, as well as initiating synchronization-related events.
In the case of two-way communication, the Web application interface module is responsible for the aggregation and serialization of the exported data. Ok, now STU and Web application can understand each other. However, the interaction must somehow be initiated. We will immediately mark the manual initiation option. Since our task is to automate the interaction process as much as possible, we consider only automatic methods of initiation.
The manual method remains as a fallback in case of emergency when one of the services fails. Automatic initiation of synchronization implies that under certain well-defined conditions, one of the sides of the interface layer raises a synchronization event, and the other party handles this event.
Today we use two different ways to initiate synchronization, depending on the available data transmission channels and the technical capabilities of the STU.
Uploading to an XML file implies that the data is saved as an XML file in a specific location on an FTP server. The web application periodically checks the files stored on the server for updates and, if a new version is found, it synchronizes. The STU interface extension behaves in the same way.
In the initiator-listener model, the Web application plays the role of a listener, and the STU interface module sends requests to the Web application, as a result of which actions to send or receive information are initiated in the Web application. This model does not require the presence of any third-party server, since its functions are performed by the Web application, but it does require that the Web client functions in the interface module of the CTS. Such functions, for example, are in 1C v8.
With regard to data parsing and updating Web application databases, for each application such module is designed separately, taking into account the current and projected needs of business logic, as well as the requirements for performance and fault tolerance. The range of solutions varies from simple XML parsing and subsequent sequential data entry into tables to multi-stream transactional models with integrity control and model-level triggers. In this case, by model, I mean the MVC architecture level used in studio projects.
As can be seen from the above, customization of the integration solution for the business processes of the customer begins at the design stage of interactions with the CTS. However, this is only a small part. More acutely, the need for a custom integrative solution begins to be felt when it comes to such things as:
When developing a custom product, we are practically unlimited in our decisions, since we do not impose upon ourselves neither an architectural framework nor an instrumental framework. We have our own component base, and we can do anything.
Well, that's it! =)
So, that's what the report was about.
Artsofte most often faces the challenges of integrating an online store with 1C \ Scala \ SuperMAG and online payment systems. We also introduce tracking systems for order status, but are not limited to them. The pool of integrable solutions may also include more complex systems, such as online banking on the customer side, integration with billing systems, etc ...
')
Problem
We deal exclusively with custom development on symfony (PHP) and .NET, and often clients who already have all kinds of experience come to us.
Coming to customers, we often encounter 3 options of hopelessness.
- a) The client does not have a clearly formulated task and asks the question: “What if I try?” In this case, we recommend the client to play with one of the box versions first, and there see p.
- b) The client makes a decision to create his own design, refers to the solution of the lower price segment, and as a result receives a “bicycle”. And, it would seem, all is well, the customer is satisfied. He periodically uploads goods to the site, clogs orders in 1C. But after a while it turns out that the wheels of the bike are square, the seat is not adjustable, and the gearshift is not provided in principle. Caught without the support of the creator of his system, the client is in a deadlock.
- c) The client chose a “boxed” solution. Historically, despite the presence of various vendors of STU solutions, such as Oracle, Scala, SuperMag, etc., in most cases STU is understood as 1C: Enterprise complex in all its configurations. Therefore, when it comes to integrating an online store with a commodity accounting system or an accounting system, first of all solutions are considered that provide integration with 1C out of the box. And it is quite logical that it comes first of all about the bundle 1C-Enterprise and 1C-Bitrix. This solution is really quite simple to implement, and for some time things are much better than in the first version. The web application will be developed along with the customer’s business requirements.
But what if an alternative solution is used as one of the links: a CMS or a custom system on the one hand and an alternative commodity accounting software from the other side?
In this case, things are not so rosy. If the commodity accounting system or CMS does not have synchronization tools, or these tools are incompatible, it is necessary to refine one or both links, which can turn into a significant cost for the customer both in time and in cost. These costs may even exceed the cost of the remaining work, or force the customer to switch to another solution (meaning the solution for commodity accounting or the store)
One way or another, “finished” or taken straight out of the box, such a solution will eventually exhaust its limits on the standard functionality. The box will be full. Dead end.
The scenarios described above, in addition to the problems of integration as such, can also have “side effects” for the customer.
An example of problems caused by curve integration:
1. Case "detsad.ru"
Our seo-division was engaged in the promotion of the site detsad.ru, developed by another company. This site is an uncomplicated online store with a three-level directory hierarchy and configured synchronization with 1C-Accounting, by manually downloading the CSV file through the CMS.
The path to the catalog page (in this case the link with the maximum nesting level is presented) looks like this: www.detsad.ru/catalogue/200/3598 , where the last two numbers are the number of the category and product, respectively.
The problem was that these numbers correspond to the identifiers in 1C, which tend to change. As a result of synchronization, the old links to products became incorrect and, as a result, the site lost positions in search engines. As a result, we rewrote the parser.
2. Case "Star"
Here is another example. Let's say my friends and I decided to have a beer. We love Hougarden, and decided to order a box. We love to buy at the Zvezdny supermarket, but we are too lazy to go to it. Therefore, we decided to order a beer in the online store "Star". Once on the homepage of the online store, choose: "Beer", "San InBev", "SB" ... what the hell is this? What else is “San InBev” ?! What kind of "SB", "RC", "PB"? Reinforced concrete or what?
I directly imagine how I enter the store and look at the huge signboard: “Beer”. I wrap myself up in the “San InBev” section, I am looking for the “SB” shelving, and on it ... Oh, no, it’s already the beer that got sick. I'll go buy water better. Before her not so deep.
As a result, the client understands that he needs a new solution, which will not only fully take into account business logic, but also improve along with his business.
Based on our experience, we can with a high degree of probability assume that during the integration of STU and Web frontend, the developer simply copies the directory structure from the STU.
The result is a low conversion.
When we start a custom integration project, in most cases we have to deal with the fact that companies, as a rule, do not have an established IT strategy. Not described the rules and policies for the development, implementation, use of information systems, principles and methods of integration. Starting work on a project, you have to develop these documents from scratch, to formalize business processes. Work on such projects usually begins with the formalization of the task, which we call the “Black Box Model”.
The scheme of this model is as follows:
At the entrance there is usually a development task containing the requirements of the customer in general, as well as general data on the commodity accounting systems used. The developed Web application and STU are black boxes, the internal structure of which, if known, is usually in the most general terms. The first step in working on a project is a decomposition of a web application, which allows you to understand the requirements for its own architecture, and on the basis of these requirements, build the logic of interaction with STU.
But why not start the implementation right away with a detailed analysis and not immediately come to a custom solution?
In most cases, the answer is simple. Offering the customer their services, the companies estimate the required amount of work differently, and when the offers differ in price up to 30 times (the real case from our practice with the B2B store shop.nag.ru ), the offers from the lower price segment look most attractive to the customer but at the same time they hide a trick.
Being one of the key factors, the low cost of implementing the system does not mean that we saved, and when choosing a solution we have to evaluate in a broader aspect, for example, calculate the cost of owning an integrated solution, but it will include:
- Actually, the cost of basic software components
- Cost of implementation of software components, if any
- The cost of implementing an integration solution
- Costs for the subsequent customization of the integration solution
The last item often becomes the basis of the project budget.
The absence in the commercial offer of mention of the cost of maintenance creates a rosy picture for the customer and the fact is that a packaged solution will be really inexpensive in terms of implementation.
Online store on Bitrix or VirtueMart with a template design takes place in a couple of hours, not counting the content. After that, the unloading is also quickly adjusted. But what to do if the necessary information block is not in the system or the description of the data exchange format does not contain information about the balances in warehouses? “Finishing” large and, especially, commercial CMS is very difficult. And therefore it is very expensive.
Integration Models:
So how does “make friends” STU with a web application? To answer this question, first consider the possible integration models.
Western model
Let us turn to the experience of our Western colleagues. It is worth noting that in the Western market the number of different products and solutions is much greater than ours, and the opportunities for interaction are often very fragmented. In addition, many systems require much more complex and deep integration than unloading commodity items from the ERP system. To ensure flexible and reliable integration in these cases, so-called. Middleware, an intermediate layer that provides data exchange between disparate subsystems, such as eCommerce, ERP, Web-services, etc. Combining many formats and data transfer protocols, providing the possibility of converting one format to another, middleware also provides a high degree of reliability and flexibility of an integrated system, since changing one of its components will only entail a change in the rules of interaction with it. Sometimes, speaking of integration Middleware, they introduce the concept of an integration bus, which provides a single channel of data exchange for all nodes of the system, regardless of their origin and geographical location.
A typical interaction diagram looks like the diagram below:
Architecture of Pervasive Data Integrator Middleware interaction with applications.
The arrows indicate the direction of data transmission and the protocols used for various tasks. We list the most popular ones.
Web Services Technologies and Protocols
- Simple Object Access Protocol (SOAP) is an extension of XML-RPC and provides the ability to exchange arbitrary messages in XML format. Initially used to implement remote procedure call (RPC). Allows the Web service to provide reusable components. Used over text protocols, mostly HTML
- Representational State Transfer (REST) ​​- in this case, data is transferred using a small number of standard formats - HTML, XML, JSON. The network transfer protocol must support caching, it should not save state information. Data exchange is built in the form of request-response. In some cases, it can be used to transfer SOAP objects.
- XML is typically used to transfer messages and objects. It has a hierarchical structure and great flexibility.
- Message Queues - There is a message pool, they are stored until processing.
- CommerceML - Russian standard for synchronization with 1C: Accounting, actively promoted by 1C and other mastodons, is absolutely stupid because it cannot be international and, as a result, is not supported by large ERP, however, it deserves attention due to the native implementation in 1C v8
Data transfer using these protocols is usually provided by HTTP / S and FTP / S transports.
Having the necessary tools, it is often sufficient to describe the data structures and select the most appropriate exchange protocol.
Web and commodity accounting in Russia
But we still have a low level of automation of business processes in mind, on the one hand, the task is narrowed, and on the other hand, it becomes more complicated. Firstly, the market for Web integration in Russia, according to experts, is underdeveloped, which means there is simply no need for Middleware. Western products also often do not suit us due to the tradition of development in Russia, when each company strives to have its own solution. But on the other hand, a small number of vendors narrows the set of possible cases and often reduces the task of integration to the need to integrate the commodity accounting system with an online storefront or online store. We divided the interactions of the Web application and the product system, depending on the complexity of the solution, into 3 levels in the areas of data exchange and data sets transmitted to both sides.
- One-way communication of the nomenclature from 1C and its output to the web-storefront of the site
- Two-way communication of the nomenclature and counterparties from 1C with the web-showcase of the site and personal account
- Two-way communication of the nomenclature, contractors and orders from 1C with the web-showcase of the site and personal account
Read more about them here.
To ensure interactions at all three levels, we use the so-called. interface layer This layer allows you to upload the necessary data for synchronization in the required format that meets the requirements of business logic. However, we do not use bloated XML standards like CommerceML, but adapt protocols to the needs of a particular model.
When designing interactions for web applications and STU, we usually work in close connection with specialists on this STU.
This allows us to receive data in a convenient format for processing and not limit ourselves to the built-in capabilities of exporting STU data. An extension (upload) is connected to the STU, which provides data for processing on the Web application side.
In the case of two-way exchange, this extension also assumes the functions of data transfer to the STU. The web application contains the same module that performs the parsing and updating of the database, as well as initiating synchronization-related events.
In the case of two-way communication, the Web application interface module is responsible for the aggregation and serialization of the exported data. Ok, now STU and Web application can understand each other. However, the interaction must somehow be initiated. We will immediately mark the manual initiation option. Since our task is to automate the interaction process as much as possible, we consider only automatic methods of initiation.
The manual method remains as a fallback in case of emergency when one of the services fails. Automatic initiation of synchronization implies that under certain well-defined conditions, one of the sides of the interface layer raises a synchronization event, and the other party handles this event.
Today we use two different ways to initiate synchronization, depending on the available data transmission channels and the technical capabilities of the STU.
- Upload to XML file, and transfer via FTP
- Model "Initiator-listener"
Uploading to an XML file implies that the data is saved as an XML file in a specific location on an FTP server. The web application periodically checks the files stored on the server for updates and, if a new version is found, it synchronizes. The STU interface extension behaves in the same way.
In the initiator-listener model, the Web application plays the role of a listener, and the STU interface module sends requests to the Web application, as a result of which actions to send or receive information are initiated in the Web application. This model does not require the presence of any third-party server, since its functions are performed by the Web application, but it does require that the Web client functions in the interface module of the CTS. Such functions, for example, are in 1C v8.
With regard to data parsing and updating Web application databases, for each application such module is designed separately, taking into account the current and projected needs of business logic, as well as the requirements for performance and fault tolerance. The range of solutions varies from simple XML parsing and subsequent sequential data entry into tables to multi-stream transactional models with integrity control and model-level triggers. In this case, by model, I mean the MVC architecture level used in studio projects.
Customization
As can be seen from the above, customization of the integration solution for the business processes of the customer begins at the design stage of interactions with the CTS. However, this is only a small part. More acutely, the need for a custom integrative solution begins to be felt when it comes to such things as:
- Interactivity and close ties between the components of the user interface in the server and in the client side. This is especially true for tasks performed on the client side and exchanging with the Web application server via AJAX requests. Such interactions require obtaining a large number of different data representations that are difficult to form based on boxed solutions.
- Data views, flexible filters, customizable formats and content of search results
- Possibilities for managing the order from the side of not only the client, but also the call center operator. Moreover, in the latter case, the specificity of the work requires that the functionality of the user interfaces of the operator be tuned for maximum productivity.
- The possibility of introducing a simple hierarchy and "pseudo categories". By pseudo-categories we mean such entities that functionally correspond to one of the directory sections, but do not have a comparison with the essence of this level in the database. To pseudocategories can be attributed such as "our products" or "goods at a discount"
- And much more.
When developing a custom product, we are practically unlimited in our decisions, since we do not impose upon ourselves neither an architectural framework nor an instrumental framework. We have our own component base, and we can do anything.
Well, that's it! =)
Source: https://habr.com/ru/post/120984/
All Articles