How to write an exchange with 50 suppliers and do not go crazy
Suppliers are different. Some are ready to adapt to our format - others are not; some exchange SOAP - others REST; some work with product codes — others with offer identifiers; some are ready to give status by request - others are not; some have reference books whose elements you need to compare with yours - others do not. In general, very different.
Data exchange with suppliers needs to be automated. 21st century in the yard. It is clear that in an ideal world it is necessary to convene a conference of suppliers and consumers in this industry (in my case, these are auto parts), at which to agree on a single format for the exchange of information and everyone will be happy. At the pharmacy is. But we live in a non-ideal world.
The initial idea is the same for everyone: the supplier is ready to give you the elements of his directories and is ready to take an order from you. But all suppliers have their own business processes, and this imposes different initial requirements on suppliers' APIs, and different programmers who implement them bring the picture to the point that the API of different suppliers has no common features other than the initial idea.
How to write an exchange in fifty formats? Write fifty exchanges? Let's assume that in the best case an exchange with one supplier is to write and test 1 week and the programmer’s rate is 3000r / hour. Multiplied? And I did not recall the support and administration of this whole zoo.
')
There are other options? My answer: write one exchange!
We have to invent a 51st supplier and write an exchange with him. This should be the ideal supplier for us, for our processes. This supplier should have common features of all suppliers, differing from them in details.
We should start thinking not “from the supplier”, but “from us” and the picture will instantly become simpler. We have one business process and one programmer as well, and everything that goes beyond what is reasonable in our opinion is not particularly interesting to us. It is clear that such an approach will inevitably lead to “delusions” in logic and some kind of “subtle” functionality will go under the knife, but such is the price of simplicity.
There are three types of such delusions:
Coming up with my ideal supplier, I shoveled suppliers' documentation to their API and developed 4 main methods with which I developed the API of my ideal supplier.
Generalization is the basic principle of developing any kind of functionality. Exchange with suppliers is no exception.
For example, one supplier can use order status “accepted”, “in operation”, “shipped”, “refused”. The other is “in the work of the manager”, “in the work in the warehouse”, “in the work of the transport company”, “closed successfully”, “canceled”. I also saw 27 different statuses with one supplier. Seriously, I specifically counted them. It is clear that in our system we do not need this whole zoo. We will summarize the statuses to the minimum set we need: “new”, “in the supplier’s work”, “shipped by the supplier”, “problem”.
Next, it remains to simply match each element of the directory of each supplier with the element of our directory.
You can not generalize everything. Sometimes you have to separate.
For example, if one supplier accepts an order entirely (all order lines in one XML), and the other is based on the basket principle of an online store (add goods to the basket - add goods to the basket - order the basket), then write this in a “general” way work out.
In this case, the exchange with our ideal 51st supplier should be divided. those. he must optionally be able to send the order in one and the other method. I call this the “method of sending an order” and it has two variants of meaning: “order as a whole” and “line by line”.
And there is also the setting “status verification method” and it already has three options: “request for order - return for order”, “request for order - return line by line” and “request line by line - return line by line”.
But to get involved in the separation is not worth much, because each fragmentation leads to the complexity of the system. And what exactly you should not do is to allow a tree-like division. Try to reduce everything to flat settings. If splitting functionality by one attribute will depend on crushing according to another - you will dig in the settings and instead of elegant simplicity you will get the hell of the settings.
The cut-off method is intuitive and is used in cases where the supplier’s functionality exceeds the one you need.
For example, a supplier may provide the ability to edit an order until a certain point (for example, before transferring to a set).
But this is not particularly interesting for us and all other suppliers do not have such an opportunity; accordingly, we just need to say that we will not use this opportunity.
It is necessary to clearly define the boundaries of automation. All that goes beyond them should be excluded from consideration. Ignore the arguments "this is an additional convenience": if you can do without the other suppliers without it, it means that you can.
The default method is the inverse of the clipping method. In this case, the problem is that the automation boundary includes functionality that the supplier does not support. For example, return order status. Some providers may not support this.
In this case, it is not worthwhile to count on all subsequent algorithms to check whether the order status is relevant or to ignore it, because the supplier does not know how to return it. It is much easier to put a “stub” that will translate such orders into the last successful status.
Thus, after analyzing your needs and capabilities - you can draw the boundaries of automation. This will be the end of the design phase of the API of your ideal 51st supplier. That is, if you go into the equipment, there will still be 50 suppliers in the system, but the upload / download code will be the same and will generate XML for all 50 suppliers in the format of the 51st supplier.
It so happened that in this article I drew attention to the unloading of the order, as the most variable and complex part of the exchange (at least for me), but everything written can be applied both to the download and to the unloading and order and master data and something anything else.
Next, you need to think about how best to arrange the upload file for the ideal 51st supplier:
Thus, we should have a file containing all the data the supplier needs to accept the order, but in a slightly non-valid form)
It’s time to move from theory to business, from ideal suppliers to real ones. I think all those who are interested have already guessed that the XML that we have obtained will be transformed in a tricky way so that we get 50 from one format. For this we will use a two-step transformation:
First we need to create the data structure itself. XML is best for this, because we have a powerful XML document transformation language (XSLT - eXtensible Stylesheet Language Transformations) that allows us to transform an XML document from one type to another.
The language is not complicated, it can be mastered at a basic level in a couple of days, and 99% of my XSLTs cost just two instructions, value-of and for-each.
Next, the created structure may need to be converted to a format other than XML. In this sense, the most complicated is the conversion of XML <-> JSON, I laid out an example of such a conversion. (but I will not show you, because the UFO has forbidden to insert a link to InfoStart here) There is also an XML conversion to the GET request parameters, i.e. in the line of the form? item = 12345 & qty = 4 & price = 19.50, but this is already quite simple. As a result of this step, the exchange text should appear, valid for the particular supplier.
After receiving a valid text for the exchange - it remains only to send. This task deals with the functionality that I conditionally call the universal dialer. According to its settings, it can call the SOAP or REST service and transfer the result of the converter operation to it. The dialer must provide very different scenarios for the development of events: it must be ready to send data as parameters and as the request body, POST and GET methods, with basic and not very authorization, etc.
Theoretically, a dialer can be screwed on at least read / write to Excel, select specific “connectors” to suit your needs.
In principle, the functions of obtaining data in the format of an ideal 51st supplier, XSL transformation, format converter and dialer can be transferred to a separate instance without linking this functionality with any of the existing systems, which I have done. I call this interface converter thing.
The interface converter code fits in 300 lines. The basic settings (XSLT texts, transform and call settings) are stored as data. As a result, the process of sending the order to the supplier is as follows:
All this happens on the fly, i.e. synchronously, i.e. ERP thinks that it exchanges with all suppliers in one convenient format, without even noticing that something is wrong. This approach, of course, is applicable only for cases of transfer of relatively small data packets. If you have 1 GB price lists, it is worth thinking about how to tie asynchronous to this whole story.
Almost every exchange has values that are simply hardcoded. In part, this may be due to too rich supplier functionality, for example, the parameter “percentage of which the price of goods can be increased without additional coordination”, in part, these are just some values like “for you it is always 4000”. What is 4000? Why 4000? Just 4000 and that's it. In order not to store such “dead” values in ERP, in Converter there is an opportunity to enrich ERP data with some other XML. Those. this is just arbitrary XML, which is stored in the interface converter database.
For each interface (provider) such an XML. Re-enrichment is based on the principle of simple gluing with a wrapper in a common tag, for example:
And further, such enriched XML is fed to the input of the XSLT engine, where both the data sent by ERP and our static data are available.
After all, we didn’t start making this whole garden for nothing, but because we wanted to reduce the time for connecting a supplier. This is what connecting a new supplier means:
In practice, all these actions take up to 3 hours. It is clear that there are cases when the supplier invents something “such”, but in 95% of cases it is possible to keep within 2-3 hours. (in this sentence I was mistaken and wrote “supporter” instead of “supplier.” Coincidence? ..)
Naturally, there are still a lot of details and nuances that cannot be described in the review article, but the goal of the article: to show one of the possible approaches to the design of the architecture of multi-format exchange, I consider it achieved.
Data exchange with suppliers needs to be automated. 21st century in the yard. It is clear that in an ideal world it is necessary to convene a conference of suppliers and consumers in this industry (in my case, these are auto parts), at which to agree on a single format for the exchange of information and everyone will be happy. At the pharmacy is. But we live in a non-ideal world.
The initial idea is the same for everyone: the supplier is ready to give you the elements of his directories and is ready to take an order from you. But all suppliers have their own business processes, and this imposes different initial requirements on suppliers' APIs, and different programmers who implement them bring the picture to the point that the API of different suppliers has no common features other than the initial idea.
How to write an exchange in fifty formats? Write fifty exchanges? Let's assume that in the best case an exchange with one supplier is to write and test 1 week and the programmer’s rate is 3000r / hour. Multiplied? And I did not recall the support and administration of this whole zoo.
')
There are other options? My answer: write one exchange!
We have to invent a 51st supplier and write an exchange with him. This should be the ideal supplier for us, for our processes. This supplier should have common features of all suppliers, differing from them in details.
We should start thinking not “from the supplier”, but “from us” and the picture will instantly become simpler. We have one business process and one programmer as well, and everything that goes beyond what is reasonable in our opinion is not particularly interesting to us. It is clear that such an approach will inevitably lead to “delusions” in logic and some kind of “subtle” functionality will go under the knife, but such is the price of simplicity.
There are three types of such delusions:
- The functionality we need, but which some suppliers will not be prepared to provide
- The functionality that vendors provide, but we do not need it.
- Functionality implemented by radically different methods from different suppliers.
Coming up with my ideal supplier, I shoveled suppliers' documentation to their API and developed 4 main methods with which I developed the API of my ideal supplier.
Method 1: Generalization
Generalization is the basic principle of developing any kind of functionality. Exchange with suppliers is no exception.
For example, one supplier can use order status “accepted”, “in operation”, “shipped”, “refused”. The other is “in the work of the manager”, “in the work in the warehouse”, “in the work of the transport company”, “closed successfully”, “canceled”. I also saw 27 different statuses with one supplier. Seriously, I specifically counted them. It is clear that in our system we do not need this whole zoo. We will summarize the statuses to the minimum set we need: “new”, “in the supplier’s work”, “shipped by the supplier”, “problem”.
Next, it remains to simply match each element of the directory of each supplier with the element of our directory.
Method 2: Separation
You can not generalize everything. Sometimes you have to separate.
For example, if one supplier accepts an order entirely (all order lines in one XML), and the other is based on the basket principle of an online store (add goods to the basket - add goods to the basket - order the basket), then write this in a “general” way work out.
In this case, the exchange with our ideal 51st supplier should be divided. those. he must optionally be able to send the order in one and the other method. I call this the “method of sending an order” and it has two variants of meaning: “order as a whole” and “line by line”.
And there is also the setting “status verification method” and it already has three options: “request for order - return for order”, “request for order - return line by line” and “request line by line - return line by line”.
But to get involved in the separation is not worth much, because each fragmentation leads to the complexity of the system. And what exactly you should not do is to allow a tree-like division. Try to reduce everything to flat settings. If splitting functionality by one attribute will depend on crushing according to another - you will dig in the settings and instead of elegant simplicity you will get the hell of the settings.
Method 3: Clipping
The cut-off method is intuitive and is used in cases where the supplier’s functionality exceeds the one you need.
For example, a supplier may provide the ability to edit an order until a certain point (for example, before transferring to a set).
But this is not particularly interesting for us and all other suppliers do not have such an opportunity; accordingly, we just need to say that we will not use this opportunity.
It is necessary to clearly define the boundaries of automation. All that goes beyond them should be excluded from consideration. Ignore the arguments "this is an additional convenience": if you can do without the other suppliers without it, it means that you can.
Method 4: Default
The default method is the inverse of the clipping method. In this case, the problem is that the automation boundary includes functionality that the supplier does not support. For example, return order status. Some providers may not support this.
In this case, it is not worthwhile to count on all subsequent algorithms to check whether the order status is relevant or to ignore it, because the supplier does not know how to return it. It is much easier to put a “stub” that will translate such orders into the last successful status.
Thus, after analyzing your needs and capabilities - you can draw the boundaries of automation. This will be the end of the design phase of the API of your ideal 51st supplier. That is, if you go into the equipment, there will still be 50 suppliers in the system, but the upload / download code will be the same and will generate XML for all 50 suppliers in the format of the 51st supplier.
It so happened that in this article I drew attention to the unloading of the order, as the most variable and complex part of the exchange (at least for me), but everything written can be applied both to the download and to the unloading and order and master data and something anything else.
Next, you need to think about how best to arrange the upload file for the ideal 51st supplier:
- First, it must be XML (why it will become clear later).
- Secondly, it should contain the maximum amount of data you can upload with this order. Those. At this stage, it does not matter to us whether it is necessary to indicate the VAT rate for the supplier who ultimately intended the order - we must unload everything that may be required.
- Third, convertible values must be converted. Those. already at the stage of XML formation for an ideal 51st supplier, for example, product identifiers should be substituted from the supplier to whom the order is intended. This should not cause any difficulties, since all the data for this is in the order.
Thus, we should have a file containing all the data the supplier needs to accept the order, but in a slightly non-valid form)
So how does it work?
It’s time to move from theory to business, from ideal suppliers to real ones. I think all those who are interested have already guessed that the XML that we have obtained will be transformed in a tricky way so that we get 50 from one format. For this we will use a two-step transformation:
- Xslt
- Format Converter
First we need to create the data structure itself. XML is best for this, because we have a powerful XML document transformation language (XSLT - eXtensible Stylesheet Language Transformations) that allows us to transform an XML document from one type to another.
The language is not complicated, it can be mastered at a basic level in a couple of days, and 99% of my XSLTs cost just two instructions, value-of and for-each.
Next, the created structure may need to be converted to a format other than XML. In this sense, the most complicated is the conversion of XML <-> JSON, I laid out an example of such a conversion. (but I will not show you, because the UFO has forbidden to insert a link to InfoStart here) There is also an XML conversion to the GET request parameters, i.e. in the line of the form? item = 12345 & qty = 4 & price = 19.50, but this is already quite simple. As a result of this step, the exchange text should appear, valid for the particular supplier.
After receiving a valid text for the exchange - it remains only to send. This task deals with the functionality that I conditionally call the universal dialer. According to its settings, it can call the SOAP or REST service and transfer the result of the converter operation to it. The dialer must provide very different scenarios for the development of events: it must be ready to send data as parameters and as the request body, POST and GET methods, with basic and not very authorization, etc.
Theoretically, a dialer can be screwed on at least read / write to Excel, select specific “connectors” to suit your needs.
In principle, the functions of obtaining data in the format of an ideal 51st supplier, XSL transformation, format converter and dialer can be transferred to a separate instance without linking this functionality with any of the existing systems, which I have done. I call this interface converter thing.
The interface converter code fits in 300 lines. The basic settings (XSLT texts, transform and call settings) are stored as data. As a result, the process of sending the order to the supplier is as follows:
- ERP forms the order and sends it in the format of the 51st supplier to the Interface Converter
- The converter receives an XML test in the format of the 51st supplier and enriches it
- determines which vendor needs to send a request (conditionally, by interface code) - performs XSL transformation and format conversion on the fly
- immediately sends the converted request further to the supplier
- receives a response from the supplier
- from the body of the answer, in the reverse order, performs format conversion, further enrichment, then XSL transformation into the response format of the ideal supplier
- returns the ideal provider response to ERP
All this happens on the fly, i.e. synchronously, i.e. ERP thinks that it exchanges with all suppliers in one convenient format, without even noticing that something is wrong. This approach, of course, is applicable only for cases of transfer of relatively small data packets. If you have 1 GB price lists, it is worth thinking about how to tie asynchronous to this whole story.
Chip with re-enrichment
Almost every exchange has values that are simply hardcoded. In part, this may be due to too rich supplier functionality, for example, the parameter “percentage of which the price of goods can be increased without additional coordination”, in part, these are just some values like “for you it is always 4000”. What is 4000? Why 4000? Just 4000 and that's it. In order not to store such “dead” values in ERP, in Converter there is an opportunity to enrich ERP data with some other XML. Those. this is just arbitrary XML, which is stored in the interface converter database.
For each interface (provider) such an XML. Re-enrichment is based on the principle of simple gluing with a wrapper in a common tag, for example:
<body> <ERPData> ...<i> ERP</i>... </ERPData> <AdditionalData> ...<i> </i>... </AdditionalData> </body> And further, such enriched XML is fed to the input of the XSLT engine, where both the data sent by ERP and our static data are available.
What do we have as a result?
After all, we didn’t start making this whole garden for nothing, but because we wanted to reduce the time for connecting a supplier. This is what connecting a new supplier means:
- Studying API documentation
- Creating an appropriate supplier in ERP and setting it up (about 10 checkboxes).
- XSLT Interchange Text Conversion Description
- Format conversion description (if required)
- Configure outgoing connection settings (another 10 checkboxes).
In practice, all these actions take up to 3 hours. It is clear that there are cases when the supplier invents something “such”, but in 95% of cases it is possible to keep within 2-3 hours. (in this sentence I was mistaken and wrote “supporter” instead of “supplier.” Coincidence? ..)
Naturally, there are still a lot of details and nuances that cannot be described in the review article, but the goal of the article: to show one of the possible approaches to the design of the architecture of multi-format exchange, I consider it achieved.
Source: https://habr.com/ru/post/353432/
All Articles