Technical partner problems and their warning in mini-stories

A year ago, I worked in an organization as a technical manager who, at an accelerated pace, connected partners to his system. The task from the sales department was simple and seemingly doable: connect everyone quickly. This article is written from the personal experience of connecting partners to the system. Under the cut are "funny stories", a lot of letters, technical trash and a small "check list" for the integration department.

... after the very first connections, I decided to create an integration department, which only wrote gateways and interacted with technical experts "on the other side of the barricades". The head of the department was instructed to foresee any emergency situation ...

I want to note that everything written down below is complete subjectivity and thrash of Russian specifics, the company's industry in which I worked and the behavior of individual organizations.

Somehow it all works ...

')
At first, my hair stood on end when I found out that on the other side there is no information system more advanced than the Excel table (at best). But, as it turned out, these are one of the best partners (the second category of the best partners is those who implemented our protocol). We very quickly wrote our own information system, which solved not only the problem of information exchange, but also automated the business partner to some extent, for which we received a plus in karma.

But those who were “technologically advanced” were causing most of the problems. Very few partners worked according to the specifications described by them: from insignificant but cunning rearrangements or “not documented capabilities” of the system to complete non-compliance with specifications. By the way, it was considered a fortunate coincidence when the description (at least some) was in general. Sometimes it turned out that here you have a “hole”, you want to connect - connect (the “hole” could easily turn out to be a personal account on the partner's website).

An example of one extreme: the partner gave us the protocol (SOAP). Ok, here's Java for you, here is JCP (because there was a GOST EDS) - we are working. The gateway is written, everything works on our test emulator - go to the tests with partners. And ... what would you think? Does not work! They searched for the reason for 3 (three) months (the support of the partner reacted very sluggishly and as a result, the connection problem had to be solved at the management level of the two companies). It turned out that on that side, SOAP was implemented with a “small feature”: instead

<tagName></tagName> 

need to send

 <tagName><\tagName> 

And what about JCP? It did not take, because there was no implementation on that side.

An example of another extreme: the sales department brings the protocol, ok, yes? The protocol is implemented, joint tests with a partner (!) Are passed, you need to enable production.
The partner is engaged in this by another division, which, as it turned out, is not aware that the protocol we implemented is the protocol of their company. Result: either implement another protocol or wait for n months (I don’t remember the exact figure, but something is about 10 months).

The first rule : everything that can be implemented not by specifications, will be realized not by specifications.
Conclusion : it is very important to get the test gateway partner as early as possible and test the implemented protocol as soon as its individual parts are ready. Although this, as the second story has shown, does not always help.

Was there a boy?

Was the partner able to obtain, and, most importantly, process the received data? If you think so, then it's time to look at the world in a new way.
One of the partners missed 10-20% of requests. Well, I missed it ... I accepted it, but I did not process it. This was a feature of their system and presented as a feature. I simplified the problem a bit, because there, “deferred data processing” was only for some types of requests and worked according to the mood of the remote side, but the essence did not change. I note once again that we received the response code “the request was successfully received” (the response code “the request was successfully processed” was not), but in fact nothing happened. In the protocol description, of course, this feature was not taken into account.

Another partner informed us about the receipt of a new request (XML-protocol, HTTP-transport), but was not particularly interested in accepting the request or not.
There was a funny joke when talking with the management of the partner company (tech. Director and com. Director) on the topic "as so." The fact is that on our side (in response to such an injustice) a request was implemented with a frequency of 5 minutes “Is there no news for us?” And those. the director of the partner called it a dos and blocked us with the indication “fixed on our side” (of course, we received the instruction only after contacting our partner to the support of the partner, and not upon disconnection). I connected all my “charm” and convincing arguments when communicating with a commercial director, and upon his agreement that we, for our part, are doing everything correctly (he did not know the position of those directors) demanded an e-mail with confirmation. As a result, the answers from com. director and those. The directors went to both of them to the post office asking for an agreement. An hour later we were turned on according to our scheme.

Rule two : the answer about the successful receipt of the request (and sometimes about the successful processing of the request) does not always mean that the request was indeed successfully processed.
Conclusion : it is necessary (of course, if possible) to make sure in another way about the successful execution of the request.

You are under reliable protection.

SSL certificates will come next, but I can't help but mention one funny story.
We received a protocol that implies SSL encryption. Everything works, tests passed, but ... not with that certificate.
As a result of a small study, it turned out that the remote side does not check the certificate at all.
Another partner (who worked according to our protocol) took the examples of requests too literally and sent a signature from the example during the tests (apparently, did not master the N section "Message Signature"). Naturally, with the words “nothing works,” our businessmen came running to the integration department (which at that time already existed).

The third partner gave one more surprise: once, having entered the partner’s personal account, I noticed that there were really some big total figures, and the transactions were not ours. As it turned out, under certain circumstances, the partner would send us information on the entire partner system in our personal account. They promised to fix it and that will not happen again. Yeah, right now. A month later, I came across the same problem. This is how I inadvertently received an insider by the turnover of the partner company and all its clients.

There is such a thing, 3-D Secure is called. Implemented on the side of the bank.
One day, Commerce approaches me and reports that 3-D Secure does not work (in fact, not my problem, but it so happened that if nobody knew who to approach with the problem, then they approached me). How not to work? Well, it works, only there at the very bottom of the page (scrolling somewhere two empty pages in the browser) there is a gray on white “cancel” button (which just skipped the payment without 3DS). This is not so cunning without registration and without SMS without authorization for one (rather large) bank 3-D Secure worked.

Rule three : self-implemented access restriction systems do not work.
Conclusion : it is necessary to further strengthen security measures with standard and proven means (but see rule 5).

Satellite in the Pacific.

Once I received a mobile call: “Hello, Alexey! This is a company **** (here the name of a well-known company). You left a request for our product **** ". I think that you understand that if I did leave a bid for this product, I would not have written about it here . As it turned out, two minutes earlier, and our business received a similar call.
The situation turned out to be simple: we commerce tested the newly-baked functionality of our system, the result of which was a registry intended for transfer to a partner. And the partner received this registry (manually sent by our commerce by e-mail to the manager on the other side asking to check the registry format). What happened on the remote side I do not know, but as a result, he somehow got into processing.

Another case: at a certain moment, our system was deployed simultaneously on two technological platforms (for business reasons) and worked here and there. After some time, we received complaints from some of our clients that their requests were not processed. Suddenly. I do not know how this could be, but for some reason, depending on the technology site, the request went either to the production server of the partner, or to the test. We found out the reason for a long time, until our support did not guess to traceroute from both sites and it turned out that, depending on the site, the request sent to the same IP address went to different servers (judging by the traceroute, in the case of a request from one of the sites, within the client's network, the request even passed through different routers).

Rule number four : if the request can go wrong, it goes wrong.
Conclusion : see rule 2.

Who is there?

Is it possible to trust respected and proven solutions? I think that you already understand what is not, if the setting of these decisions is in the hands of a partner. About routing the story is higher, about SSL too, but here's a couple more:

Once we were given a gateway with addresses similar to: gateXXX.comp_name.com (where XXX is not adult, but the view of the gateway). Do I have to say that after the first week of production, the DNS of the partner fell? I think it would be superfluous to say that after lifting the DNS, he gave the IP-addresses in the reverse order. But by this time, we were already “protracted” and did not trust the primary DNS servers.

SSL is a separate song.
Do you still trust certificates? Then we go to you . It is good if the partner had at least some kind of data protection out of the box (at least from the substitution in the form of a key and md5). In general, we had a whole department of information security (which worked in parallel with other security departments), which at its level pushed the idea of ​​cryptography. But very often it did not help. Some partners offered to simply transfer md5 from the message body, someone used, at best, self-signed certificates, at worst - from godaddy, but issued to another domain. About cryptography in accordance with GOST (and how I set up the equipment), I don’t want to write at all.

One thing we knew for sure: without a declaration of war , anything can happen at any time: from the expired certificate validity (we, as a rule, didn’t pay attention to it at all - only another alarm appeared in monitoring) before the encryption algorithm changed.

Rule number five : if your partner has already set up a well-known system, consider that this is a different system.
Conclusion : ingenuity and resistance to stress (both support and gateway).

(Again) Nothing works.

Oh, how often I heard this phrase from ... noooo ... not an accountant, but from techies on the other side.

Rule Six : If something “broke,” you will not know what it is.
Conclusion : full logging and convenient log view.

White and fluffy

No, I do not want to say that we were so white and fluffy that everything worked for us. Naturally, our shoals met. There were even very unpleasant ones, about which I will not write (since I recently was engaged not so much in technology as in “business”). But adherence to technology, a deep understanding of the relevant standard (as well as ours!) Protocols, responsibility, willingness to think “for everyone” - these were requirements for candidates for the position of head of the integration department.

In general, integration with on-line systems is fun, exciting and interesting. I met many good experts, saw many interesting architectures of protocols and systems. And he began to treat the photo.