How to minimize errors when integrating with external services: the online broker experience
For a year and a half, we integrated via API with twenty external services. The first five integrations went through pain and tears - we made all the possible mistakes. We rewrote the code several times, parted with partners just before the release, because we could not agree on the modifications. Lost time and money.
But with each new integration, we came up with solutions for recurring problems. As a result, we did the last integration four times faster than the first. What now we do differently - read the article.
So that you better understand the specifics of our integrations, we will briefly describe what we are doing. We are developing an online broker. Principle of work: a user comes to the site, fills in a questionnaire, we transfer it to microfinance organizations (MFIs) and receive approval or refusal of a loan from them. The user selects the right offer and gets paid. To make it all work online, we integrate with the IFIs through an API.
Let us examine two scenarios: when a potential partner already has an API for receiving applications, and when not. Both scenarios suggest that the partner wants to integrate with us and is willing to spend time on it.
')
Previously, we explained to a partner verbally or in correspondence what we need for integration, and the partner based on these explanations made an API for receiving applications from Finspin. We did not agree on the requirements for models, objects, fields of the questionnaire and the rules for their validation. Quickly described the assignment of methods and possible answers. The result turned out to be infinitely far from the expected, because our understanding of the API strongly diverged from the partner. We had to redo everything.
Now We wrote our specification - a YAML file, which can be opened in Swagger. In the specification, we described the most appropriate API for integrating Finspin with MFIs: questionnaire fields and rules for their validation, formats and types of requests with answers, method names, possible errors and statuses. We recorded the status of the application, which we plan to receive, for example: “accepted for processing”, “scoring is in progress”, “denied a loan”, “approval”, etc.
We send the specification to a partner, and he decides whether he can provide such an API. If it cannot, marks problem points in the specification, for example, not all the application states are ready to be passed to the API or there are not enough fields in the questionnaire. And we are already discussing specific points, not abstract entities. The partner does not spend time writing their documentation, but corrects ours.
We spent on the creation of the specification for two days, but now we save dozens of hours on approvals and API modifications. Also, using the specification, we quickly screen out partners who are not ready for integration. In the early stages, it becomes clear that our online application processing is very different: it is not profitable to integrate.
Previously, we received a partner specification and asked questions about it on the go. Often forgot to clarify something important, and then this important surfaced in the final stages of development and testing, when correcting and rewriting became particularly costly. Once, at the end of integration with one partner, it turned out that it would transfer API loan statuses with a delay of several days. For us, this is unacceptable. We had to abandon the partnership.
Now We wrote a checklist to evaluate the API and the processes it serves. The checklist collected questions that need to get answers. First, our manager looks for answers in the specification. If not found, addresses questions to the partner.
The checklist helps to find bottlenecks before starting development, and not after - when you have to edit the code, and customers suffer because of poor service.
We constantly replenish the checklist when faced with new situations. The more detailed the checklist, the lower the risk of missing mistakes in the development.
API checklist fragment
Previously, it seemed to us that in the field of online lending, everybody understands the proftermines equally, there can be no different interpretations. Practice has shown that we were wrong.
For example, with one MFI, we interpreted primary and repeat clients differently. For us, the primary client is a client, whose profile first got into the MFI database through Finspin, and the repeat client was already in the database. The partner considered repeated clients by the number of loans issued: the repeated ones received two loans and came for the third. Such terminological confusion could lead to inconsistencies in financial reconciliations.
Now we use a small glossary of terms, which we check with partners. As a rule, it is enough to clarify five or six basic terms in order to synchronize ideas about integration and assess the prospects for collaboration.
Once a glossary of terms helped us avoid unprofitable integration. Our interpretation of "approval" was very different from the affiliate. The partner's “approval” included many different aspects that require manual processing. We strive for maximum automation, so we immediately refused to cooperate.
We clarify the term “Approval”
There used to be cases when we started working with a potential partner with a specification. Our manager received the specification, carefully studied it, clarified with the partner the details for integration, discussed controversial issues with the developers. And then a message arrived from the management: “End of release, integration is canceled, we did not agree on the terms”. The staff wasted time.
When you change your mind, and the work is done
The iron rule now applies: the manager proceeds to study the specification only when he receives an unambiguous integration decision from management.
Previously, the first call to the partner API occurred during testing of our application, from dev servers. Often the first requests received errors at the connection setup stage: can not connect to server or just a timeout. The most popular reasons are:
It took hours to correct such minor mistakes, because they followed one after another: until you correct one, you will not see the next one.
Now , before we take the task into the development plan, we clarify all the features of accessing the API using a small checklist. The checklist lists the points that need to be clarified, for example:
If there is a test API, we will definitely find out what the difference is when accessing the combat server and the test server. We take into account the differences when building requests from our application to partners in the dev and prod environments. Such measures help us to understand whether the systems are ready for our requests or if we need additional settings.
Previously, we immediately wrote the code according to the partner’s specification, made up the TZ, revised it, tested it. And at the stage of integration tests, it turned out that not everything written in the specification corresponds to reality - from the format of requests to the process of passing an application.
For example, according to the spec, when accessing a method, we expect a line in the answer in a certain format, hang processing for the validity of the answer, exit the tests, and suddenly we get an array in the answer. We find out the reasons from the developers of the partner. They refer to outdated documentation. Again a circle of improvements, reviews and tests.
Now , as soon as we receive the documentation and connection details, we run the process through the API client. As a rule, the tester loads the specification in Postman and simulates the complete business process of sending a loan application, checks the most popular cases with different data sets for the request. That is, manually doing what the application will do later.
At the manual run stage, 80% of inaccuracies in the documentation are revealed. We describe the errors and pass on to the partner. The partner either eliminates inaccuracies in itself, or modifies the specification. As a result, by the time the code starts working, developers will receive working documentation.
The most popular discrepancies are:
Write a specification for integration with your service. We spent on the development of specifications in YAML for two days, but now we save dozens of hours on modifications and approvals.
If the partner submits its specification, check it on the checklist. In the checklist, collect questions that need to get answers. Do not rely on experience and memory, anyway miss something.
Get a glossary of terms to avoid discrepancies with your partner. Our experience has shown that the differences can even be in obvious terms.
Do not take the task into development until you receive from the management a principle approval of integration with a partner. The idea is obvious, but it helps us to avoid wasting work.
Get a checklist to clarify the peculiarities of accessing the partner's API: connection details, white-list, SSL certificate validation, traffic encryption requirements, etc. Check the checklist as early as possible in order to avoid slipping during the final stages.
Received the specification - do not rush to write the code immediately. First, manually run the process through the API client, for example through Postman. So you will find errors in the specification at an early stage and with small resources. And they will.
But with each new integration, we came up with solutions for recurring problems. As a result, we did the last integration four times faster than the first. What now we do differently - read the article.
So that you better understand the specifics of our integrations, we will briefly describe what we are doing. We are developing an online broker. Principle of work: a user comes to the site, fills in a questionnaire, we transfer it to microfinance organizations (MFIs) and receive approval or refusal of a loan from them. The user selects the right offer and gets paid. To make it all work online, we integrate with the IFIs through an API.
Assessment of the partner’s readiness for integration using checklist and specification
Let us examine two scenarios: when a potential partner already has an API for receiving applications, and when not. Both scenarios suggest that the partner wants to integrate with us and is willing to spend time on it.
')
Partner has no API - send specification
Previously, we explained to a partner verbally or in correspondence what we need for integration, and the partner based on these explanations made an API for receiving applications from Finspin. We did not agree on the requirements for models, objects, fields of the questionnaire and the rules for their validation. Quickly described the assignment of methods and possible answers. The result turned out to be infinitely far from the expected, because our understanding of the API strongly diverged from the partner. We had to redo everything.
Now We wrote our specification - a YAML file, which can be opened in Swagger. In the specification, we described the most appropriate API for integrating Finspin with MFIs: questionnaire fields and rules for their validation, formats and types of requests with answers, method names, possible errors and statuses. We recorded the status of the application, which we plan to receive, for example: “accepted for processing”, “scoring is in progress”, “denied a loan”, “approval”, etc.
We send the specification to a partner, and he decides whether he can provide such an API. If it cannot, marks problem points in the specification, for example, not all the application states are ready to be passed to the API or there are not enough fields in the questionnaire. And we are already discussing specific points, not abstract entities. The partner does not spend time writing their documentation, but corrects ours.
We spent on the creation of the specification for two days, but now we save dozens of hours on approvals and API modifications. Also, using the specification, we quickly screen out partners who are not ready for integration. In the early stages, it becomes clear that our online application processing is very different: it is not profitable to integrate.
The partner has an API - we run on the checklist
Previously, we received a partner specification and asked questions about it on the go. Often forgot to clarify something important, and then this important surfaced in the final stages of development and testing, when correcting and rewriting became particularly costly. Once, at the end of integration with one partner, it turned out that it would transfer API loan statuses with a delay of several days. For us, this is unacceptable. We had to abandon the partnership.
Now We wrote a checklist to evaluate the API and the processes it serves. The checklist collected questions that need to get answers. First, our manager looks for answers in the specification. If not found, addresses questions to the partner.
The checklist helps to find bottlenecks before starting development, and not after - when you have to edit the code, and customers suffer because of poor service.
We constantly replenish the checklist when faced with new situations. The more detailed the checklist, the lower the risk of missing mistakes in the development.
API checklist fragment
Glossary of terms
Previously, it seemed to us that in the field of online lending, everybody understands the proftermines equally, there can be no different interpretations. Practice has shown that we were wrong.
For example, with one MFI, we interpreted primary and repeat clients differently. For us, the primary client is a client, whose profile first got into the MFI database through Finspin, and the repeat client was already in the database. The partner considered repeated clients by the number of loans issued: the repeated ones received two loans and came for the third. Such terminological confusion could lead to inconsistencies in financial reconciliations.
Now we use a small glossary of terms, which we check with partners. As a rule, it is enough to clarify five or six basic terms in order to synchronize ideas about integration and assess the prospects for collaboration.
Once a glossary of terms helped us avoid unprofitable integration. Our interpretation of "approval" was very different from the affiliate. The partner's “approval” included many different aspects that require manual processing. We strive for maximum automation, so we immediately refused to cooperate.
We clarify the term “Approval”
First business, then development
There used to be cases when we started working with a potential partner with a specification. Our manager received the specification, carefully studied it, clarified with the partner the details for integration, discussed controversial issues with the developers. And then a message arrived from the management: “End of release, integration is canceled, we did not agree on the terms”. The staff wasted time.
When you change your mind, and the work is done
The iron rule now applies: the manager proceeds to study the specification only when he receives an unambiguous integration decision from management.
Readiness for integration: urls, requisites, surroundings
Previously, the first call to the partner API occurred during testing of our application, from dev servers. Often the first requests received errors at the connection setup stage: can not connect to server or just a timeout. The most popular reasons are:
- wrong method names (sealed or confused),
- invalid domain
- Incorrect connection details
- did not add our servers IP to the white list.
It took hours to correct such minor mistakes, because they followed one after another: until you correct one, you will not see the next one.
Now , before we take the task into the development plan, we clarify all the features of accessing the API using a small checklist. The checklist lists the points that need to be clarified, for example:
- restrictions on the load on the service,
- the number of requests per unit time
- connection details
- white-list by ip-addresses,
- SSL certificate validation
- traffic encryption requirements
- presence of special headers in queries,
- availability of a test server with an API or the ability to send test requests to a combat server
If there is a test API, we will definitely find out what the difference is when accessing the combat server and the test server. We take into account the differences when building requests from our application to partners in the dev and prod environments. Such measures help us to understand whether the systems are ready for our requests or if we need additional settings.
Manually sending API requests
Previously, we immediately wrote the code according to the partner’s specification, made up the TZ, revised it, tested it. And at the stage of integration tests, it turned out that not everything written in the specification corresponds to reality - from the format of requests to the process of passing an application.
For example, according to the spec, when accessing a method, we expect a line in the answer in a certain format, hang processing for the validity of the answer, exit the tests, and suddenly we get an array in the answer. We find out the reasons from the developers of the partner. They refer to outdated documentation. Again a circle of improvements, reviews and tests.
Now , as soon as we receive the documentation and connection details, we run the process through the API client. As a rule, the tester loads the specification in Postman and simulates the complete business process of sending a loan application, checks the most popular cases with different data sets for the request. That is, manually doing what the application will do later.
At the manual run stage, 80% of inaccuracies in the documentation are revealed. We describe the errors and pass on to the partner. The partner either eliminates inaccuracies in itself, or modifies the specification. As a result, by the time the code starts working, developers will receive working documentation.
The most popular discrepancies are:
- wrong request format, promise to take json, form-data was needed;
- errors in the names of the fields to be transmitted in the request;
- errors in field formats;
- the field validation rules are not specified or are wrong;
- some fields may be completely forgotten;
- absent or different from the actual description of the method response format;
- erroneous marks about the obligatory fields - “asterisks” can stand where the field is in fact not obligatory, and vice versa;
- often, all states and statuses in which an object may be located are not documented;
- discrepancies between expected and received states of objects. For example, at some point we are waiting for the application to be in the X state - and, in fact, according to the API, we get Y.
The recipe for happiness: how to avoid errors when integrating API
Write a specification for integration with your service. We spent on the development of specifications in YAML for two days, but now we save dozens of hours on modifications and approvals.
If the partner submits its specification, check it on the checklist. In the checklist, collect questions that need to get answers. Do not rely on experience and memory, anyway miss something.
Get a glossary of terms to avoid discrepancies with your partner. Our experience has shown that the differences can even be in obvious terms.
Do not take the task into development until you receive from the management a principle approval of integration with a partner. The idea is obvious, but it helps us to avoid wasting work.
Get a checklist to clarify the peculiarities of accessing the partner's API: connection details, white-list, SSL certificate validation, traffic encryption requirements, etc. Check the checklist as early as possible in order to avoid slipping during the final stages.
Received the specification - do not rush to write the code immediately. First, manually run the process through the API client, for example through Postman. So you will find errors in the specification at an early stage and with small resources. And they will.
Source: https://habr.com/ru/post/441152/
All Articles