Billing in a big project

There are different ways to monetize a project. But they have one common component - how the money goes from the user's wallet to the organization’s account. Today we will talk about how payment acceptance is organized at Badoo and what can be found on the payment gateway market. We immediately warn you that in the article you will not find specific figures on the turnover of the company's funds, but everything else will be no less interesting.

What is "billing"

For us, billing is all about getting money from users: the configuration of prices, the payment acceptance page, the direct receipt and processing of payments, the provision of paid services, various promotions and, of course, the monitoring of everything described above.

Initially, as in all startups, we did not have paid services. The first serious steps towards monetization began back in 2008, despite the fact that the site was officially launched in 2006. France was chosen for experiments, and payment was accepted only via SMS. The acceptance of payments was organized on files. Each request was recorded in a separate file, which was then transferred by bash scripts from one folder to another, which meant a change in processing status. The database was used only to account for successfully processed transactions. This scheme successfully worked for a little over a year, after which it became difficult to maintain, and we decided to abandon the files and rewrite everything using the database.

The development of the new version was quite fast, as there were not many countries where paid services were available. But it was designed only for receiving payments via SMS, because of this we even still have some funny artifacts, for example, the MSISDN (phone number) and short code (short number sent to a paid SMS) fields in the processed table payments.
')
Now we accept payments almost all over the world. Every second, users are trying to pay something on the website or in applications for all popular mobile platforms. And if you put it on the map, you get the picture "View of Earth from space at night":



We have about 50 payment methods provided by different partners. The most popular are bank cards, SMS & Direct billing and purchases in mobile applications.



Among them there are also exotic, for example, direct debiting from the account of an Internet provider or payment via a landline. And once we received a payment through regular mail!

Bank charges

All payment systems allow you to accept payments from their users. Such direct integration is convenient to do, as long as there are not many of them and you connect well-known systems with debugged processes. But when you need to enter local markets, problems begin to appear. Supporting the “zoo” of various APIs is becoming increasingly difficult, the requirements of regulators differ, a popular local payment system may refuse to work with foreign clients at low speeds at all, or signing a contract and settling legal problems may drag on for a long time. Despite such difficulties, local payment systems may pleasantly surprise you with their conversion. For example, Holland, which we thought was not very promising, after connecting a popular payment method in this country, iDeal began to bring in 30-40% more money.

If there is a demand, there will be an offer. There are many aggregator companies on the market, or, in a different way, payment gateways ( English payment gateway), the purpose of which is to unite all popular payment systems, including local ones, under a single API. By making one such integration, we are able to accept payments through dozens of payment systems worldwide. You can not even make the payment page on your site, but use the already finished one provided by the aggregator, and only customize the design. Particularly advanced companies make it possible to upload their CSS and JS files, change images, translations and even register the resulting page on your subdomain, for example, payments.example.com. This gives a very rich opportunity for "customization", and the user gets the feeling that he does not leave your site. We do not use this opportunity in our country, as we work simultaneously with several aggregators, but for some it may be a very convenient solution.

Which way to use — direct integration or through an aggregator — depends primarily on the size of the commission. The more your customers use the payment system, the more profitable it may be to save on commission and connect to it directly. The second important factor is the quality of the API, the usability and stability. Here aggregators allow smoothing the roughness, and sometimes provide a more stable service than the direct connection.

SMS payments

Independently are payments via SMS and direct debit from the balance of a mobile phone. They are under very tight control in many countries, especially in Europe. Local regulators or the state itself may impose special requirements on how the payment page should look or what the text of sent SMS messages should be. Changes to such requirements should be monitored and timely made changes on your site. For example, in Belgium there is a rule that a short number should be written in white font on a black background, and its value should be indicated next to it.



Different type of SMS-billing - MO (Mobile Originated) or MT (Mobile Terminated). With MO-billing, everything is quite simple: the user sent an SMS to a short number, we received money. But for MT there are several options. Payment does not occur at the time the user sends an SMS, but after he receives a special paid SMS from us. That is, the fact of payment is the notification received from the aggregator that a paid SMS has been successfully delivered. The main goal of this approach is to add an additional check before sending a paid SMS message to the user and prevent errors associated with incorrect SMS text. For this, payment takes place in two phases. The first is that the user expresses a desire to pay for something, the second is a confirmation. For example, the payment process might look like this:

• we send an SMS to a short number, respond to the incoming SMS with a certain text or without it;
• send an SMS to a short number, enter the received PIN-code on the site;
• enter the phone number on the site, get the PIN code, enter it on the site.

Fortunately, there are aggregator companies on the SMS-payments market, whose services should be used. They are for a modest, and sometimes not very, fee deprive you of the "pleasure" to deal with such details. Another nice bonus is that they often take on some of the responsibilities for end-user support. Users start writing about their problems directly to the owner of the short number, i.e. aggregator.

Technical details

Badoo works on a PHP + MySQL bundle, so we use the same technologies for payment processing. The code is executed on a separate server group, separated from the common pool. Inside, we divided it into several logical subgroups: servers for processing incoming requests, servers for background operations and collecting statistics, database servers, servers for processing bank card payments. The latter are separated into a separate group, because they must comply with the PCI DSS security standard developed with the participation of Visa, MasterCard, American Express, JCB and Discover for organizations operating or storing data of bank card holders.

For payment processing, we use two database servers with MySQL from Percona, working in master-master replication. The main load goes only to one of them, the second is used for hot-swapping in the event of an accident or for replacing the main one (for the time it is serviced, for requests from a monitoring system or collecting statistics).

The whole billing system can be divided into several large parts:

• Core. This includes basic entities, such as Order , Payment , Services and the rules for their accounting and delivery, various infrastructural things.
• Plugins aggregators. All that is responsible for the communication between us and the payment system.
• Page selection and payment services.

Connecting a new payment method to the system is to create its plug-in. He is responsible for all the interactions between us and the payment gateway, which are of two kinds: when we are the initiator (pull request) and when the initiator is the aggregator (push request). For pull requests, HTTP is usually used as a protocol, pure or as a transport for JSON / XML, or SOAP. The concept of REST API, which has been popular lately, was not common for us, only for new companies in the market, for example, for the British GoCardless, or for old ones that decided to rework their API, for example, PayPal. For push-requests, SOAP is practically not used (of those who offer such a format of push-notifications, only Qiwi is easily remembered). Mainly pure HTTP is used or it is used as a transport for all the same JSON and XML.

After the implementation of the API comes the testing phase. On Habré there were already articles about how our process of development and automation looks. But for billing there are some features related mainly to the fact that we have to test not just our code, but also the interaction with aggregators. It is very convenient if they have a test environment for this, which fully emulates the actual acceptance of payments. If it is not there, we do stubs that emulate the behavior of the aggregator. This makes manual testing easier for us and allows us to write autotests that verify the entire payment process. Here is an example of what one of the stubs looks like.



After the test environment, you need to check how everything will work in life, to make a real payment. But for SMS payments, you often have to get approval from regulators or operators, and this can take several months. In order not to lay out the semi-ready code on the production servers, we came up with such a thing as External Shot. This is our usual Shot, which is the directory with the task branch and is intended for testing it on production servers, but in addition to the local domain, it has an additional external address where anyone can go and see the changes made. For security, such “shots” are not created for every task, but only in cases where it is really necessary. We give links to them to our partners, and they can check the changes made at any time of the day or night. This is especially true for countries located in another hemisphere, with which the time difference can reach 12 hours.

Support and operation

After the new integration is laid out on production servers, the stage of its operation and support begins. Technical support takes about 60-70% of our time.



This includes, firstly, the analysis of complaints from users. All simple situations are solved by the first-line support team, it also translates for us complaints from different languages ​​into English. Therefore, we get only the most complex cases that really require the attention of developers.

The second component of technical support is the correction of errors or the introduction of changes to existing integrations. Errors occur for various reasons. For example, due to inattentive reading of documentation or gaps in it. Once, instead, we even had to use chat logs with the aggregator developer, because the documentation for their new system was not yet ready. There were cases when the aggregator changed the interaction protocol or its parameters without notice. Another time, the acquiring bank disconnected our gateway, and we had to urgently redirect traffic to another location. As it turned out, it was an ancient server from the 80s, which, according to the bank, did not have to process anything at all. In general, there is no need to be bored, especially considering that every minute of idle time is a lost profit.

To solve such problems, we write detailed logs of the application. It includes not only errors, but also all interactions with aggregator systems or just important events that occur during the execution of queries. Each request has its own unique identifier, by which you can find all records associated with it and restore its processing. This is especially useful when you have to deal with errors, since which several weeks or months have already passed.

That's how billing is organized in Badoo. Of course, there are still a lot of interesting topics that we plan to tell the future about, such as monitoring, PCI DSS certification and processing of bank card payments. If you have questions or suggestions on future articles, welcome to the comments.

Anatoly Panov
Lead Developer