ISPsystem, forgive and forgive! Why and how we wrote our server control panel

Hello! We "Hosting Technology" and 5 years ago launched VDSina - the first vds hosting created specifically for developers. We strive to make it as convenient as DigitalOcean, but with Russian support, payment methods and servers in Russia. But DigitalOcean is not only reliability and price, it is also a service.

The software from ISPsystem turned out to be a rope that tied our hands on the way to the steep service. Three years ago, we used Billmanager billing and the VMmanager server control panel and quickly realized that it was almost impossible to provide good service without our panel.

')

How ISPsystem killed convenience

Bugs

We could not fix the bug ourselves - every time we had to write to someone else’s support and wait. Solving any problem required a reaction from a third-party company.

ISPsystem support responded normally, but fixes only came through a few releases, and that was not always and not all. Sometimes critical bugs were corrected for several weeks. We had to reassure customers, apologize, and wait for the ISPsystem to fix the bug.

Downtime threat

Updates could generate unpredictable downtime that provoked new bugs.

Each update was a lottery: it was necessary to cover the billing and sacrifice updates to the gods - a couple of times the update caused downtime for 10-15 minutes. Our admins at this time are saddles in front of us - we never knew how long downtime would last and could not predict when ISPsystem would decide to release a new update.

In the fifth generation, Billmanager became better, but in order to gain access to the necessary features, it was necessary to bet, which was updated every week. If something broke, I had to give access to foreign developers so that they would fix something.

Inconvenient panel interface

Everything was divided into different panels and controlled from different places. For example, customers paid through Billmanager, and they had to reboot or reinstall VDS in VMManager. Our employees also had to switch between windows to help the client, check the load on his server, or see which OS he uses.

This interface takes time - and ours, and customers. Neither of which convenience, like that of DigitalOcean, is out of the question.

Short life cycles with frequent API updates

We wrote our own plugins - for example, a plugin with additional payment methods that are not available in VMManager.

In recent years, VMManager has had a relatively short life cycle, and in new versions, the names of variables or functions in the API could be arbitrarily changed - it broke our plugins. Support for old versions was quickly folded and had to be updated.

You can not modify

More precisely it is possible, but extremely inefficient. License restrictions do not allow to make changes to the source, you can only write plugins. Maximum plugins - some menu items, step by step wizard. ISPsystem is sharpened for versatility, and we needed specialized solutions.

So the decision to write a panel has matured. We set goals:

• Quickly respond to errors, bugs and be able to fix them yourself, without forcing the client to wait.
• Freely modify the interface for workflows and customer needs.
• Enhance usability with a clean and clear design.

And they began to develop.

The architecture of the new panel

We have a self-sufficient development team, so we wrote the panel ourselves.
The main work was done by three engineers - technical director Sergey invented the architecture and wrote the server agent, Alexei made the billing, and the front-end was assembled by our front-end Artysh.

Step 1. Server Agent

The server agent is a python web server that manages the libvirt library, which in turn manages the Qemu-kvm hypervisor .

The agent manages all services on the server: creating, stopping, deleting vds, installing operating systems, changing parameters, and so on through the libvirt library. At the time of publication of the article it is more than forty different functions, which we supplement depending on the task and needs of the client.

In theory, libvirt could be managed directly from billing, but this required too much additional code and we decided to spread these functions between the agent and billing - the billing simply makes requests to the agent via the JSON API.

The agent is the first thing we did because it did not require any interface and it could be tested directly from the server console.

What the server agent has given us: there appeared an interlayer that makes life easier for everyone - billing does not need to send a whole bunch of commands, but just make a request. And the agent will do everything that is needed: for example, allocate disk space and RAM.

Step 2. Billing

For our developer Alex, this was no longer the first control panel - Alex was in the hosting for a long time, so he generally understood what the client needs and what the hoster needs.

Billing is what we call the “control panel” between us: it’s not only money and services, but also their management, customer support and much more.

To switch from ISPSystem software, it was necessary to fully retain previous functionality for customers, transfer all financial actions of users from the old billing system to the new one, as well as all the services and connections between them. We studied what is in the current product, then the decisions of competitors, mainly DO and Vultr. We looked at the advantages and disadvantages, collected feedback from people who worked with old products from ISPsystem.

The new billing used two stacks: classic PHP, MySQL (and in the future we plan to switch to PostgreSQL), Yii2 as a backend framework and VueJS at the front. Stacks work independently of each other, are developed by different people, and communicate using the JSON API. For development, then and now we use PHPStorm and WebStorm from JetBrains and love them dearly (guys, hello!)

The panel is designed according to the modular principle: payment system modules, domain registrar module or, for example, the SSL certificate module. You can easily add a new function or remove the old one. The reserve for expansion is laid architecturally, including in the opposite direction, “to the gland”.

What we got : a control panel over which we have complete control. Now, bugs are corrected for hours, not weeks, and new functions are implemented at the request of customers, and not at the request of the ISPSystem.

Step 3. Interface

The interface is our team brainchild.

First, we looked at what would happen if we made an add-on over the ISPsystem API, without fundamentally changing anything in the interface. It turned out so-so and we decided to do everything from scratch.

We believed that the main thing was to make the interface logical, with a clean and minimalistic design, and then we would get a beautiful panel. The arrangement of the elements was discussed in Megaplan and the interface that users see in the control panel now is gradually being born.

The first appeared billing page design, because we already made payment plugins for ISPsystem.

Frontend

The panel decided to make the SPA application - resource-undemanding and with fast data loading. Our front-end Artysh decided to write it on Vue - at that time Vue just appeared. We assumed that the framework will evolve dynamically, like React, after some time the Vue community will grow and there will be a sea of ​​libraries. We put it on Vue and did not regret it - now adding new functions to the front that have already been programmed on the backend takes a little time. We will tell you more about the front-end panel in a separate article.

Frontend connection with backend

Frontend tied to the backend through the push. I had to sweat and write my own handler, but now updating the information on the page takes place almost instantly.

What happened: the panel interface has become easier. We made it adaptive, and fast loading allows it to be used even from cell phones in the last minutes before takeoff, without installing a separate application for working with the panel.

Step 4. Testing and migration pattern

When everything started and passed the first tests, the question of migration arose. First of all, we set up the billing and started testing his work with the server agent.

Then they wrote a simple script that transfers the database from the old billing to the new one.

We had to test and recheck literally everything, since the data was merged into one new database of the three old ones: Billmanager, VMmanager and IPmanager manager. Perhaps test migration is the most difficult thing that we encountered in the process of developing a new panel.

After rechecking, we covered the old billing. The final data migration was a very disturbing moment, but, thank God, it was completed in a few minutes and without noticeable problems. There were minor bugs that we repaired during the week. Most of the time it took to test what happened.

Then we sent letters to customers with the address of the new panel and billing and made a redirect.

In summary: IT'S ALIVE!

Happy end

From the first hours of our software, we felt all the delights of the transition. The code was completely ours and with a convenient architecture, and the interface is clean and logical.

The first review after the launch of the new panel

We launched the transition process in December, on the eve of the New Year 2017, when the load was the least to make the transition easier for customers - almost no one works on the eve of holidays.

The main thing that we got during the transition to our system (except for general reliability and convenience) is the ability to quickly add functionality under the key customers - to be a person to them, not an ass.

What's next?

We grow, the amount of data, customers, customer data. On the backend, I had to add a Memcached server and two queue managers with different tasks. On the frontend there is caching and its queues.

Of course, we still had adventures as the product was developed and complicated, for example, when we added HighLoad.

In the next article we will describe how the Hi-CPU tariff was launched: about hardware, software, what tasks we solved and what we did.