Interview with Moses Uretsky, co-founder and director of Digital Ocean

How did the idea of ​​DO? There were already thousands of hosting providers on the market, not to mention such giants as Amazon, Google, Microsoft. Surely everyone said your idea would fail?

Before DO, my brother and I were engaged in hosting for many years, and at some point it became clear that everyone was moving towards the “cloud”. Many companies started much earlier than us - we ourselves then worked with various providers. All of them built their clouds as they considered necessary and correct, but it turned out somehow unreasonably difficult.
')
So we decided that we would take the clouds and do everything in our own way, create our own version, which we would like to enjoy, which probably was not very reasonable. The fact is that everyone with whom we discussed this said that it was a bad idea and we shouldn’t take it at all :).

So, perhaps, in this story there was no magic. There was a well-known area in which we have already worked a lot. We did not like the solutions on the market, and we wanted to make our own. So the idea arose to create something that we could use ourselves and see if anyone else would like to use it too.

Who were the founders except you? Have you all been programmers?

The founders were myself, my brother Ben, Jeff Kar, Alec Cartman and Mitch Weiner. We were all developers, but with different skill sets and different focus. Ben is more knowledgeable about networking, system administration and configuration management. Jeff is a backend, low-level programming specialist. I have a mix of front-end development and system administration, plus I have an artistic background, so I look after how our product looks. Mitch is our marketer, but in the past he was a lot involved in development and managed to establish a small company that produced a CRM system. Alec is very good at Rails.

At the very beginning, our team had enough to solve all the problems, but then, of course, new problems began to appear and we had to expand. Still, building a cloud is much more difficult than just writing software and deploying it. For example, much is tied to the physical infrastructure and a bunch of servers.

Initially, you wanted to take some ready-made technologies (like OpenStack)?

At the very beginning of development, we really looked at what could be found in the public domain. We chose between CloudStack, OpenStack, Eucalyptus and several more projects, but none of them caused us any enthusiasm. With any of them would have to redo everything for themselves. Still, when you create a high-tech product that focuses on developers, then all internal technologies and APIs are as important as the interface. Even a novice developer sooner or later goes beyond the standard admin panel, and then it comes down to the fact that the product is "under the hood."

Success does not come immediately

How did you get the first users? Did everything work out right away?

We then worked in coworking and just put a pack of flyers in the elevator and invited people to try out our new Digital Ocean service, get a free cloud server to the test. Of course, no one came to us, three people were registered, and that could all have ended. In general, the launch was somehow not very successful :). However, we invested so much time and effort into the DO that we decided to continue working.

We were unexpectedly presented with a chance to demonstrate the DO at the New York Tech Meetup, which is organized by Meetup.com. Among those present, there were 700–800 very technically savvy people, all of them were delighted. I remember when we conducted a demonstration, we tried to test a couple of new pieces, for example, we tried to build integration with GitHub, but everything was not quite ready, so we had to show the fake on stage and present it as a beta version.

Then, during the after-party, people approached us and asked questions, they were delighted with what we were doing. We have registered fifty people. So, our platform was used not by five, but by fifty people. Yes, it was a long way.

And you turned to investors, tried something else?

Yes, for example, we decided to try to participate in TechStar and even became finalists of New York City TechStars, but David Tish told us: “I don’t quite understand what you are doing, guys, because I am not a techie. So I'm not sure that I can help you. ” Anyway, he advised us to take part in the Boulder program (boulder.me). We turned to them, and they agreed to conduct a technical examination. But in the end, we heard the same thing that other investors had told us before: that Amazon AWS is our biggest competitor, that there will be no success, that we will fail. We replied: "Well, you" helped us very much "" - and continued to do what we did.

And then, in January, a few months later, we still started up. The response arose gigantic and almost instantly. Before that, we had five or six people connected, and suddenly everything just exploded - every day hundreds of developers registered. We realized that we need to immediately register a company, hire support for users, make sure that there are enough servers in the data center, and so on. This process, in general, spun and does not stop to this day. We very quickly increased the company's staff to a hundred people, but, in fact, we are doing everything the same as before, we just have new challenges. At the same time, you need to try not to upset customers who believed in us and loved our product. We try to do everything to repay them in the same way and continue to delight them further.

Digital ocean inside

How has your technological stack changed over time? Are you really using Go for development?

Any company goes through the same phases. They all start with the minimum product (MVP), produce a prototype, which they were able to create with a small number of people at their disposal. In such a situation, they are repelled not from normal planning, but primarily from what a particular person wants to develop and how he will write it. Because of this, you really can’t imagine when you have a problem.

In particular, in January 2013, when everything began to turn and began to grow exponentially, we had to rebuild the entire system, because it became more and more confusing. In particular, we distributed all the services so that if one of the services falls, the entire system does not crash.

In general, we killed a lot of time trying to rebuild the code and reschedule the entire architecture. And then Go into our field of vision and got Go, because it is a very fast, new language, in which there are many interesting things. I think it is very rare to have an ideal test site for such things. And when you have thousands of servers distributed all over the world, Go goes to this distributed system very naturally.

In fact, we went through all the flours of growth, our work was not limited to creating new chips and finishing the product. For example, planning looked like this: we evaluated our scale, multiplied them by ten, and already relied on this figure, working with our architecture. And almost every time we practiced this “exercise,” Go fit perfectly. In addition, people who are interested in Go are usually very good developers, and they are interested in the problems we are working on. Thanks to this, we were able to recruit a large team of excellent engineers who are ready both to support existing products and to make new ones.


Sammy Shark, service mascot

What is now “under the hood” of Digital Ocean? Frankly, I'm surprised that you are using all yours.

We really almost do not use third-party tools. The exception is some low-level tools, but they are few. Everything related to management, planning, events, account management, we do ourselves.

I do not argue, OpenStack is an interesting thing. But his openness is greatly overestimated - after all, this project is being developed by a commercial organization, and this is somehow wrong. This is not a typical open source story in which two or three people got together and wrote something in their free time, and then the community supported them, helped them, and everyone began to use the product. There are many examples of projects that are not very supported by the community and developed mainly due to companies. In almost all cases, everything ends in the same way: the developers try to please everyone at once, too many stakeholders are involved in the process, in the end - complete disorganization. We hear from all sides about the problems with OpenStack and that after deploying all the time, developers are going to catch bugs and try to make everything work normally. We believe that if we spend our resources on bug fixes, then at least be our own bugs.

What iron are you using?

Mostly Dell and SuperMicro. They supply reliable iron, so we have been working with them for several years. They treat us very well and try to support us in every way. We have to make purchases in various countries, and each new data center only adds to the difficulties with logistics, ordering servers, their delivery, assembly, and so on. Therefore, it is very convenient to have such a partner as Dell, which, as a rule, has a representative office in each country. For example, we recently opened a data center in Amsterdam, and it took 12 days to deliver a batch of servers there. Previously, it took us about two months. Therefore, we would be happy to build our own servers, but now for us the main priority is logistics.

Security and other issues

For any PaaS and hosting security is a headache. Especially when they start doing all kinds of nasty things from your power. How do you keep track of how you fight?

Security is a big problem. We communicated with different companies and startups, asked them what abuzz and fraud numbers they were observing. We talked to many large companies in New York. About 1-2% of their transactions are fraudulent. But a couple of percent is nothing, because we have to deal with 30–40%! That is, every third registered user of Digital Ocean is fake.

This is a really huge issue affecting the whole industry. If you have low-level access to the server, you can do anything: scan ports, DDoS, phishing, send spam and so on. That is, the list of possible nastiness is almost endless, but many attackers do not even understand what they are doing harm to someone. Many of them just learn to write code and understand technologies, for this they are engaged in various dubious experiments. They do not understand that their actions cause a lot of problems, they just are curious. Our job is to identify such incidents and not interfere with “law-abiding” users. Unfortunately, it is more art than science.

But you have achieved some success?

Of course, we use many different technologies. Sometimes we cooperate with other companies that specialize in solving such problems, but we do a lot of automation ourselves. However, even companies for which this is the main work, where mega-businesses and geniuses work ... even these companies will tell you: “Listen, we will gradually work on this, we will work day after day, but still we cannot guarantee you a perfect result. Once we find a solution, the attackers will figure out how to get around it. ”

But we understand this, we treat it normally.

The most unpleasant situation is when, as a result of automated tests, a normal user is marked as an intruder.

Naturally, for them this becomes a complete surprise and leads to various problems. We try to communicate with such users, explaining to them why we are doing this. Unfortunately, this, of course, spoils them the entire user experience and grieves them, and such users have every right to be angry and indignant. But, alas, if you do nothing, then there will be even more problems, in the end all users will suffer. To get around this question is simply impossible. Therefore, we must continue to work, try our best.


By the end of 2013, Digital Ocean outperformed AWS in terms of the growth in the number of web servers — according to Netcraft. Naturally, the total number of instances is not counted here.

When it comes to such large-scale projects as DO, it may be foolish to ask about the main problems that you faced. But still - what would you highlight?

I think that the main problem is not in what language to write the product, but how to design the optimal architecture. In our case, the main problems associated with the integration of distributed systems - you need to constantly optimize how data is synchronized across different parts of your architecture. And the more extensive the geography of your project, the more acute this problem is.

We had an interesting case when we opened a representative office in Singapore. Before that, we had a central place where we stored the data, since the delay between the East and West coasts of the United States and Amsterdam was the same and nothing particularly slowed down.

But when we arrived in Singapore, it became obvious that the delay to Singapore was clearly much higher than we had expected. Plus, in addition to the problem with lag, there is the problem of channel reliability - this is another task that is solved within the framework of distributed architectures.

In short, this is a very interesting set of problems.

When everything continues to grow, even the most trivial problems reveal new aspects. Collecting analytics from virtual machines on such a scale is an interesting challenge, especially when you want to receive data in real time. At a minimum, you need to make sure that all notifications are sent correctly and timely. So, even realizing the most basic things, in such scales it is necessary to understand the smallest details.

DO today

I am quite actively using the API. Over the past three years, he has changed twice. What was the problem with the original API?

There are mistakes that are somehow made; something turns out well, something worse. The first API was amazing, it was used by a huge number of people around the world. But our clients "grew up" and eventually discovered a number of restrictions. There were two main things we wanted to change.

First, we really wanted to make the API completely RESTful. The original version could not boast about it, which created certain problems with Google, writing wrappers and so on. It did not satisfy many.

Secondly, users started doing something with our API, something we would never have thought of. For example, people started writing mobile applications for working with DO. Our API did not support OAuth, so for authorization you had to copy an ID and a long key into the application - obviously not the best user experience. All this we have taken into account in the new version.

In the new version, a lot of attention is paid to integration. This made it easier to think about developing new services and intelligent applications. Now they can flexibly assign various roles and access rights. In short, we began to think of our users not as individual developers, but as a large single ecosystem.

What feature in DO is the coolest, in your opinion?

I believe that the coolest feature of any product is not what you see, but what you don’t see. In other words, the best features are those that we did not introduce. This is especially important when it comes to product oriented developers. When you do something for advanced users, there is always a desire to cram more functions and settings. But the result is an inconvenient interface and poor user experience. So I think our main advantage is the ability to find a balance.
We always think about why adding a particular feature to the product, how to get rid of it if something happens. If the product is already overloaded, we think about how to simplify it, how to take part of the load and the difficulties on ourselves, and not to force our users to deal with all this on their own.

Why is the community important?

I like the way you work with your community and users. Every time you write to the caliper and get a response essentially from a person who is clearly in the subject. Around DO already formed a serious community, how did you manage it?

Community is undoubtedly one of our priorities. We have three main principles: love, simplicity and community. Everything begins with love - we ourselves must love our product, because even if we do not love it, then why should someone else love it? You also need to love your users. When these conditions are met, you realize that you can do a lot to help people. I think we are concentrating so closely on this, because without the support that the community provides us, nothing would work out.

It's not even about the Digital Ocean community, which, of course, is just wonderful. Speaking of what, for example, without the capabilities that Linux provides today, there would be no DO. Linux provides virtual servers, without Linux we would not exist at all. We use programming languages ​​for which we do not make any deductions, we do not pay any authors, copyright holders, and so on.

It seems to me that for our generation and later it is already a way of life and a given, after all, there was simply no such thing. Without this, creating a company would be much more difficult. Now, many say that the costs of creating a company have become much smaller, and this is largely due to open source. For example, if you decide to launch your own cloud hosting today, you can either pay a lot of money or turn to open source alternatives.

And what do you, for your part, do for the community?

We sponsor several conferences to personally communicate with users. You need to always stay in touch with the community, because if you do not do this, you will very soon cease to understand what is important to them. They are wonderful, they give us so much love and support, inspire us to work on. And the fact that they are extremely honest is also wonderful. When we make mistakes, they almost immediately tell us: “Hey, you guys are messing up,” and this is very cool. After all, when you make a mistake, the best thing you can do is to admit that you made a mistake and work on correcting it. This is a very high bar, but we have to compete with companies worth billions of dollars, they force us to compete at a high level and motivate us to do this with a positive, rather than sit to one side, looking at these giants, and think: “we don’t never reach. They literally force us to stay on our toes, to keep our promises. I think we owe our success to this. If we continue in the same spirit, I believe we will achieve even more.



First published in the magazine "Hacker" from 08/2014.

Subscribe to "Hacker"

• Paper version
• Hacker on iOS / iPad
• "Hacker" on Android