How I managed to owe Amazon $ 12,000 in 1 day

I am the head of a small IT company from Zelenograd. We are engaged in the integration of 1C programs and telephony. The company employs a little less than 20 people, and it so happens that I myself am responsible for the entire IT infrastructure.

In principle, I love to do this and get acquainted with various new technologies. One of these technologies has become virtualization and, in particular, such an interesting service as Amazon AWS.

It is often necessary to quickly deploy multiple virtual machines with white IP addresses for laboratory purposes, use them for a couple of days, and nail them without regret. In Amazon terminology, this service is called EC2, and allows you to perform such manipulations in minutes. It is very convenient and worth a penny, because Payment is according to the hourly rate.

So, back in 2011, I actively studied Asterisk and used the service, at the end of the month they charged $ 20 to $ 30 from my card. Then the need for the service disappeared, I turned off all the cars, made some backups and started receiving monthly bills of $ 3.66 and somehow didn’t get to know what this amount was charged for.
')

First swallow

We started a new project and there was a need to create a fault-tolerant configuration. I, of course, remembered Amazon AWS. I went to the admin panel, looked at what appeared new, walked through the menu items, figured out what 3.66 was written off for, it was a backup of a virtual machine of 50 gigabytes in size, deleted it, and went to sleep with a sense of accomplishment.

Monday morning, a lot of work and in a hundred letters from customers and colleagues, an obscure letter from Amazon with the following content:



And I am not strong in English either, well, I think that's eccentric, I went there yesterday and pressed all the buttons there after half a year, and they are already beating the panic, okay, I'll go in the evening to continue what I started ...

In the evening

Before bed, I like to do something useful, especially from 12 to 2 nights. I opened the letter, entered the account, and I see ... that I have 20 virtual machines running, and these are quite powerful, and therefore expensive.



Heart pounding, I select them in a crowd and for a couple of seconds I try to remove, it does not work, then I just stop. Well, I think, reptiles were hacked, I got a hundred other dollars, and in fact all my cards are tied to my account.

I go to billing, and there is such a picture:



It turns out that I did not run 20 machines, but 140. On every continent , in every Amazonian data center . That's what I got!

Characteristics of rented cars:



For the next hour, I tried to stop all this, changed the login password, turned on two-factor authentication, and created a case for help desk with the question: “What should I do with this all?”

In the morning

In the morning I looked at the price for my car, well, about half the amount is. Naturally, I didn’t have any desire to part with such a huge amount, as of today. Case continued to hang in the status of not considered, but I had another "fun" story, about how to get on $ 12000, and I have a rather careful attitude to security issues and the question: "How did I get this password?" Continued me torment

I tried to untie my cards from Amazon billing, to which he replied with a refusal and a request to go to the Amazon.com site and manipulate my account there.

Amazon.com met me with a couple of questions about who my first leader is and what is the name of a childhood friend, and allowed me to untie my credit card. Half done.

Then the bell rings on the cell, with clearly not Russian country code. Since I do not speak any languages ​​except 1C, C ++, C # and PHP, I didn’t pick up the phone. Fortunately, after 15 minutes I saw the first answer of the support.

A small conversation ensued with the help of a colleague and Google translator. Support understood the situation and began to give me assignments. Change passwords, delete all virtual machines in all data centers, remove all bids at auctions for the purchase of computing power. And since I didn't have anything in my account at all, I politely asked me to delete everything centrally, which I also politely refused.

Each virtual machine was protected from deletion, I had to go into the settings and remove a special check mark. This entertainment took me another hour and a half. When I deleted the machines, I noticed that they all had the same time to create and launch them, so they were created by some kind of script.

After removing all 140 cars, I decided to go over all the tabs and check if everything is ok. And what do I see? Right before my eyes, new cars appear, cluster together and start. And this is after the change of all passwords and the inclusion of two-factor authentication ...

I turn off the machines, read attentively the support task again, and I see a line about the rotation of the API keys. Here's a pancake, I find them in the depths of the authorization system settings, I look at the date - 2011, and without regret I delete nafig, I don’t even remember, I myself created them or received a load during registration.

After that, I still have about 40 minutes to remove the new 60 cars.

I inform the support of the quest, to which I get an answer that my question will be submitted to the financial group, the amount is not small and do not expect a quick answer.

Here is an extract from my account:

Call from Luxembourg

After some time, the bell rings again. Again, I do not take, to no purpose to moo something into the phone, not understanding the issues. In the mail there is a letter, in Russian, from some dude who is responsible for the European segment, with the question of when you can call and talk. I was delighted, finally it is possible to communicate with a Russian-speaking person and find out prospects. Please call me back.

The dude seems to be a sales manager, wondering what my project is, which required so much power and whether I need a discount or some kind of help. I explained the situation, said that I had been hacked, gave the case number. The person at that end reacted with understanding to the problem, sympathized and said to contact for any questions. It's nice when the company has a human face.

To be continued...

From the useful, I have already included two-factor authorization wherever I could. I tortured my MacBook with various antiviruses that did not find anything.

It took about 5 days, while there is no news, the amount of debt hangs behind me, April 4 is a settlement day ...

upd. There is a suspicion that the hacking occurred after the attack on the Bitrix site, in which the S3 storage was connected and the API key was specified, because the day before the hacking, in the productive protection module it was written that 314K hacking attempts were reflected, and 1800 forms were filled out on my website feedback :(

upd2. The guys from Bitrix support conducted an analysis of the site logs. An attempt to attack was, trying to find passwords to the admin panel, use SQL injections and XSS vulnerabilities, but there is no confirmation of hacking.
More API key is not used anywhere ... Amazon is silent.

upd3. The issue is resolved, as a one-time action I agreed to write off the debt. Respect guys from Amazon :)


I asked them logs with IP addresses. I want to understand all the same, hijacked the password or API keys. I have static IP addresses both at home and at work, and I can set up the first unauthorized entry by logs.