In this article, I’ll explain what Amazon Web Services is, or AWS for short, what it can be used for, and I’ll provide step-by-step instructions on how to set them up right from the start. The article will focus on
Amazon Elastic Compute Cloud or EC2 and
Amazon Simple Storage Service or S3 .
Amazon CloudFront , which is a logical continuation of S3, is well discussed in the article
mish :
Configure CloudFront to work with S3 . You can familiarize yourself with the rest of the Amazon services:
Amazon SimpleDB ,
Amazon Simple Queue Service or SQS , as well as other
Amazon services . This article is a continuation of the
first and
second articles. So:
What is Amazon Web Services
This is a set of convenient services that can be used both individually and together. First of all, these are “cloud computing” (eng. Elastic Compute Cloud). What does it mean? This means that you can start in the “cloud” any number of computers of the configuration you need with the operating system you need in just a couple of minutes. It really is. The launch time of one instance (instance is called one virtual server in Amazon) usually does not exceed two minutes. Immediately after starting, the instance starts working, it has root access via SSH or Remote Desktop, if Windows is installed there. Isn't it great! From the moment of start payment for instans goes on hours. At any time you can stop the instance and the money for its use will no longer be removed. In addition to the instance time, incoming and outgoing traffic is also paid separately.
Amazon’s second great service is the Amazon Simple Storage Service or S3. This is a distributed, reliable data warehouse. In S3, you can put any file from several bytes to several gigabytes and have access to it from a regular browser by the link, i.e. via HTTP. To access S3, it is not necessary to have a running EC2 instance. For example, here is a link to a picture:
http://s3.amazonaws.com/moodbox-server-data/channel_logo/1 . In S3, you can put your entire site if it consists of static images, moreover, you can bind to a specific basket (English bucket, in S3 all your files are stored in them) the domain name of your site, after which you can access it to the usual site in the browser. Files in S3 can be closed for public access, then they can be accessed only with keys, which I will discuss below.
Of course, these are not all services, but in the article I will describe these two services as the most interesting and which we actually use in our
Risas project .
')
AWS Registration
Before using any of the Amazon services, you must create an account there and link your credit card to it. To do this, go to
the registration page , select the option "I am a new user" and create a new account. The second step is to set the billing
information . For Russia, in fact, the only payment option is a credit card. I personally used my virtual Alfa Bank MasterCard system. Virtual card allows you to pay for goods and services only through the Internet and is not suitable for conventional ATMs. Amazon did not accept the usual debit cards from me, maybe it depends on the bank.
And the final required step is to get the
access keys . You need two types of keys to work with EC2 and S3:
Access Key ID and Secret Access Key and
X.509 Certificate . Both are obtained on the specified page. Important: on the same page at the top right your
Account Number is written, like this:
8311-6132-8591 . This number is useful when creating your image.
All the above points can be done in 5-10 minutes. Then you can subscribe to the services you need. It is very simple to subscribe, for example, to subscribe to SimpleDB go to
http://aws.amazon.com/simpledb/ and on the right click on the “Sign Up for Amazon SimpleDB” button. Similarly with any other service. For the subscription to the service money is not charged, so subscribe to whatever you want. In general, Amazon’s payment policy is as follows: you only pay for what you use and pay after you provide the service, i.e. post-payment.
And now we will pass to more detailed description of services.
Amazon EC2 or Amazon Clouds
I will list the main advantages of EC2:
- starting and stopping any number of instances for a couple of minutes
- flexible choice of instance types for different tasks
- reliability: agreement on service level guarantees 99.95% availability of instances. In general, ordinary dedicated servers fall more often :)
- two geographical areas of server location: America and Europe
- Elastic IP - the IP address that is always yours, whether the instance is working or stopped
- Low enough price
Description EC2 can be divided into three parts: the choice of the instance type, the selection and configuration of the operating system image (
Amazon Machine Image ) and the start and work with the instance or several instances.
Amazon provides several types of instances divided into two categories: Standard Instances and High-CPU Instances. The minimum instance has the following characteristics: Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform and costs $ 0.10 per hour if installed on it is a Unix system and $ 0.125 in the case of Windows. Often, this “minimum” instance is enough for work. In a month it will cost you $ 74 and you will still need to pay an additional $ 0.10 per gigabyte of incoming traffic and $ 0.17 per gigabyte of outgoing traffic. Our practice shows that traffic usually does not exceed $ 15 per month. Details of the instance types and prices are listed
on the EC2 main page .
More recently, it became possible to reduce the cost of an instance when using it for a year or more. This is a reserved instances. The point is this: you pay in advance $ 325 for an annual contract or $ 500 for a 3-year contract and then at any time you launch such an instance and its cost will be $ 0.03 per hour. For comparison, a regular instance for a year will cost you $ 846, and a Reserved Instance will cost $ 325 + $ 259 = $ 584. Read more about this on
the EC2 homepage.
After selecting the instance, you need to select and configure the image of the operating system that will work there. An image is both the operating system itself and all the programs necessary for the operation (Apache, MySQL, Memcached, Erlang, etc.), as well as all the data you need can be included in the image: the contents of the site, the source of the program, and t .d There are a large number of
ready -
made images from Amazon and enthusiasts. For example, we used the non-Amazonian image of Ubuntu from
http://alestic.com/ .
There are two ways to interact with EC2:
a utility set (API) from Amazon. For this method, you also need to install the
Java Runtime Environment . The next required step, which is necessary for further work with the instance via
Putty, is to create the
c: \ ec2 directory. The set of utilities from Amazon is unpacked into this folder in the
bin and
lib folders. In the same folder, an
ec2.bat file is
created with the following contents:
@echo off
set EC2_HOME=c:\ec2
set PATH=%PATH%;%EC2_HOME%\bin
set EC2_PRIVATE_KEY=%EC2_HOME%\PrivateKey.pem
set EC2_CERT=%EC2_HOME%\509Certificate.pem
set JAVA_HOME=C:\Program Files\Java\jre1.x.0_xx
PrivateKey.pem and
509Certificate.pem you received earlier when getting access keys. They also need to be put in the folder
with: \ ec2 . In fact, it is better to register all these variables in the Windows environment variables so that you are always at hand.
Next, we generate a pair of session keys to access the instance via ssh by running the command line:
ec2-add-keypair gsg-keypair
gsg-keypair - the actual name of the key pair, you will need to come up with your own
on the screen we get a private key of the form:
----BEGIN RSA PRIVATE KEY----
MIIEpAIBAAKCAQEAwCh8PpHU8ceulZVD7HOIl2U+KW5osbPkfZ+qQH+AJyiz+VMjkfJK2JY7qmO+
... skipped...
P2RGGwVzpMl1/QCPO29jT80pZRA9YN/3vkp9ntCtlDnH5YLnrQOKMsfcS43yvay+pdjnA00Kvbru
zsX7a7uic2rGpSXW+/8K0AvxuOwjq/zxgi6Vuqg2QqvhxnBQenayCgBMVyjRNty2Hn2EKA==
----END RSA PRIVATE KEY----
copy it (from the screen as it is) and save it to a file, for example in
my_private_key.ppk . For Putty, you need to convert this key to putty format. To do this, run
puttygen , tell it load private key file. He should open it without error messages. We save the private key (putty converted it) to the same other file, for example, to
c: \ ec2 \ my_putty_private_key.ppk .
I hope you did not give up and did it all, because in fact, it was the most difficult thing :) Then everything is much nicer. Of course, you can continue to manage your instances through the command line, using a set of utilities from Amazon, but I recommend to install a plug-in to FireFox for EC2 work with
comfort :
Elasticfox and then work with EC2 will be an order of magnitude easier. Installing ElasticFox I think will not cause any difficulties to anyone, and the whole setup consists in opening the plug-in from the Tools menu (English Tools) on a separate browser page, clicking on the Credentials button and entering
AWS Access Key and
AWS Secret Access in a separate window
Key . Additionally, to work with EC2, you need to install the
Putty program I mentioned above.
So we have a computer set up, it is worth Putty and ElasticFox plugin. It remains quite a bit to run and work with your instance. The fact is that as soon as you start working with EC2, all ports of access to your instances are closed. They need to open. To do this, in ElasticFox go to the tab
“Security groups” on the left select the group
“default” and add port 22 (work via ssh) by clicking the button
“Grant Permission” in the right-top. In the same way, you need to open all the ports that you will need to access instances outside of EC2: 80, 443, etc. A great feature of EC2 is that all your instances work in an isolated space and are not accessible to other instances in EC2. But at the same time, they perfectly see each other from the inside through their internal IP.
We run our first instance in EC2
In the ElasticFox plugin on the Images tab we find the image you need. In our case, this is an image with AMI
ami-71fd1a18 with a description of
alestic / ubuntu-8.04-hardy-base-20090216.manifest.xml , click the
Launch Instance (s) button above the grid and in the
Available window select the desired geographic instance launch area (America or Europe).
A remarkable feature of the distribution on Ubuntu is the ability to enter a shell script in the
User Data field that automatically executes when the instance is started. You can also specify the number of instances running more than one. Push the
Launch button.
On the Instances tab, you can observe the startup process. Usually you need to wait two minutes until the text
“running” appears in the State column. The public name of your machine appears in the Public DNS column. And now an important point: every time you start the Public DNS instance and the external IP will be different. To avoid the constant change of IP from your instance after it stops and then start a new instance in EC2, there is an Elastic IP. You can get one or more on the Elastic IPs tab. Once received, Elastic IP is always yours. After receiving the Elastic IP, you need to bind it to the running instance on the same tab. It is important to remember that as long as you do not tie up Elastic IP to any instance, you pay a little money for it. This is done in order not to take a lot of unnecessary Elastic IPs. When you stop an instance, Elastic IP is automatically unbound from it and can be tied to a new instance. So you can safely bind your Elastic IP to your domain. It will never change.
After starting the instance and linking Elastic IP to it, we start Putty and first of all configure it:
- On the Session tab, in the Host Name field, enter the IP or domain name of your instance. Here enter the previously obtained Elastic IP
- on the tab Connection -> Data in the field Auto-Login username enter: root
- On the Connection tab -> SSH, select the checkbox “Enable compression” to reduce traffic
- on the Connection tab -> SSH -> Auth in the last field, enter with: \ ec2 \ private_putty_key.ppk
- On the Session tab, in the Saved Session field, enter the name of the configuration and click the Save button.
In the future, you will just have to start Putty, select the saved configuration from the list and click the
Open button. Push it.
The console of your instance with root access appears. You can install and customize any program. After completing the entire setup, you can create your own image, which you can later use when starting your instances, as I described above. To create your image you need to follow these steps:
- copy the access keys in mnt to the instance, for example using the following commands:
C:\Program Files\PuTTY\pscp.exe" -i C:\ec2\key_2_private.ppk c:\ec2\509Certificate.pem root@%1:/mnt/
C:\Program Files\PuTTY\pscp.exe" -i C:\ec2\key_my_private.ppk c:\ec2\PrivateKey.pem root@%1:/mnt/
- execute the following command in the instance console:
ec2-bundle-vol -d /mnt/ -k /mnt/PrivateKey.pem -c /mnt/509Certificate.pem -u YourAccountNumber -r i386 -e /root/.mc,/root/.ssh -p name_of_your_image
- upload the created image in S3 with the command:
ec2-upload-bundle -a YourAccessKey -s YourSecretKey -b YourPathInS3 -m /mnt/name_of_your_image.manifest.xml
- register a new image in the AMIs list using ElasticFox by selecting the Images tab, clicking the Register AMI button and entering YourPathInS3 + name_of_your_image.manifest.xml indicated above
Mounted volumes or EBS
An unpleasant feature of EC2 is the loss of all data when the instance is stopped or frozen. On the one hand, this is a good challenge when designing a system. On the other hand, there is a way out now - this is
Amazon Elastic Block Store (EBS) . EBS is a mountable volume with capacities from 1GB to 1TB. Immediately after starting the instance, you can mount it to it using the
Volumes and Snapshots tab in Elasticfox. Instance itself, you need to mount this volume as usual in Linux.
The speed and reliability of EBS is higher than that built into the instance storage. When the instance is stopped, the mounted volume along with its data is not lost anywhere and it can be remounted to another instance. For even greater reliability, you can periodically make snapshots of a volume in S3. The only drawback of EBS is that you need to pay extra for it, but in reality we only get $ 2 per month.
Amazon S3 or High-Load Distributed Storage
I will list the main advantages of S3:
- low storage cost: you spend only $ 0.15 per gigabyte per month and usually $ 1-10 per traffic
- reliability: any file put in S3 is automatically copied to several servers to avoid losing it
- maintains any load: S3 is built in such a way that it automatically increases the number of copies of a file, while increasing the number of people who want to receive this file
- ease of working through the HTTP protocol
- opportunity to attach metadata to each file in S3, up to 2 kb in size.
- The ability to get statistics on access to each file via S3
- two geographic locations for your baskets: America and Europe
- the ability, using Amazon CloudFront , to reduce the response time to receive a file from S3 with a slight increase in costs
The easiest way to work with S3 is to install a
Forefox S3Fox
plugin . After installation, simply open the plugin through the Tools menu, click the Manage Accounts button and enter any
Account Name and
AWS Access Key and AWS Secret Access Key received earlier. This is all to start working with S3.
Your file system is shown on the left, and your baskets, folders, and files in S3 are shown on the right. The first step is to create your first basket (eng. Bucket) by clicking the
“Create Bucket / Directory” button on the top right. This is where your files are stored (they are called objects in S3). When creating a basket, its geographical location is indicated: America or Europe. It is also important to understand that the name of the basket must be unique among all S3 users. Why so, because the access path in your file is formed either like this:
“ s3.amazonaws.com/your_bucket/your_file ” , or
“ your_bucket.s3.amazonaws.com/your_file ” . After creating the recycle bin, you can copy files into it, as in a regular file manager. After copying the file to S3, you can select it in the right pane and set permissions for it by clicking
the Edit ACL button at the top-right. For example, you can specify whether the file is visible to everyone, only authorized users, or only to you, as the owner of the file.
You can copy your entire website to S3 and even link a specific basket to your domain. If your website is written only with the use of JavaScript and does not store data in the database, then you get a very cheap website hosting for only $ 0.15 per gigabyte per month + traffic of $ 0.17 per gigabyte. In practice, expenses usually do not exceed $ 5-10 dollars per month. Given the reliability and withstanding large loads, this solution looks much better than shared hosting, even in price.
If you fantasize further, then you can use Amazon SimpleDB as a database with access to it from JavaScript, but I'm not an expert in this topic, so I will only outline the possible direction of research for those who want it.
Detailed work with S3 is described in the
Getting Started Guide.
To be continued...
The following articles will describe: Erlang's interaction with other programming languages (Java, C) and the use of the distributed Mnesia database (which is part of Erlang / OTP) in
Risovaska .