Cost Optimization with Amazon S3

Amazon S3 is convenient to use to store files of any format. In addition to the convenient API, we get almost dimensionless storage. Excellent accessibility and low cost make S3 mega-attractive for young and small projects.

However, over time, files become more and more. And you will have to pay not only for new data, but for all of history. In addition, Amazon is charging money for GET and POST requests, as well as for traffic.

Despite the low cost at the start, with the growth of this solution will be more and more expensive.

Should I use S3 at all?

Definitely worth it. Especially for small projects. Several tens of thousands of files and several thousand requests per day. All this will cost a couple of bucks a month.
')
In return, you will not need to spend time and money on various issues:

• Availability. AWS will always work.
• Scaling. No need to solve complex problems of file distribution between servers.
• Load. Sudden spikes in popularity will not lead to iron failure.

S3 is perfect for storing user files of any format - photos , documents and even video.

When not to use?

Definitely not to use S3 as a CDN. It is better to maintain all the statics (CSS, Javascript and icons) on your own or use the appropriate CDN provider for this.

How much will you have to pay?

For storing each terabyte of files you need to pay about 30 bucks a month. In addition, $ 90 will cost every terabyte of traffic.



Storage of 1 terabyte - about 5 million photos or 50 thousand short videos.
A site with 100 thousand views per day, on each page of which there are 5 photos, will generate traffic of 1 terabyte per month.

Cost increase

However, with an increase in the number of views by 10 times, S3 traffic will cost almost 1,000 bucks a month.
An alternative to this solution would be to rent several servers with a budget of $ 300.

Hybrid solution

A significant part of the cost when using S3 is just traffic.
Storage and loading is relatively inexpensive.



As a good solution, it makes sense to use S3 only as a reliable storage.
In this case, a significant reduction in the cost of traffic can be obtained using a simple caching scheme.



Then the files are downloaded directly to Amazon S3, and the return is configured through their own caching servers.

Varnish based solution

Varnish is a cool HTTP caching server. With it, you can customize caching files from Amazon S3. Configuration example:

backend s3 { .host = "s3.amazonaws.com"; .port = "80"; } sub vcl_recv { if (req.url ~ "\.(gif|jpg|jpeg|png)$") { unset req.http.cookie; unset req.http.cache-control; unset req.http.pragma; unset req.http.expires; unset req.http.etag; set req.backend = s3; set req.http.host = "my_bucket.s3.amazonaws.com"; lookup; } } sub vcl_fetch { set obj.ttl = 3w; } 

The example uses the bouquet my_bucket. You should also monitor the hit rate. Acceptable value - 99%. This means that 99% of all requests will fall into the cash register and only 1% will go to S3.

Additional measures

To further optimize costs, you need to follow the rule - to minimize file sizes. If the pictures are saved, you can convert them to Webp format before saving to S3. They will occupy much less space than JPG. Even without this, make sure that the correct image format is selected before saving.

Text files should be compressed with gzip.

Other cloud storage

There is an alternative. At S3, the world has not gone away with the wedge, so it’s worth trying to work with other clouds, for example Azure or Google Cloud, especially considering that almost all services offer to test them for free.

Abstract

• Amazon S3 is very convenient to use at start and for small projects.
• With the growing popularity of S3 becomes more expensive than its own iron.
• To reduce the cost of the solution it is necessary to reduce traffic. To do this, you need to give files through the caching server.
• For caching it is convenient to use Varnish.