Cloud storage

With the rise of social applications such as Facebook, Instagram, YouTube and many others, managing user-generated content has become a problem, and problems need to be solved. Amazon AWS S3, Google Storage, Rackspace Cloud Files, and other similar services began to appear like mushrooms after the rain to help application developers solve an actual problem — managing a scalable asset repository. And of course they all use the “Cloud”!

Problem

Popular social applications, scientific applications and applications generating media content are capable of generating a huge amount of information in a short period of time. Here are a couple of examples:

• 72 hours of video are uploaded to YouTube by users every minute. ( source )
• 20 million photos are uploaded every day. ( source )
• Pinterest has saved 8 billion objects and 410 terabytes of data since its launch in 2009. ( source )
• Twitter generates approximately 12 terabytes of data per day. ( source )

When your application starts saving a huge amount of content that your users have generated, your team will have to decide where to invest their time to solve this problem. If your application is designed to place assets on your hardware / infrastructure, your team will spend a lot of time and money trying to efficiently store and manage assets. Alternatively, you can save assets with the cloud storage provider. By choosing this path, you can allow application content to scale almost unlimitedly, paying only for the space and resources used to serve this content. As a result, cloud storage will untie the hands of your engineers and allow you to concentrate on creating unique applications, instead of inventing a bicycle, when scalability becomes a problem.

')

When should you think about using cloud storage for your application?

• When the content generated by your users is part of the application. Does your application allow users to upload files? Does the application generate server-side files? If the application will receive files or the generated content will be stored in the file system, sooner or later you will have to think about cloud storage.
• When your application is closely on the same server . If your application is compact enough to run on a single server or web host and you do not expect changes, it does not make sense to keep your assets in the cloud. You can configure their storage in the local file system and forget about it. If you notice that you need a second application server, you can immediately reap the rewards and delights of cloud storage of assets. You will be able to scale your service horizontally to an unlimited number of application servers without the need to replicate your file system assets on new servers. Since your assets will be stored centrally, they will be accessible from anywhere, regardless of the number of servers your application will be running on.
• When it is more important for your team to concentrate on developing the capabilities of your program that are critical to your business, rather than developing a scalable file system. If you have little either money or time and you anticipate the growth of your application, cloud storage will be a win-win option. It will give you the opportunity to quickly deploy the application and scale it as needed.

Integration and access

Most of the leading cloud storage provides access through its API, allowing developers to integrate cloud storage of assets into their application. Below we take a look at a few code examples using the SDK or library for storing assets on Amazon S3.

Ruby & Carrierwave

Code samples are adapted from the CarrierWave repository.

• Install CarrierWave: gem install carrierwave or into your gemfile gem 'carrierwave'
• Install Fog: gem install fog or in your gemfile gem "fog", "~> 1.3.1"
• Add the following to the initialization file:
  

  CarrierWave.configure do |config| config.fog_credentials = { :provider => 'AWS', # required :aws_access_key_id => 'xxx', # required :aws_secret_access_key => 'yyy', # required :region => 'eu-west-1' # optional, defaults to 'us-east-1' } config.fog_directory = '_' config.fog_public = false config.fog_attributes = {'Cache-Control'=>'max-age=315576000'} config.asset_host = 'https://assets.example.com' end 

• Create an uploader class:
  

   class AvatarUploader < CarrierWave::Uploader::Base storage :fog end 

• Using uploader directly:
  

   uploader = AvatarUploader.new uploader.store!(my_file) uploader.retrieve_from_store!('my_file.png') 

• Use your uploader with ActiveRecord:
  Add a field to your database table and require CarrierWave:
  

   add_column :users, :avatar, :string in a database migration file require 'carrierwave/orm/activerecord' in your model file. 

• Connect your uploader to the model:
  

   class User < ActiveRecord::Base mount_uploader :avatar, AvatarUploader end 

• Work with model and files:

   u = User.new u.avatar = params[:file] u.avatar = File.open('somewhere') u.save! u.avatar.url # => '/url/to/file.png' u.avatar.current_path # => 'path/to/file.png' u.avatar.identifier # => 'file.png' 

Here are examples of using CarrierWave for uploading to Amazon S3, Rackspace Cloud Files and Google Storage , as well as a few jams for ORM, such as DataMapper , Mongoid and Sequel .

PHP & AWS SDK

Amazon provides PHP SDK for working with AWS APIs and services. For this example, we will use the instructions from the README repository SDK.

• Copy the contents of config-sample.inc.php and add your credentials as indicated in the file.
• Move your file to ~ / .aws / sdk / config.inc.php .
• Remember that getenv ('HOME') should point to your user folder. Otherwise, you need to specify this putenv ('HOME = <your-user-folder>')
  

   //  AmazonS3  $s3 = new AmazonS3(); //   s3 bucket   $bucket = '--BUCKET' . strtolower($s3->key); if (!$s3->if_bucket_exists($bucket)) { $response = $s3->create_bucket($bucket,AmazonS3::REGION_US_E1); if (!$response->isOK()) die('Could not create `' . $bucket . '`.'); } //   . $response = $s3->get_object('aws-sdk-for-php', 'some/path-to-file.ext',array( 'fileDownload' => './local/path-to-file.ext' )); //  . $response = $s3->create_object($bucket, 'some/path-to-file.ext', array( 'fileUpload' => './local/path-to-file.ext' )); 

Node & Knox

For Node.js, I adapted sample code from a Knox Amazon S3 client on Github.

 //   var client = knox.createClient({ key: '<api-key-here>' , secret: '<secret-here>' , bucket: 'BUCKET-NAME' }); //    S3 client.putFile('some/path-to-file.ext', 'bucket/file-name.ext', function(err, res){ // Logic }); //    S3 client.get('/some/path-to-file.ext').on('response', function(res){ console.log(res.statusCode); console.log(res.headers); res.setEncoding('utf8'); res.on('data', function(chunk){ console.log(chunk); }); }).end(); //    S3 client.del('/some/path-to-file.ext').on('response', function(res){ console.log(res.statusCode); console.log(res.headers); }).end(); 

Conclusion

As you can see, working with AWS S3 APIs is extremely simple and there are a bunch of available libraries for most languages. I would recommend to look at cloud storage when developing your future project. You will save yourself time by not inventing new asset allocation solutions and you will be given an almost unlimited opportunity to scale cloud storage.