Copying Amazon S3 Objects and Baskets to Windows Azure Storage

In this article - how to copy an object and a basket (bucket) from Amazon S3 to the Windows Azure blob storage.

One of the major innovations since June 7, 2012 was the improvement of the Copy Blob function. When writing this article I used the materials of the development team, which can be found here: http://blogs.msdn.com/b/windowsazurestorage/archive/2012/06/12/introducing-asynchronous-cross-account-copy-blob.aspx . What caught my attention - the copy blob functionality allows copying blobs from outside Windows Azure, if they are publicly available. That is, they do not need to be in Windows Azure.

IT IS VERY COOL!!!

This made me sit down and think .. So now I have the option of simply transferring my files to the Windows Azure blob storage, with most of the work being done by the platform itself. That's why I thought instead of writing a simple application that tries to copy a file (object) from Amazon S3 to the Windows Azure blobs storage. It turned out to be done in a few hours (hours because there was no account in S3 until a certain point, respectively, knowledge of the client library for the repository was somewhat limited). But what I want to say is: it really works and it is really easy to do.
')

Why do this?

I thought about several scenarios in which it might be necessary to copy content from one cloud provider to a Windows Azure blob storage.

Windows Azure blob storage really competitive

With all the new features and lower prices, Windows Azure Storage is a real alternative to other cloud providers that provide cloud storage services. This may be one of the reasons why you will switch from your provider to Windows Azure Storage. Or you always wanted to go, but could not figure out how to transfer data from the repository. The new function Copy Blob makes the solution to this problem super simple.

Windows Azure blob storage as a backup for existing data

Now you can easily use the blob storage as a place to back up data from your cloud storage. It was possible before, but it was a rather painful process, as it required a lot of code.

With the latest innovations it has become really simple. Now you do not need to write code to copy the bytes from the source to the blob in Windows Azure. All this makes the platform. You just say where the source is and where it should be transferred, call the copy function and that's it.

Show me the code !!!

OK, stop talking! Let's look at the code. I created a simple console application (code below), but before I go to it, I would like to discuss a couple more points. .

Prerequisites

Before running the code, you need to perform several actions:

1. Create a storage account: note that the new functionality will work only if the storage account was created after June 7, 2012 (why - see note , for an interpreter).

2. Download the latest version of the client library of the repository: at the time of this writing, the officially released version of the library is 1.7. Unfortunately, the new functionality does not work in it, so you need to download 1.7.1 from GitHub . Download the source and compile.

3. Check the public availability of the object / blob: Copy Blobs function can only copy publicly available blobs from outside Windows Azure. Therefore, in the case of Amazon S3 , you need to make sure that the object has at least READ permissions for anonymous users.

Code

As I said above, the code is very simple:

using System; using System.Collections.Generic; using System.Linq; using System.Text; using Microsoft.WindowsAzure; using Microsoft.WindowsAzure.StorageClient; namespace CopyAmazonObjectToBlobStorage { class Program { private static string azureStorageAccountName = “"; private static string azureStorageAccountKey = ""; private static string azureBlobContainerName = ""; private static string amazonObjectUrl = ""; private static string azureBlobName = ""; static void Main(string[] args) { CloudStorageAccount csa = new CloudStorageAccount(new StorageCredentialsAccountAndKey(azureStorageAccountName, azureStorageAccountKey), true); CloudBlobClient blobClient = csa.CreateCloudBlobClient(); var blobContainer = blobClient.GetContainerReference(azureBlobContainerName); Console.WriteLine("Trying to create the blob container...."); blobContainer.CreateIfNotExist(); Console.WriteLine("Blob container created...."); var blockBlob = blobContainer.GetBlockBlobReference(azureBlobName); Console.WriteLine("Created a reference for block blob in Windows Azure...."); Console.WriteLine("Blob Uri: " + blockBlob.Uri.AbsoluteUri); Console.WriteLine("Now trying to initiate copy...."); blockBlob.StartCopyFromBlob(new Uri(amazonObjectUrl), null, null, null); Console.WriteLine("Copy started...."); Console.WriteLine("Now tracking blob's copy progress...."); bool continueLoop = true; while (continueLoop) { Console.WriteLine(""); Console.WriteLine("Fetching lists of blobs in Azure blob container...."); var blobsList = blobContainer.ListBlobs(true, BlobListingDetails.Copy); foreach (var blob in blobsList) { var tempBlockBlob = (CloudBlob) blob; var destBlob = blob as CloudBlob; if (tempBlockBlob.Name == azureBlobName) { var copyStatus = tempBlockBlob.CopyState; if (copyStatus != null) { Console.WriteLine("Status of blob copy...." + copyStatus.Status); var percentComplete = copyStatus.BytesCopied / copyStatus.TotalBytes; Console.WriteLine("Total bytes to copy...." + copyStatus.TotalBytes); Console.WriteLine("Total bytes copied...." + copyStatus.BytesCopied); if (copyStatus.Status != CopyStatus.Pending) { continueLoop = false; } } } } Console.WriteLine(""); Console.WriteLine("=============================================="); System.Threading.Thread.Sleep(1000); } Console.WriteLine("Process completed...."); Console.WriteLine("Press any key to terminate the program...."); Console.ReadLine(); } } } 

The code is quite simple. What I do: I specify credentials for accessing Windows Azure storage and the source blob URL (in Amazon S3); I create the blob container in Windows Azure and start copying the blob from the source URL. After sending a request for copying everything that the application does, it monitors the copying process. As you can see, not a single line of code to copy bytes from the source to the receiver. All this makes Windows Azure.

This is what an object in Amazon S3 looks like:


After the object is copied, I can see it in the Windows Azure blob storage using Cloud Storage Studio .



As you saw, Windows Azure provides the ability to easily transfer data. If you are considering the possibility of moving from another cloud platform to Windows Azure, but were worried about any related problems, one problem has become less.

I created a simple example that copies a single file from Amazon S3 to the Windows Azure blob storage. This functionality can be extended to copy all objects from the Amazon S3 recycle bin to the Windows Azure blob container. Let's move on to the next article - how to copy a cart from Amazon S3 to the Windows Azure blob storage using Copy Blob.

Prerequisites

Before running the code, you need to perform several actions:

1. Check the public availability of the object / blob: the Copy Blobs function can only copy publicly available blobs from outside Windows Azure. Therefore, in the case of Amazon S3 , you need to make sure that the object has at least READ permissions for anonymous users.
2. Create a storage account: note that the new functionality will only work if the storage account was created after June 7, 2012 (why - see note , for an interpreter).
3. Download the latest version of the client library of the repository: at the time of this writing, the officially released version of the library is 1.7. Unfortunately, the new functionality does not work in it, so you need to download 1.7.1 from GitHub . Download the source and compile.
4. Amazon SDK for .Net : You need to download the Amazon SDK for .Net from Amazon AWS .
5. Keep Amazon credentials close by : You’ll need to use Amazon AccessKey and SecretKey to get a list of objects stored in your Amazon S3 bucket.

Code

The code is very simple. Note that it is only for demonstration. You can modify it as you need.

 using System; using System.Collections.Generic; using System.Linq; using System.Text; using Amazon.S3; using Amazon.S3.Model; using Amazon.S3.Transfer; using Amazon.S3.Util; using Microsoft.WindowsAzure; using Microsoft.WindowsAzure.StorageClient; using System.Globalization; namespace CopyAmazonBucketToBlobStorage { class Program { private static string azureStorageAccountName = “”; private static string azureStorageAccountKey = “”; private static string azureBlobContainerName = “”; private static string amazonAccessKey = “”; private static string amazonSecretKey = “”; private static string amazonBucketName = “”; private static string objectUrlFormat = “https://{0}.s3.amazonaws.com/{1}”; private static Dictionary<string, CopyBlobProgress> CopyBlobProgress; static void Main(string[] args) { //     Windows Azure CloudStorageAccount azureCloudStorageAccount = new CloudStorageAccount(new StorageCredentialsAccountAndKey(azureStorageAccountName, azureStorageAccountKey), true); //    ,    . var blobContainer = azureCloudStorageAccount.CreateCloudBlobClient().GetContainerReference(azureBlobContainerName); //  ,   . Console.WriteLine(«Trying to create the blob container....»); blobContainer.CreateIfNotExist(); Console.WriteLine(«Blob container created....»); //    Amazon AmazonS3Client amazonClient = new AmazonS3Client(amazonAccessKey, amazonSecretKey); //  CopyBlobProgress = new Dictionary<string, CopyBlobProgress>(); string continuationToken = null; bool continueListObjects = true; //  ListObjects        1000 , //    ,      . while (continueListObjects) { ListObjectsRequest requestToListObjects = new ListObjectsRequest() { BucketName = amazonBucketName, Marker = continuationToken, MaxKeys = 100, }; Console.WriteLine(«Trying to list objects from S3 Bucket....»); //    Amazon S3 var listObjectsResponse = amazonClient.ListObjects(requestToListObjects); //  var objectsFetchedSoFar = listObjectsResponse.S3Objects; Console.WriteLine(«Object listing complete. Now starting the copy process....»); //,     .    ,        continuation token. continuationToken = listObjectsResponse.NextMarker; foreach (var s3Object in objectsFetchedSoFar) { string objectKey = s3Object.Key; //ListObjects    ,  ,     S3 Object Key.     /,    . if (!objectKey.EndsWith(«/»)) { // URL   string urlToCopy = string.Format(CultureInfo.InvariantCulture, objectUrlFormat, amazonBucketName, objectKey); //  CloudBlockBlob CloudBlockBlob blockBlob = blobContainer.GetBlockBlobReference(objectKey); var blockBlobUrl = blockBlob.Uri.AbsoluteUri; if (!CopyBlobProgress.ContainsKey(blockBlobUrl)) { CopyBlobProgress.Add(blockBlobUrl, new CopyBlobProgress() { Status = CopyStatus.NotStarted, }); // .     try/catch //    Amazon S3       . try { Console.WriteLine(string.Format(«Trying to copy \»{0}\» to \»{1}\»«, urlToCopy, blockBlobUrl)); blockBlob.StartCopyFromBlob(new Uri(urlToCopy)); CopyBlobProgress[blockBlobUrl].Status = CopyStatus.Started; } catch (Exception exception) { CopyBlobProgress[blockBlobUrl].Status = CopyStatus.Failed; CopyBlobProgress[blockBlobUrl].Error = exception; } } } } Console.WriteLine(««); Console.WriteLine(«==========================================================«); Console.WriteLine(««); Console.WriteLine(«Checking the status of copy process....»); //  bool checkForBlobCopyStatus = true; while (checkForBlobCopyStatus) { //      var blobsList = blobContainer.ListBlobs(true, BlobListingDetails.Copy); foreach (var blob in blobsList) { var tempBlockBlob = blob as CloudBlob; var copyStatus = tempBlockBlob.CopyState; if (CopyBlobProgress.ContainsKey(tempBlockBlob.Uri.AbsoluteUri)) { var copyBlobProgress = CopyBlobProgress[tempBlockBlob.Uri.AbsoluteUri]; if (copyStatus != null) { Console.WriteLine(string.Format(«Status of \»{0}\» blob copy....», tempBlockBlob.Uri.AbsoluteUri, copyStatus.Status)); switch (copyStatus.Status) { case Microsoft.WindowsAzure.StorageClient.CopyStatus.Aborted: if (copyBlobProgress != null) { copyBlobProgress.Status = CopyStatus.Aborted; } break; case Microsoft.WindowsAzure.StorageClient.CopyStatus.Failed: if (copyBlobProgress != null) { copyBlobProgress.Status = CopyStatus.Failed; } break; case Microsoft.WindowsAzure.StorageClient.CopyStatus.Invalid: if (copyBlobProgress != null) { copyBlobProgress.Status = CopyStatus.Invalid; } break; case Microsoft.WindowsAzure.StorageClient.CopyStatus.Pending: if (copyBlobProgress != null) { copyBlobProgress.Status = CopyStatus.Pending; } break; case Microsoft.WindowsAzure.StorageClient.CopyStatus.Success: if (copyBlobProgress != null) { copyBlobProgress.Status = CopyStatus.Success; } break; } } } } var pendingBlob = CopyBlobProgress.FirstOrDefault(c => c.Value.Status == CopyStatus.Pending); if (string.IsNullOrWhiteSpace(pendingBlob.Key)) { checkForBlobCopyStatus = false; } else { System.Threading.Thread.Sleep(1000); } } if (string.IsNullOrWhiteSpace(continuationToken)) { continueListObjects = false; } Console.WriteLine(««); Console.WriteLine(«==========================================================«); Console.WriteLine(««); } Console.WriteLine(«Process completed....»); Console.WriteLine(«Press any key to terminate the program....»); Console.ReadLine(); } } public class CopyBlobProgress { public CopyStatus Status { get; set; } public Exception Error { get; set; } } public enum CopyStatus { NotStarted, Started, Aborted, Failed, Invalid, Pending, Success } } 

The code is quite simple. Application:

1. Creates a blob container in Windows Azure storage, if necessary.
2. Gets a list of objects from the Amazon S3 bucket. Note that Amazon S3 can return up to 1000 objects in one function call, otherwise the continuation token is returned, which is used to get a marker that represents the next subset of entities.
3. For each object returned, a URL is generated that is used for the Copy Blob function.
4. The application checks the status of the copy process. After all copying processes are completed, steps 2-4 are repeated until all objects are copied.

To manage content in Amazon S3, I used Bucket Explorer , which showed the following state:



After the end of the copy, the next state of Windows Azure blob storage was shown by Cloud Storage Studio .



Summary

I created a simple example that copies all the content from the Amazon S3 bucket to the Windows Azure blob storage.

Resume from translator

Emotions of Gaurav about the new features of the API are clear - to be honest, before June 7, 2012, the API for using the storage services in some places was lame in functionality, as evidenced by the translation series of articles by the same Gaurav about the comparison of Windows Azure and Amazon storage. Now, the ability to use Windows Azure storage as the same backup space, and even with the ability to adjust the degree of redundancy (two types are available now), allows you to avoid the consequences of any cataclysm and create a truly tolerant (:) service.


Note from the translator

Windows Azure storage services can receive requests that indicate a specific version of each operation (service), while the developer can specify the version using the x-ms-version header. Naturally, the functionality of different versions and mechanisms of work (non-conceptual) may vary.

It is necessary to take into account the rules of the system by which the blob service determines which version to use to interpret the SAS parameters:

1. If there is an x-ms-version header in the request, but no signedversion is present, version 2009-07-17 is used to interpret the SAS parameters, while the version specified in x-ms-version is used for the operation.

2. If there is no x-ms-version header in the request and there is no signedversion, but the account owner has installed the service version using the Set Blob Service Properties operation, version 2009-07-07 will be used to interpret the SAS parameters, for the operation the version specified account owner for the service.

3. If the request does not have an x-ms-version header, there is no signedversion and the account owner has not installed the service version, version 2009-07-17 will be used to interpret the SAS parameters. If the container has public rights and access restrictions have been set using the Set Container ACL operation using version 2009-09-19 (or newer). Version 2009-09-19 will be used for the operation.

Learn more about the version and the functionality they support.

Translation related posts Gaurav Mantri:

Objects

Baskets