Running TechDays.ru in Clouds
On February 15, a remarkable event happened - our site TechDays.ru moved to the Windows Azure platform.
At the same time, the portal also acquired new functionalities that should be liked by both portal visitors and report authors:
I was able to talk with Sergei Pugachev , who was engaged in porting the portal to Windows Azure. Sergey is a Silverlight MVP, as well as the founder of the Silverlighter.ru community. Further I bring to your attention our interview.
Sergey, please tell us a little about the TechDays.ru portal?
TechDays.ru is a portal where you can find records of reports from technology conferences held by Microsoft in Russia, for example, DevCon, TechEd Russia, Platform, etc., as well as training videos on technologies and products from both Microsoft and Microsoft. and other vendors. All reports can be viewed and downloaded absolutely free of charge and without registration.
')
Most of the reports on TechDays were created by independent developers and system administrators, which is especially important since such reports are based on the practical experience of their authors. More than 60,000 people visit TechDays.ru per month, and more than 185,000 users are registered. Over the year, users view and download reports, more than 5,000,000 (five million) times, either directly from the site or through links to other sites.
When was the decision to transfer the site to Windows Azure? How did you plan the migration process?
We migrated to the cloud gradually. In early 2011, it was decided to transfer all the videos to Azure, which we implemented in a fairly short period of time, in just a week. The reason for the transfer was the fact that Azure provides a very convenient data storage (Blob storage). This storage has such advantages as infinite volume (at affordable prices per gigabyte of data), high availability, as well as excellent scalability. Before transferring video to Azure, we constantly rested on either channel restrictions, disk system limitations, or server restrictions. Now the channel is not a problem. Thousands of users can download videos in parallel, without slowing down the speed of return.
After that, in the fall of 2011, we decided to create a new version of the site engine using ASP.NET MVC / Razor technology (the old version used ASP.NET Web Forms), as well as rework the structure and interface of the site. It was logical that the new version will work fully in Window Azure. February 15, 2012 we launched this version.
Porting the portal to Azure solved a number of problems. First of all, we were able to adapt to increasing loads. We really like the fact that in Azure there is no need to do manual configuration and administration of hardware and operating systems, configuration of NLB (Network Load Balancer) clusters and other things that took a lot of time before. We simply indicate the number of virtual machines that we need at the moment. And it works.
We also use the functionality of test environments. In Azure, there is a separation between Staging and Production. We can roll out a new version on Staging, test it, and immediately transfer to Production. In this case, everything at any time can be returned back.
Please tell us more about the architecture of the portal and about the used services and services of Windows Azure?
We use most of the services provided by Azure. We have a web-frontend (Web role) and backend (Worker role). Backend is needed for video encoding and other utilitarian tasks. Communication between parts of the application occurs asynchronously using queues. In addition, the roles communicate with each other directly using WCF services. It is necessary to control the process of encoding specific files.
At the moment, the web-frontend, like the backend, consists of 2 medium virtual machines. When transcoding video, we used up to 32 medium virtual machines on the backend in parallel, which made it possible to transcode all videos in just a couple of days. The change in the number of used virtual machines is currently implemented manually, but we are working on automating this process so that resources are allocated depending on the number of video files waiting to be encoded.
Another service we work with is Window Azure Cache. This is a distributed and very fast cache, which is surprisingly easy to use in ASP.NET sites. Including thanks to him, we were able to increase the productivity of the portal. Another factor in increasing the speed of work was the use of a CDN. Through the use of CDN, we can quickly give files to users anywhere in the world. In many cases, the increase in speed when using CDN reaches more than two times.
SQL Azure is used as data storage. The name of the experience with SQL Server to understand SQL Azure was not difficult, in many ways, so SQL Azure is used so widely. In addition, the database schema did not even have to be modified to work with SQL Azure. Although all development was done on SQL Server 2008 R2.
We are gradually increasing the share of another service provided by Azure for data storage, namely the Tables Storage service. Tables are not relational storage, but it is convenient to work with them and they can store huge amounts of data. In addition, storing data in tables is much cheaper than in SQL Azure. While the tables we have used to store logs and service information.
Sergey, it turns out that in terms of cost, the transition was economically beneficial. And what can you say about the DDos-attacks on techdays.ru, because in Windows Azure the payment goes on the fact of use and the actual load?
Yes, the transition to Azure has proven cost-effective. You should not expect that using Azure will be cheaper than the simplest virtual hosting, but in our case, the cost of Azure turned out to be no more than the cost of building your own infrastructure. However, many components of our own infrastructure were not fault tolerant.
About DDoS, you can say the following. On the one hand, the Windows Azure platform itself provides NLB and DDoS protection at the network level, but you still have to pay for the amount of resources that DDoS can withstand. On TechDays, attacks are regularly carried out, and once, when we did not work in Azure, a DDoS attack brought the portal down for about a day. The problem was in the network infrastructure. The network infrastructure in Azure does not have such problems and can withstand a similar load. But in any case, the number of virtual machines must be such as to withstand DDoS.
In Azure, in general, there is always a choice, either to write faster code that can withstand heavy loads, or to pay for more resources. We try to write optimized code in order to use as few resources as possible, and in case of high load we simply increase the number of virtual machines in a couple of minutes.
And, of course, the fewer errors in the code and omissions, the more resistant your system is to threats. It will take a lot of resources (which can not be found) to put a well-designed and “coded” site using DDos.
What are the main difficulties in the migration process?
The main difficulty was the difficulty of debugging code directly in Azure. There were situations when the code worked on the emulator, but did not work in Azure. With the release of the new versions of the SDK, these difficulties seriously leveled out. I have been working with Azure since 2010, and the progress in the development of the SDK is amazing. If earlier I lacked many things, now what I needed was already implemented. And this is good news. Naturally, not all my wishes were realized. But the speed of development of the SDK pleasantly surprised me.
Another difficulty was pouring data into SQL Azure. Regular funds for the migration of large volumes of data yet. Therefore, we used the SQL Azure Migration Wizard , c http://codeplex.com . This product coped with its tasks.
I would also like to note that for more than a year of my work with Azure, I had to contact technical support once and then on organizational and administrative matters.
The transition to Windows Azure can be called successful?
Yes, the transition can be called successful. We have seriously increased the speed of the portal and now we are not afraid of the influx of visitors. Also rose uptime. Now we can not think about the channel, hard drives, physical servers and infrastructure in general, so we have more time to create useful features for users.
Sergey, thank you very much for your efforts on techdays.ru processing and interviews!
At the same time, the portal also acquired new functionalities that should be liked by both portal visitors and report authors:
- The portal interface has been completely redesigned. The new version is made in the Metro style. The portal has become more modern, faster, more dynamic and more convenient to use. It should be noted that work on improving the interface continues.
- Added video encoding service. Now TechDays has become a full-fledged video hosting. Previously, the video had to be encoded by the authors of the reports in several formats, now it happens automatically.
- Thanks to the encoding service, all videos are now available in h264 format and played using the HTML5 player. Now reports from TechDays can be viewed on mobile devices, for example, on Windows Phone 7, on iPhone and iPad.
I was able to talk with Sergei Pugachev , who was engaged in porting the portal to Windows Azure. Sergey is a Silverlight MVP, as well as the founder of the Silverlighter.ru community. Further I bring to your attention our interview.
Sergey, please tell us a little about the TechDays.ru portal?
TechDays.ru is a portal where you can find records of reports from technology conferences held by Microsoft in Russia, for example, DevCon, TechEd Russia, Platform, etc., as well as training videos on technologies and products from both Microsoft and Microsoft. and other vendors. All reports can be viewed and downloaded absolutely free of charge and without registration.
')
Most of the reports on TechDays were created by independent developers and system administrators, which is especially important since such reports are based on the practical experience of their authors. More than 60,000 people visit TechDays.ru per month, and more than 185,000 users are registered. Over the year, users view and download reports, more than 5,000,000 (five million) times, either directly from the site or through links to other sites.
When was the decision to transfer the site to Windows Azure? How did you plan the migration process?
We migrated to the cloud gradually. In early 2011, it was decided to transfer all the videos to Azure, which we implemented in a fairly short period of time, in just a week. The reason for the transfer was the fact that Azure provides a very convenient data storage (Blob storage). This storage has such advantages as infinite volume (at affordable prices per gigabyte of data), high availability, as well as excellent scalability. Before transferring video to Azure, we constantly rested on either channel restrictions, disk system limitations, or server restrictions. Now the channel is not a problem. Thousands of users can download videos in parallel, without slowing down the speed of return.
After that, in the fall of 2011, we decided to create a new version of the site engine using ASP.NET MVC / Razor technology (the old version used ASP.NET Web Forms), as well as rework the structure and interface of the site. It was logical that the new version will work fully in Window Azure. February 15, 2012 we launched this version.
Porting the portal to Azure solved a number of problems. First of all, we were able to adapt to increasing loads. We really like the fact that in Azure there is no need to do manual configuration and administration of hardware and operating systems, configuration of NLB (Network Load Balancer) clusters and other things that took a lot of time before. We simply indicate the number of virtual machines that we need at the moment. And it works.
We also use the functionality of test environments. In Azure, there is a separation between Staging and Production. We can roll out a new version on Staging, test it, and immediately transfer to Production. In this case, everything at any time can be returned back.
Please tell us more about the architecture of the portal and about the used services and services of Windows Azure?
We use most of the services provided by Azure. We have a web-frontend (Web role) and backend (Worker role). Backend is needed for video encoding and other utilitarian tasks. Communication between parts of the application occurs asynchronously using queues. In addition, the roles communicate with each other directly using WCF services. It is necessary to control the process of encoding specific files.
At the moment, the web-frontend, like the backend, consists of 2 medium virtual machines. When transcoding video, we used up to 32 medium virtual machines on the backend in parallel, which made it possible to transcode all videos in just a couple of days. The change in the number of used virtual machines is currently implemented manually, but we are working on automating this process so that resources are allocated depending on the number of video files waiting to be encoded.
Another service we work with is Window Azure Cache. This is a distributed and very fast cache, which is surprisingly easy to use in ASP.NET sites. Including thanks to him, we were able to increase the productivity of the portal. Another factor in increasing the speed of work was the use of a CDN. Through the use of CDN, we can quickly give files to users anywhere in the world. In many cases, the increase in speed when using CDN reaches more than two times.
SQL Azure is used as data storage. The name of the experience with SQL Server to understand SQL Azure was not difficult, in many ways, so SQL Azure is used so widely. In addition, the database schema did not even have to be modified to work with SQL Azure. Although all development was done on SQL Server 2008 R2.
We are gradually increasing the share of another service provided by Azure for data storage, namely the Tables Storage service. Tables are not relational storage, but it is convenient to work with them and they can store huge amounts of data. In addition, storing data in tables is much cheaper than in SQL Azure. While the tables we have used to store logs and service information.
Sergey, it turns out that in terms of cost, the transition was economically beneficial. And what can you say about the DDos-attacks on techdays.ru, because in Windows Azure the payment goes on the fact of use and the actual load?
Yes, the transition to Azure has proven cost-effective. You should not expect that using Azure will be cheaper than the simplest virtual hosting, but in our case, the cost of Azure turned out to be no more than the cost of building your own infrastructure. However, many components of our own infrastructure were not fault tolerant.
About DDoS, you can say the following. On the one hand, the Windows Azure platform itself provides NLB and DDoS protection at the network level, but you still have to pay for the amount of resources that DDoS can withstand. On TechDays, attacks are regularly carried out, and once, when we did not work in Azure, a DDoS attack brought the portal down for about a day. The problem was in the network infrastructure. The network infrastructure in Azure does not have such problems and can withstand a similar load. But in any case, the number of virtual machines must be such as to withstand DDoS.
In Azure, in general, there is always a choice, either to write faster code that can withstand heavy loads, or to pay for more resources. We try to write optimized code in order to use as few resources as possible, and in case of high load we simply increase the number of virtual machines in a couple of minutes.
And, of course, the fewer errors in the code and omissions, the more resistant your system is to threats. It will take a lot of resources (which can not be found) to put a well-designed and “coded” site using DDos.
What are the main difficulties in the migration process?
The main difficulty was the difficulty of debugging code directly in Azure. There were situations when the code worked on the emulator, but did not work in Azure. With the release of the new versions of the SDK, these difficulties seriously leveled out. I have been working with Azure since 2010, and the progress in the development of the SDK is amazing. If earlier I lacked many things, now what I needed was already implemented. And this is good news. Naturally, not all my wishes were realized. But the speed of development of the SDK pleasantly surprised me.
Another difficulty was pouring data into SQL Azure. Regular funds for the migration of large volumes of data yet. Therefore, we used the SQL Azure Migration Wizard , c http://codeplex.com . This product coped with its tasks.
I would also like to note that for more than a year of my work with Azure, I had to contact technical support once and then on organizational and administrative matters.
The transition to Windows Azure can be called successful?
Yes, the transition can be called successful. We have seriously increased the speed of the portal and now we are not afraid of the influx of visitors. Also rose uptime. Now we can not think about the channel, hard drives, physical servers and infrastructure in general, so we have more time to create useful features for users.
Sergey, thank you very much for your efforts on techdays.ru processing and interviews!
Source: https://habr.com/ru/post/140223/
All Articles