AppScale - and build me a Google AppEngine myself? Will be done!

Today is one of the most popular and active topics? Cloud Computing in general , and one of the best, original and bright of its representatives - Google App Engine in particular. The good news about adding Java to the platform is that I can try, although I’m not going to write to him, to a certain extent, the competitor Stax , which I’m sad to say, I’m much more close to like it. But if you are still a Python supporter and want something similar, but completely yours, there is good news for you. The open AppScale project allows you to deploy your own cloud computing system to provide the ability to deploy and run applications there on Python, in principle, without changing the code, that for GoogleAppEngine that is written specifically for the system.

So, AppScale is some interface and tools for deploying and supporting applications in a Cloud-based environment, while applications have interface compatibility with a similar service from Google. At a lower level, AppScale relies on a number of proprietary subsystems-services, written in different languages, providing the necessary system functions for user scripts. Below is also a stack of open technologies, for example, Apache Hadoop , about this part we will discuss in more detail below. And at the most basic level, the entire system relies on an infrastructure based on the Xen virtualization system, while one server can act as a basic platform, as well as a cluster or a system of clusters, using either Eucalyptus or an open system. I briefly reviewed it , at the very beginning of development, today the system has already been seriously updated and actively developed, which is worth including its inclusion in the next Ubuntu distribution or commercial service Amazon EC2 .

If you're a fan of loud and sexy phrases, here's a description from the developers themselves - "AppScale is a multi-language, multi-component framework for executing GAE applications on virtualized
cluster systems. ”, which in translation will mean that AppScale is multilingual (meaning that it is possible to use for projects not only in Python, although the same applies to the system itself, it is also written in different languages), a multi-component framework for launch applications for Google AppEngine in a virtualized clustered environment. And now, with the arrangement, by points and in more detail.
')
The main system serves three components:

• AppServer (AS) - an application server, which, in fact, is responsible for the operation of applications;
• AppLoadBalancer (ALB) is a load balancer between nodes, using Nginx as this
• AppController (AC) - a component to control the interaction between the rest of the system.

In addition, the platform includes two more low-level components - DatabaseMaster, the main database server (or rather, data warehousing) and one or more DatabaseSlaves components, which provide fault-tolerant data storage and processing.



AppServer is responsible for executing user applications, and also directly accesses the data storage system using the HTTPS protocol, it accesses the DatabaseMaster for data, and then the data is processed either on the main server, or the request is sent to one of the slave servers.

AppController is one of the main components. It is responsible for the installation, initialization and proper shutdown of AppScale instances and user applications, it also ensures the interaction of all components of the system. In addition, it handles the authorization of user applications (using the same system as in Amazon EC2 with keys and certificates). All interactions between components of the system occur over SSL.

The load balancer usually works on one of the nodes, which is accessible from the outside and provides balancing of requests to application servers (AppServer), as well as monitoring the availability of components and their loading.

The entire infrastructure is based on the deployment of one or more nodes (in terms of the system - GuestVM), on which instances of the AppScale component are executed. Each node is a virtual machine, 64-bit Linux, within which you pick up one or all of the components. It is possible to combine both the balancer and the application server by carrying out the database, and in general all the components on one machine. Only AppController will be required, which will ensure that the node is included in the overall system. It all works on top of Xen, so on one system you can raise multiple servers, simulating your entire Cloud within the same physical server.



The developer uses a special toolkit, AppScale Tools to prepare your application and deploy it to the AppScale environment.

If you take a closer look at AppController, then this is a daemon written in Ruby that implements a SOAP client / server, deployed on each of the nodes in the system, and which is loaded at the start of GuestVM. After the start, the controller starts the load balancer on the main node, after which the other nodes can automatically connect to the system. Next, DatabaseMaster starts (which can start other slave servers) and, finally, AppServer starts. The controller on the main node periodically polls all servers (heartbeat) and collects statistics on the load of basic resources.

AppLoadBalancer (ALB) is a simple Ruby script that runs the Nginx server and three Mongrel replicas of the server. Initially, ALB provides user authorization (in fact - applications), and then processes application requests to AppServer and sends them to one of the free servers, then the application works with its server directly, without the participation of a balancer.

AppServer is compatible with the Python module from the Google AppEngine SDK. But, at a lower level there are differences, in particular, in the data storage system (Database). At the lowest system level, the system is based on Apache Hadoop , a distributed file system that implements MapReduce. On its basis, two data warehouses can be deployed - HBase , a distributed database, an open Google implementation of BigTable , and also a Hypertable . At the same time, one instance of AppServer can only execute one application, so for several applications to work, you can either spread them along the nodes, or you can raise several instances of AppServer on one node.

Google Buffers Protocol over HTTPS is used as a protocol for working with a database. The request from the application first comes to the front-end of the master server, called PBServer, which implements a unified interface for working with data, and also hides the nuances between storage systems. For example, inside PBServer uses other protocols, Thrift (developed by Facebook, now transferred to Apache) to work with HBase. Only four types of command queries are used:

• Put to insert / save data, if there is no table yet, it will be created
• Get - get data based on ID (since this is, in fact, Key-value storage, which we have already written about )
• Query - a query for retrieving data using a query language similar to SQL (probably, this is HQL)
• Delete - delete data by ID.

For example, here is a typical system configuration, which consists of one main node, on which AppLoadBalancer and DatabaseMaster are deployed, used to store both applications and other system information, and three (or more) working nodes, which have one or more AppServer to run applications, as well as a DatabaseSlave component to access the database. The AppController component is mandatory for all system nodes and ensures their communication.



As already mentioned, at the lower system level everything works on top of Xen, which means it can use any environment based on this virtualization system. Now on the project site is offered as an option based on Amazon EC2, and on the basis of another open source development, in fact, open source Amazon EC2 - Eucalyptus . But, if you wish, everything can be deployed without them, using your own server and deploying Xen there . By the way, if you just want to look at Eucalyptus and build the AppScale infrastructure on it - welcome, on the system’s website, after registering, you can touch the system in reality. For experiments, you need to register with the Eucalyptus Public Cloud program, having received up to 4 virtual machines (instances) in the cloud, albeit for 6 hours, after which they will be turned off.

What you want (and, perhaps, will be soon) - access to the usual SQL database (similar to Stax), the ability to work with other platforms (first of all, PHP and Java). But now the system is interesting enough to begin to study and try it.

PS Perhaps this is a bit strange, but the developers themselves warn you - in the current version of AppScale, your data is not saved (it’s not clear whether the data or the execution state) when you disable AppServer, on which your application runs. This will be fixed in the next version.