Scaling web applications with HMVC

For the last ten years, we have been witnessing a second cycle of web design - websites are turning into applications and almost no new projects appear that do not have a certain amount of interactivity. The increasing complexity of software developed for the Internet has necessitated a structured and balanced design of applications.

To date, the most frequently used site design pattern is Model-View-Controller (MVC). Its widespread use is in part due to the success and popularity of the Ruby on Rails framework. Now MVC is almost synonymous with web development among all platforms.

In carrying out tasks that actively load the processor, modern websites increasingly rely on the allocated resources. This, in particular, contributed to the opening of cloud services by Amazon and Google, which allow developers to significantly reduce the load on the processors of their own servers. Each service is usually designed as a separate piece of software that runs within its own domain and uses its own resources.
')
When dealing with modest budgets, it is usually quite difficult to convince clients of the advantages of financing more than one completed piece of software. As my experience shows, many of them hold the opinion that scalability is not an urgent task. They are "looking forward to the day when they have to worry about it . "

To reduce the initial investment, it is usually decided that the application should be designed as a complete program containing all the required functions. If the site quickly gains popularity, it will become a problem. I still have not very pleasant impressions of the refactoring of poorly scalable code bases. In addition, it may require a large amount of resources and money. Ideally, applications should grow as needed and not require large financial expenditures in the process.

Pattern Hierarchical-Model-View-Controller

The Hierarchical-Model-View-Controller (HMVC) pattern is an MVC extension that solves many of the scalability problems already mentioned. It was first described in July 2000 in a JavaWorld blog article entitled “HMVC: A Multi -Layer Pattern for Developing Client Levels” ; although it is believed that the authors actually rethought another pattern - Representation-Abstraction-Management (PAC) - described in 1987. The purpose of the article was to demonstrate how HMVC can be used to create scalable sites and when designing desktop applications with graphical interfaces.

HMVC is a collection of traditional MVC triads operating as one application. Each triad is completely independent and can be performed in the absence of any other. All triad requests should use the controller interface; never connect models or libraries outside their domain. The physical location of the triad on the server does not matter, as long as it is accessible from all other parts of the system. Characteristic features of the HMVC are to encourage code reuse, simplify testing of individual parts of the system, and ensure that the application can be complicated and expanded without much difficulty.

Successfully designing an application based on the HMVC pattern is impossible without splitting its functions into separate systems. Each such system must be one MVC triad, independently controlling storage methods and presentation within a larger HMVC application. Currently, there are several frameworks that support HMVC without additional modules. One of them is Kohana PHP of the third version , which was originally created with a HMVC orientation. Further in all examples I will use this framework.

Kohana 3 uses the built-in Request object to call other controllers. Requests can be both internal - to the application controllers, and external - to web services. In both cases, the same class Request is used. If the MVC triad is moved to another server, only one parameter needs to be changed.

<?php class Controller_Default extends Controller { public function action_index() { //    $internal_request = Request::factory('controller/action/param') ->execute(); //    $external_request = Request::factory('http://www.ibuildings.com/controller/action/param') ->execute(); } } 

An internal request requires a correct path specifying a controller and an action, and creating a request to an external resource is reduced to providing a full URL. This feature makes internal and external requests interchangeable without any difficulties, so taking the triads out of the main server is a fairly simple task.

Using Kohana's Request class to retrieve data from internal controllers can remind you to redirect actions in other frameworks, for example, Zend Framework. Actually, these are two rather different methods. Request objects in Kohana can work in isolation as unique requests, and when redirecting actions, each of them exists within the newly created request. Below is an example to demonstrate this.

Default controller - /application/controllers/default.php

 <?php //     class Controller_Default extends Controller { public function action_index() { //     GET if ($this->request->method === 'GET') { //  POST,  array (0) { empty } var_dump($_POST); //       $log = Request::factory('/log/access/'.$page_id); //  POST     $log->method = 'POST'; //  -    $log->post = array( 'uid' => $this->user->id, 'ua' => Request::user_agent('browser'), 'protocol' => Request::$protocol, ); //   $log->execute(); //  POST,   array (0) { empty } var_dump($_POST); } } } 

Log Controller - /application/controllers/log.php

 <?php //     class Controller_Log extends Controller { public function action_access($page_id) { //     index ( ) //       // array(3){string (3) 'uid' => int (1) 1, string (2) 'ua' => string(10) 'Mozilla... var_dump($_POST); //    Log $log = new Log_Model; //     $log->set_values($_POST) ->save(); } } 

The example above shows the independence available to the Request object. Initially, a GET request causes the index action of the Default controller, in response to which a POST request is created for the Action access of the Log controller. The index action assigns values ​​to three variables that are not accessible to the global $_POST array from the first controller. When the second request is executed, $_POST already contains variables that we have made available to it. Note that after $log->execute() in the Index controller, the data from $_POST disappears. In other frameworks, the implementation of such an interaction requires the creation of a new query using, for example, Curl.

Gazouillement, microblogging service

Demonstrate the power of HMVC will help a fictional short message service (statuses) called Gazouillement , which is essentially the equivalent of Twitter. Suppose that it was designed based on a service-oriented architecture (SOA) so that the web interface and the components for working with messages and relationships are separated.

At first, traffic will be relatively small, because the service is new and people still do not know about it. Therefore, nothing terrible will happen if, for the time being, all the application logic will be executed on one server.

Let's write a controller to display the user's home page. It will contain his most recent statuses and a list of people whom the user has listed.

Index Controller - /application/controllers/index.php

 <?php //    http://gazouillement.com/samsoir/ class Controller_Index extends Controller { public function action_index() { //   (samsoir)   $user = new Model_User($this->request->param('user')); //    ,   404 if ( ! $user->loaded) throw new Controller_Exception_404('Unable to load user :user', array(':user' => $this->request->param('user'))); //      xhtml $messages = Request::factory('messages/find/'.$user->name.'.xhtml') ->execute() ->response; //      xhtml $relations = Request::factory($user->name.'/following.xhtml') ->execute() ->response; //        //     ,    $this->request->response = View::factory('user/home', array( 'user' => $user, 'messages' => $messages, 'relations' => $relations, )); } } 

Now let's look at what Controller_Index::action_index() does. Initially, the action tries to load the user based on the user parameter from the URL. If this fails, page 404 is displayed. Then, in order to receive messages in xhtml format, a new request is created, which in the request URI uses the user name from the corresponding class property. Similarly, in the same format, user relationships are requested. As a result, an object of the class View is created with the user, its messages and relations attached to it.

When loading existing messages and relationships through a new request, the entire application logic for each service remains abstracted from the website. This architecture provides two significant advantages compared with the usual implementation of controllers:

1. No part of the message service operation logic concerns the controller. The only requirement is that the result must be in xhtml format. During the execution of the controller, no additional library or extension was loaded.
2. Each controller is responsible for only one specific task, thus significantly simplifying the writing of unit tests for them.

Because of the level of abstraction that has just been demonstrated, it is impossible to see what the services are doing. Therefore, let's pay attention to the message service controller. Let's start with the route in the boot file, which will be responsible for the internal management of message requests. Kohana's Route class parses internal URLs by associating the passed URI elements with controllers, actions, and parameters.

Route Setup - /application/bootstrap.php

 Route::set('messages', 'messages/<action>/<user>(<format>)', array('format' => '\.\w+')) ->defaults(array( 'format' => '.json', 'controller' => 'messages', )); 

This creates a route for a messaging service that is currently located within the domain of the main application. The request via the messages / find / samsoir.xhtml address will be redirected to the messages controller, whose find() action will be passed as the parameters 'user' => 'samsoir' and 'format => '.json' .

Messages Controller - /application/controllers/messages.php

 <?php class Controller_Messages extends Controller { //  ,    protected $supported_formats = array( '.xhtml', '.json', '.xml', '.rss', ); //      protected $user; //      //       public function before() { //        if ( ! in_array($this->request->param('format'), $this->supported_formats)) throw new Controller_Exception_404('File not found'); //     $this->user = new Model_User($this->request->param('user')); if ( ! $this->user->loaded()) throw new Controller_Exception_404('File not found'); return parent::before(); } //      public function find() { //         1:M $messages = $this->user->messages; //    ,   prepare    $this->request->response = $this->_prepare_response($messages); } //      protected function _prepare_response(Model_Iterator $messages) { //  ,      switch ($this->request->param('format') { case '.json' : { $this->request->headers['Content-Type'] = 'application/json'; $messages = $messages->as_array(); return json_encode($messages); } case '.xhtml' : { return View::factory('messages/xhtml', $messages); } default : { throw new Controller_Exception_404('File not found!'); } } } } 

The process of receiving user messages in detail is shown in the Controller_Messages Controller_Messages . All methods and properties are bound only to the context of messages, including user relations. Let's look at the code of this controller to understand what is happening.

The Request object always calls the before() method before() calling the specified action. This allows you to perform before the main work various routine operations. The before() method first checks the requested format for maintainability and then the user's name for correctness. If before() completes without thrown exceptions, then the Request object will call the find() action. It loads user messages as a Model_Iterator object. It should be noted that the iterator will be empty if no messages are found in the user relation. It ends with the transfer of the message iterator to the _prepare_response() method, which correctly formats the data for output and sets all the necessary headers.

The Controller_Index and Controller_Messages were both executed in a single request to the application. On the existence of each other, however, they did not know. Each developer is able to read the current code base and understand what is being executed and where. This is another great feature of HMVC - ease of maintenance.

After completing work on the remaining services code, company managers expressed satisfaction with the first stage of development and allowed deployment on an open server for limited beta testing. After a couple of weeks the application is ready for mass use. Over the coming months, the audience will constantly grow, and the whole site architecture will be improved and optimized.

Stephen Fry Effect

Stephen Fry ( @stephenfry ) is one of the most famous Twitter users with over 2 million followers. He is able to disable sites just by publishing the URL in his microblogging, creating a DDOS attack on the server.

Gazouillement has been actively recruiting users for the past few months. The response time has therefore increased, but it is still within the limits of the permissible. Suddenly, a certain Twitter user with a huge number of followers, like Stephen Fry, posts a message with a link to Gazouillement . And here the service begins real problems.

The first will be a large amount of traffic. The application instead of the usual several hundred requests per second will have to process thousands. In this scenario, Gazouillement will begin to experience overload and may “put” the server. It will become apparent that the application needs some optimization and improvement in order to cope with markedly increasing traffic.

Code analysis to improve performance is not an easy task. In previous articles on TechPortal , utilities like XHProf , designed for deep analysis of memory and processor during code execution, were considered. Kohana PHP has a built-in performance analysis engine called Profiler. But it is a supplement, not a replacement for XHProf and similar utilities. Profiler allows developers to find slow spots in running code, which can then be examined in detail using XHProf or another Codebench module embedded in the framework.

You can enable Profiler by changing only one parameter in the bootloader.

Loader - /application/bootstrap.php

 //-- Environment setup -------------------------------------------------------- Kohana::$profiling = TRUE; 

Finally, add the Profiler view to the end of the response displayed by the controller. This is best done through the after() method, then the profiler will be shown for all actions.

Index Controller - /application/controllers/index.php

 public function after() { //       $this->request->response .= View::factory('profiler/stats'); } 

After turning on the profiler, its information starts to appear at the bottom of each action from this controller. It is best to create a class that inherits Controller, and insert the profiler view into the after() method of this class. Now all system controllers display debug profiler information at design time.



The profiler reports on the framework's work, grouping together various contexts (initialization, requests, database accesses, etc.) and displaying the corresponding processor time and amount of allocated memory. Seeing in front of you the information on each created query, it is much easier to calculate those that take too much time. And when this happens, you can already use tools like XHProf for more detailed analysis.

Scale Gazouillement

Gazouillement's global performance analysis showed that great difficulties arise in receiving messages. Although the development team refactored and optimized the corresponding MVC triad, the required performance indicators were never obtained. After not particularly effective improvement of processors and memory on the server, company executives agree to scale the application horizontally, starting with the message system.

With the MVC architecture, a new service usually has to go through the stages of design, development, acceptance testing and implementation. This process can take months and require significant investment. But in Gazouillement , in essence, a new piece of software will be integrated, not a new program.

In HMVC-based applications, new services based on the existing codebase can be created in the shortest possible time intervals. Since all interaction with the message service was carried out through the main controller, only the message requests need to be changed.

The messaging service was moved to another server and optimized for working with the database. This new server performs only message-related actions, thus markedly increasing the speed of all such operations.

Index Controller - /application/controllers/index.php

 <?php //    http://gazouillement.com/samsoir/ class Controller_Index extends Controller { protected $_messages_uri = 'http://messages.gazouillement.com'; public function action_index() { //   (samsoir)   $user = new Model_User($this->request->param('user')); //    ,   404 if ( ! $user->loaded) throw new Controller_Exception_404('Unable to load user :user', array(':user' => $this->request->param('user'))); // --    -- //  URI     $messages_uri = $this->_messages_uri.'/find/'.$user->name'.xhtml'; //       xhtml $messages = Request::factory($messages_uri) ->execute() ->response; // --    -- //      xhtml $relations = Request::factory($user->name.'/following.xhtml') ->execute() ->response; //        //     ,    $this->request->response = View::factory('user/home', array( 'user' => $user, 'messages' => $messages, 'relations' => $relations, )); } } 

The code highlighted a tiny change in the controller Controller_Index::action_index() . The request for messages now goes not to the action of the internal controller, but to the message service that runs on the sub-domain messages.gazouillement.com . What would normally lead to a revision of the entire architecture of the project would be a small addition to it. And since so little code has changed, associative and acceptance testing will be much shorter.

This is just one example of how the Hierarchical-Model-View-Controller pattern allows developers to design web applications that can be expanded from the very beginning both vertically and horizontally. We saw how the Request object allows controllers to return valid data for each context using a simple interface. You saw how it makes the removal of a part of the application outside the server very easy. And the cost of scaling was relatively small due to the small number of changes, which the company leaders were very pleased about.

More information about Kohana can be found on the framework site , and Kohana PHP 3 files can be downloaded from Github. There is also api documentation: http://v3.kohanaphp.com/guide .

The query class used in this example is currently available in the Kohana kernel development branch in my personal github account - http://github.com/samsoir/core . If the official distribution of Kohana PHP 3.0 is used, a modified class extension is required for queries .