Aggregator Architecture: Web Services Patterns (Part 1)

Today, many web applications and services have been created that have the same goal, but a different approach to execution. Since the information is scattered across the network, users have to visit many similar services in order to increase the effect of the work. For example, the customer wants to place the task on the tender site. In order to increase the number of applications submitted, he spends time on repetitive work: creating an offer and filling out project data on various freelance exchanges. There are sites aggregators that are trying to solve this problem, but their support is becoming increasingly difficult with the emergence of new services theme aggregator. It is necessary to integrate all new functions and data structures that differ from service to service. Fortunately, we are not the first to create and maintain such things: there are already patterns that simplify the support of such applications and allow you to create a flexible architecture. In this article I would like to give an example of the architecture of the aggregator, which allows you to combine the tender platforms for freelancers - such as Odesk, Freelancer, Elance and others.

The main problems faced by developers:

1. Different principles of working with services: some of them provide an API, and others have to work through a bot - imitating human actions.
2. Heterogeneity of responses from services — some return json, others xml, and others return a web page.
3. Stem from 2 - heterogeneity of data structures - for example, the structure of a Project object
   c freelancer looks like this:
   

   <?xml version="1.0" encoding="UTF-8"?> <xml-result xmlns="http://api.freelancer.com/schemas/xml-0.1"> <id>588582</id> <name>sign design web software</name> <url>http://www.sandbox.freelancer.com/projects/PHP-ASP/sign-design-web-software.html</url> <buyer> <url>http://www.sandbox.freelancer.com/users/1353095.html</url> <id>1353095</id> <username>billk89</username> </buyer> <short_descr>Create and upload sign design web software for our client nlsigndesign.com Public internet users must be able to create there own signs online, then save and submit as a file. </short_descr> <jobs> <job>PHP</job> </jobs> ... </xml-result> 

   
   while Odesk responds as follows:
   

     <response> .... <profile> .... <buyer>  <cnt_assignments>0</cnt_assignments>  <op_contract_date>December 18, 2011</op_contract_date>  <timezone>Russia (UTC+06)</timezone> </buyer> <op_title>A social network client app for iPhone/iPad</op_title> <ciphertext>~~05d405f2d5b8eb27</ciphertext> .... <op_required_skills>  <op_required_skill>   <skill>ipad,ui-design,iphone-development</skill>  </op_required_skill> </op_required_skills> .... <op_desc_digest> Hello, We need a social network client app for iPhone/iPad to be developed. It should support Facebook, Twitter and Linked-In. </op_desc_digest> .... </profile> </response> 

   
   

For ease of understanding, I decided to describe the solution to each of the problems in a separate article. In this article I will show how to unify work with various services using the example of two popular freelance exchanges Odesk and Freelancer. The source code is written in PHP5 using the Yii framework.

We create the interface

So, the first thing that needs to be done is to create an interface in order to hide the implementation in each particular class and give the opportunity to call a single set of methods for various services.
')

 interface ServiceInterface { public function authorize(); public function getServiceName(); public function setCredential(ModelCredential $credential); public function searchProjects(); } 

Create service adapters

Further, for each service we create service adapters (a variation of the “Adapter” pattern applied to web services) that will implement this interface. Both classes work with services through the API, but hide the difference in the implementation of calls.

Freelancer Service Adapter:

 class FreelancerService implements ServiceInterface { private $_serviceName = 'freelancer'; private $_service = null; public function __construct(ModelCredential $credential = null) { Yii::import('ext.freelancer-api-wrapper.*'); $this->_service = new Freelancer(Yii::app()->params['freelancer']['token'],Yii::app()->params['freelancer']['secret'], 'http://geeks-board.local/authorizeService/freelancer/?'); if($credential !== null) { $this->setCredential($credential); } } public function authorize() { if(!isset($_GET['oauth_token'])) { $requestToken = $this->_service->requestRequestToken(); $redirectUrl = $this->_service->getRedirectUrl($requestToken); header('Location: ' . $redirectUrl); } else { $oauth_verifier = $this->_service->getRequestTokenVerifier($_GET['oauth_token']); $auth = $this->_service->requestAccessToken($oauth_verifier,$_GET['oauth_token']); $this->_service->oauth->setToken($auth['oauth_token'],$auth['oauth_token_secret']); $this->_service->auth = $auth; return $auth; } return false; } public function getServiceName() { .... } public function setCredential(ModelCredential $credential) { .... } public function searchProjects() { .... } } 

Odesk Service Adapter:

 class OdeskService implements ServiceInterface { private $_serviceName = 'odesk'; private $_service = null; public function __construct(ModelCredential $credential = null) { Yii::import('ext.odesk-api.*'); Yii::import('classes.mapper.service.RequestMapper'); $this->_service = new oDeskAPI(Yii::app()->params['odesk']['secret'], Yii::app()->params['odesk']['token']); if($credential !== null) { $this->setCredential($credential); } } public function authorize() { return $this->_service->auth(); } public function getServiceName() { .... } public function setCredential(ModelCredential $credential) { .... } public function searchProjects() { .... } } 

Note that both classes implement the ServiceInterface interface, but the implementation of the authorize method is different for each. Here it is also necessary to mention the ModelCredential model, which stores the data for authorization. With Oauth, authorization is token and secret.

Create a factory

In the future, we need to easily add new services without changing the code of existing classes (we follow the OCP principle). To do this, use the pattern "Factory Method" .

 class ServiceFactory { public static function create($serviceName, $credentials = null) { $className = ucfirst(strtolower($serviceName)).'Service'; $serviceObject = Yii::createComponent($className, $credentials); if(!($serviceObject instanceof ServiceInterface)) { throw new Exception('Not an instance of Service'); } return $serviceObject; } } 

We also added a check that this class implements ServiceInterface. What is the use of the interface and the check for its implementation by the service classes? If the developer makes a mistake and forgets to implement any method of the interface, php will not allow to run the code in principle. This gives us confidence that the method is implemented. It also gives an understanding of what specific methods the system will need to work. On this occasion, I will share my story.
One of the projects I was working on was tasked with implementing the service adapter for Google+. The project has already integrated Facebook and Twitter adapters. When I opened the class of one of them, I was horrified by the amount of code inside. I did not understand which of the methods I needed to implement in order for the service to work, and which were secondary. I had to compare several classes, check with those developers who wrote this code. It took time. If we had one interface for such service adapters, it would be immediately clear which of the methods needed to be created.

Putting it all together

So, we have prepared the classes of service adapters and the factory that will create them. Let's see how these parts work together:

  Yii::import('application.models.ModelCredential'); //       -   ,    Credentials -   //     . $credentialsRecords = ModelCredential::model()->findAllByAttributes(array( 'type' => 'public' )); if(!empty($credentialsRecords)) { Yii::import('classes.service.ServiceFactory'); $aggregatedProjects = array(); foreach($credentialsRecords as $serviceCredential) { try { //          credentials. $service = ServiceFactory::create($serviceCredential->service, $serviceCredential); $foundProjects = $service->searchProjects(); //       if(is_array($foundProjects)) $aggregatedProjects = array_merge($aggregatedProjects,$foundProjects); } catch(Exception $e) { print_r($e); } } } else { echo 'empty '. "\n"; } 

Total

This architecture allows you to integrate new services more transparently and the developer does not need to remember the differences in the names of methods or their calls.

In the next article I will explain what to do with the heterogeneity of returned responses and data structures.

PS

Simplified factory code by calling Yii :: createComponent (). Thanks Ekstazi .