Declarative web programming

What is declarative programming? Wikipedia will tell us:

Declarative programming is a programming paradigm in which the specification of the solution to a problem is specified, that is, it describes what the problem and the expected result represents.

Further in the article we will talk about how to use this paradigm in modern web programming. In particular, I would like to raise the issue of validation / verification of input data for web services. Examples will be in php, since this language is closest to me in professional terms.

Simple form

Let's start with a simple one - processing data from web forms. Subject long jaded, I know, but nonetheless. Suppose we have a user authorization form on the site:

In this case, on the server we will have a certain endpoint that will process requests from this form. Immediately a small caveat - it is a RESTful service, i.e. the form in this case is processed by the JS application. Let's try to describe it:

• Receives POST request data
• Waits for content like application/x-www-form-urlencoded
• Takes 2 parameters - username and password
• Returns application/json; charset=utf-8 application/json; charset=utf-8
• Must return a “distinct” response in case of an authorization error.
• In case of successful authorization, return user profile data

Swagger (Open API standard)

Such a description is great for a tester from your team, but is hardly easy to automate. And with that, Swagger, a tool for developing and testing APIs, will help us. Swagger is based on JSON-schema (we'll talk about it later), an open standard for describing JSON objects.

Declaration

If we take as a basis the list proposed above and “translate” it into the Swagger format, we will get something similar:

 { "swagger": "2.0", "host": "example.com/login.php", "basePath": "/v1", "tags": [{"name": "login", "description": "User login form"}], "schemes": ["http", "https"], "paths": { "/user/login": { "post": { "tags": ["login"], "summary": "Authenticate the user", "consumes": ["application/x-www-form-urlencoded"], "produces": ["application/json; charset=utf-8"], "parameters": [ { "in": "formData", "name": "email", "description": "User email", "required": true, "schema": {"type": "string","maxLength": 50,"format": "email"} }, { "in": "formData", "name": "password", "description": "User password", "required": true, "schema": {"type": "string","maxLength": 16,"minLength": 8} } ], "responses": { "200": { "description": "successful login", "schema": { "type": "object", "properties": [ { "name": "status", "schema": {"type": "string"} } ], "example": {"status": "ok"} } }, "422": { "description": "Invalid login data", "schema": { "type": "object", "properties": [ {"name": "status","schema": {"type": "string"}}, {"name": "code","schema": {"type": "integer"}} ], "example": {"status": "fail","code": 12345} } } } } } } } 

So, having a swagger specification for our endpoint, we can start testing the back-end without a ready-made front-end , which often significantly speeds up and simplifies interaction within the team. Using Swagger UI , you can generate requests to the back-end directly in the browser.

NOTE: to do this, you must place the Swagger UI files on the same domain as your backend or enable the above cross-domain requests. CORS Cheat Sheet .

Probably the most enjoyable part in the Swagger declaration is the ability to reuse identical objects through definitions . In this example, we have not touched them, but they are in the examples on the official website. Since Swagger is based on the JSON schema , we’ll look at the defnitions example below when validating JSON data.

In the case of complex input data, there is a very convenient opportunity to specify an example for a particular object. If you use Swagger UI, it will be automatically substituted into a form for testing, which reduces the time and the likelihood of error without typing everything manually.

http://petstore.swagger.io/#/user/createUsersWithArrayInput

IDE support

To make working with the swagger file even more enjoyable, you can install a plugin for your favorite IDE:

• IntelliJ platform
• Eclipse

I did not manage to find a plugin for NetBeans, although I’m pretty sure that it is there. If you know where to get it - I will be grateful for the link.

Generation

In order not to turn the support of the Swagger file into a separate monotonous and tedious task, you can use the Swagger JSON file generator based on your source code. Thus we kill several “birds with one stone”:

• Developers always have up-to-date information on input data to a specific controller / method.
• External documentation remains relevant and changes with the code.
• You can avoid creating a JSON file manually and possible errors in it

Swagger php annotation example

 /** * @SWG\Post( * path="/product", * summary="Create/add a product", * tags={"product"}, * operationId="addProduct", * produces={"application/json"}, * consumes={"application/json"}, * @SWG\Parameter( * name="body", * in="body", * description="Create/alter product request", * required=true, * type="object", * @SWG\Schema(ref="#/definitions/Alteration") * ), * @SWG\Response( * response=201, * description="Product created", * @SWG\Schema(ref="#/definitions/Product") * ), * @SWG\Response( * response=400, * description="Empty data - nothing to insert", * @SWG\Schema(ref="#/definitions/Error") * ), * @SWG\Response( * response=422, * description="Product with the specified title already exists", * @SWG\Schema(ref="#/definitions/Error") * ) * ) */ 

An example of a command to generate a JSON file:

 ./vendor/bin/swagger --output wwwroot/swagger.json //   public  --exclude vendor/ //    

To summarize : using Swagger we declared how our enpoint will work for the outside world.

Having such an intermediate UI, you can generate all sorts of input data for our enpoint, and make sure that it works exactly as intended. At this stage, our UI is “emulated”, go to the server part.

For validation and data cleanup in a declarative manner, the native function filter_var_array is filter_var_array :

 $data = filter_var_array($_REQUEST, [ 'email' => FILTER_SANITIZE_ENCODED, 'password' => FILTER_SANITIZE_ENCODED ]); $result = (false === $data) ? ['status' => 'fail', 'code' => 12345] : ['status' => 'ok']; die(json_encode($result)); 

It is clear that this example is very primitive. We now turn to a more complex example.

Json

To validate JSON data, we will use the same JSON-schema. Suppose we need to collect data from students of the school for further operational use. The form will contain information about the student, parents, contact details. Validating the data we will be justinrainbow/json-schema library.

And here is our scheme:

 { "$schema": "http://json-schema.org/draft-04/schema#", "title": "EntryPoll", "type": "object", "definitions": { "contacts": { "type": "object", "properties": { "email": {"type": "string", "format": "email"}, "phone": {"type": "string", "pattern": "^\\+7\\(845\\)[0-9]{3}-[0-9]{2}-[0-9]{2}$"} } }, "name": { "type": "object", "properties": { "firstName": {"type": "string"}, "lastName": {"type": "string"}, "gender": {"type": "string", "enum": ["m", "f", "n/a"]} }, "required": ["firstName", "lastName"] } }, "properties": { "student": { "type": "object", "description": "The person who will be attending classes", "properties": { "name": {"$ref": "#definitions/name"}, "contacts": {"$ref": "#definitions/contacts"}, "dob": {"type": "string","format": "date"} }, "required": ["name", "dob"] }, "parents": { "type": "array", "minItems": 1, "maxItems": 3, "items": { "type": "object", "properties": { "name": {"$ref": "#definitions/name"}, "contacts": {"$ref": "#definitions/contacts"}, "relation": { "type": "string", "enum": ["father", "mother", "grandfather", "grandmother", "sibling", "other"] } }, "required": ["name", "contacts"] } }, "address": { "type": "object", "description": "The address where the family lives (not the legal address)", "properties": { "street": {"type": "string"}, "number": {"type": "number"}, "flat": {"type": "number"} } }, "legal": { "type": "boolean", "description": "The allowance to use submitted personal data" } }, "required": ["student", "address", "legal"] } 

The JSON-schema format supports a variety of data types, ranging from simple string and int to complex and widely used data types: date-time , email , hostname , ipv4 , ipv6 , uri , json-pointer . As a result, quite simple shapes can be built from simple “bricks”.

Example php code for validation:

 (new \JsonSchema\Validator())->validate( json_decode($request->getBody()->getContents()), //   (object) ['$ref' => 'file://poll-schema.json'], //   //  Exception   \JsonSchema\Constraints\Constraint::CHECK_MODE_EXCEPTIONS ); 

XML

Everything is much simpler than with JSON, but for some reason most of the developers with whom I had the opportunity to work either do not know about this feature, or simply ignore it. We will need native DOMDocument and the lib-xml extension, available by default in most php assemblies.

To begin with, we will form an example of the request, which we will validate. Suppose we have a payment system service with a complex configuration and we want to send a request to it to form a user interface link. The request will include information about the allowed payment systems, the cost of the subscription, user information, etc.

 <?xml version="1.0" encoding="UTF-8"?> <paymentRequest> <forwardUrl>https://www.example.com</forwardUrl> <language>EN</language> <userId>13339</userId> <affiliateId>my:google:campain:5478669</affiliateId> <userIP>192.168.8.68</userIP> <tosUrl>https://www.example.com/tos</tosUrl> <contracts> <contract name="14-days-test"> <description>14 days test</description> <note>Is automatically converted into a basic package after expiration</note> <termOfContract period="days">14</termOfContract> <contractRenewalTerm period="month">1</contractRenewalTerm> <cancellationPeriod period="days">14</cancellationPeriod> <paytypes> <currency type="EUR"> <creditCard risk="0"/> <directDebit risk="100"/> <paypal risk="85"/> </currency> <currency type="USD"> <creditCard risk="58"/> </currency> </paytypes> <items> <item sequence="0"> <description>14 days test</description> <dueDate>now</dueDate> <amount paytype="creditCard" currency="EUR">1.9</amount> <amount paytype="directDebit" currency="EUR">1.9</amount> <amount paytype="paypal" currency="EUR">1.9</amount> <amount currency="USD">19.9</amount> </item> </items> </contract> </contracts> </paymentRequest> 

And now we check the received request:

 function validateString($xml = '') { $dom = new \DOMDocument('1.0', 'UTF-8'); // load, validate and try to catch error if (false === $dom->loadXML($xml) || false === $dom->schemaValidate($this->schema)) { $exception = new ValidationException('Invalid XML provided'); $this->cleanUp(); throw $exception; } return true; } 

“Out of the box” we have support for the following types: xs:string , xs:decimal , xs:integer , xs:boolean , xs:date , xs:time and a few more . But the good news is that we are not limited to them - you can create your own data types by expanding or narrowing existing ones, combining them and so on and so forth. The following is an example schema for the above XML request:

 <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE schema SYSTEM "https://www.w3.org/2001/XMLSchema.dtd"> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:include schemaLocation="xsd/paymentRequest.xml" /> <xs:element name="paymentRequest" type="PaymentRequest" /> </xs:schema> 

This document contains the declaration of the parent element. Consider the child, the connected document separately:

 <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE schema SYSTEM "https://www.w3.org/2001/XMLSchema.dtd"> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> <xs:include schemaLocation="type/urls.xsd" /> <xs:include schemaLocation="type/bank.xsd" /> <xs:include schemaLocation="type/address.xsd" /> <xs:include schemaLocation="type/ip.xsd" /> <xs:include schemaLocation="type/contract.xsd" /> <xs:include schemaLocation="type/voucher.xsd" /> <xs:include schemaLocation="type/riskinfo.xsd" /> <xs:include schemaLocation="enum/layouts.xsd" /> <xs:include schemaLocation="enum/schemes.xsd" /> <xs:include schemaLocation="enum/languages.xsd" /> <xs:complexType name="PaymentRequest"> <xs:annotation> <xs:documentation>Initial create enrollment request</xs:documentation> </xs:annotation> <xs:all> <xs:element name="tosUrl" type="TosUrl" /> <xs:element name="serviceHotline" type="xs:string" minOccurs="0" /> <xs:element name="userId"> <xs:simpleType> <xs:union> <xs:simpleType> <xs:restriction base='xs:string'> <xs:minLength value="1" /> </xs:restriction> </xs:simpleType> <xs:simpleType> <xs:restriction base='xs:integer' /> </xs:simpleType> </xs:union> </xs:simpleType> </xs:element> <xs:element name="userIP" type="ipv4" /> <xs:element name="contracts" type="ContractsList" /> <xs:element name="layout" type="AvailableLayouts" minOccurs="0" default="default" /> <xs:element name="colorScheme" type="AvailableSchemes" minOccurs="0" default="default" /> <xs:element name="forwardUrl" type="ForwardUrl" minOccurs="0" /> <xs:element name="language" type="xs:string" minOccurs="0" default="DE" /> <xs:element name="affiliateId" type="xs:string" minOccurs="0" /> <xs:element name="voucher" type="xs:string" minOccurs="0" /> <xs:element name="userBirth" type="xs:date" minOccurs="0" /> <xs:element name="userAddress" type="UserAddress" minOccurs="0" /> <xs:element name="userBankaccount" type="BankAccount" minOccurs="0" /> <xs:element name="userRiskInfo" type="UserRiskInfo" minOccurs="0" /> <xs:element name="vouchers" type="VouchersList" minOccurs="0" /> <xs:element name="voucherCodes" type="VoucherCodesList" minOccurs="0" /> </xs:all> </xs:complexType> </xs:schema> 

As another bonus, you can connect (include) XSD documents from one to another. Thus, once declaring a custom data type, you can then use it in several schemes. For details, again, see the repository with examples. And you can also include comments with the documentation directly in the body of the document.

In the last example, we also connect a whole bunch of smaller XSDs. As you can see, the description of complex objects can include both complex complex types and more simple, basic ones. In order to fully cover the topic, consider an example of one of the simple composite types:

 <?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE schema SYSTEM "https://www.w3.org/2001/XMLSchema.dtd"> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:vc="http://www.w3.org/2007/XMLSchema-versioning" vc:minVersion="1.0"> <xs:simpleType name="ipv4"> <xs:annotation> <xs:documentation>An IP version 4 address.</xs:documentation> </xs:annotation> <xs:restriction base="xs:token"> <xs:pattern value="(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])"/> <xs:pattern value="[0-9A-Fa-f]{8}"/> </xs:restriction> </xs:simpleType> </xs:schema> 

One of the pleasant moments of working with XSD schemes is that it has existed for quite some time and, if you wish, you can find entire libraries of someone composed and validated user data types. In particular, the example above is taken from the email mailing of December 2005.

LibXML Error Handling

I think that I’m not the only one who would not be satisfied with the “Invalid XML provided” error, especially if we are talking about debugging and testing tools. Therefore, let's expand the information about errors in the document. As a result, we want to get a clear message for further action and the number of the line containing the error.

 /** * @link http://php.net/manual/en/domdocument.schemavalidate.php */ class Xml { /** * @var string */ protected $schema; /** * @var bool */ protected $errors; /** * Xml constructor. * @param string $schemaPath */ public function __construct($schemaPath = null) { $this->schema = null === $schemaPath ? __DIR__ . '/../config.xsd' : $schemaPath; $this->errors = libxml_use_internal_errors(true); } /** * Restore the values and remove errors */ protected function cleanUp() { libxml_use_internal_errors($this->errors); libxml_clear_errors(); } /** * @param string $xml * @return bool * @throws InvalidArgumentException * @throws ValidationException */ public function validateString($xml = '') { $dom = new \DOMDocument('1.0', 'UTF-8'); // load and try to catch error if (false === @$dom->loadXML($xml) || false === @$dom->schemaValidate($this->schema) ) { $exception = new ValidationException('Invalid XML provided'); $exception->setErrorCollection(new ErrorCollection(libxml_get_errors())); $this->cleanUp(); throw $exception; } return true; } } 

The code is deliberately shortened for readability. The basic idea is to collect libxml errors, “wrap” them into a custom collection of special classes with error information. Classes and collections, in turn, are implementations of \JsonSerializable so that they can be transferred to the client with the necessary degree of availability of information. For example, we excluded from the standard \LibXMLError information about the file in which the error occurred.

 /** * Decorator for native LibXmlError to hide file path. */ class LibXMLError implements \JsonSerializable { /** * @var int */ protected $code; /** * @var int */ protected $line; /** * @var string */ protected $message; /** * LibXMLError constructor. * @param \LibXMLError $error */ public function __construct(\LibXMLError $error = null) { if (null !== $error) { $this->line = $error->line; $this->message = $error->message; $this->code = $error->code; } } /** * @return array */ public function jsonSerialize() { return [ 'code' => $this->code, 'message' => $this->message, 'line' => $this->line, ]; } } 

Testing

As we have said, in the case of using Swagger, manual testing can be performed directly in the browser in the Swagger UI. And in order to automate the validation testing, you can write 2 very simple tests. For writing unit tests we will use phpUnit . I will cite the code only for XML, but the same approach is perfectly ported to JSON:

Check for no errors on valid XML / JSON

 class SuccessfulTest extends \PHPUnit_Framework_TestCase { public function setUp() { $this->validator = new XmlValidator(XSD_SCHEMA_PATH); } public function tearDown() { $this->validator = null; } /** * @param string $filename * @param string $xml * @dataProvider generateValidDomDocuments */ public function testValidateXmlExample($filename, $xml = '') { try { $this->assertTrue($this->validator->validateString($xml)); } catch (ValidationException $ve) { $this->fail($ve->getMessage().' => '.json_encode($ve, JSON_PRETTY_PRINT)); } } public function generateValidDomDocuments() { $xml = []; $directory = new \DirectoryIterator('valid-xmls-folder'); /** @var \DirectoryIterator $file */ foreach ($directory as $file) { // skip non relevant if ($file->isDot() || !$file->isDir() || 'xml' === $file->getExtension()) { continue; } $xml[] = [$file->getBasename(), file_get_contents($file->getRealPath())]; } return $xml; } } 

Check for the presence of an expected error for invalid XML / JSON

 class SuccessfulTest extends \PHPUnit_Framework_TestCase { public function setUp() { $this->validator = new XmlValidator(XSD_SCHEMA_PATH); } public function tearDown() { $this->validator = null; } /** * @param string $filename * @param string $xml * @expectedException \Validator\Exception\ValidationException * @expectedExceptionCode 422 * @dataProvider generateInvalidDomDocuments */ public function testValidateXmlExample($filename, $xml = '') { $this->assertFalse($this->validator->validateString($xml)); } /** * @return array */ public function generateInvalidDomDocuments() { $xml = []; $directory = new \DirectoryIterator('valid-xmls-folder'); /** @var \DirectoryIterator $file */ foreach ($directory as $file) { // skip non relevant if ($file->isDot() || !$file->isDir() || 'xml' === $file->getExtension()) { continue; } $xml[] = [$file->getBasename(), file_get_contents($file->getRealPath())]; } return $xml; } } 

I have it all. Enjoy your declaration!

PS I would be grateful for additions / comments.