Pure Python Architecture: A Walkthrough. Part 1

A year ago, my friend Roberto Ciatti introduced me to a concept that Robert Martin calls pure architecture. Uncle Bob talks a lot about this concept at conferences and writes very interesting articles about it. "Pure architecture" is a way of structuring a software system, a set of agreements on different layers and roles of their participants, something more than strict rules.

As he said in his article “Pure Architecture” (Habre), the idea of ​​the approach itself is not new, it is based on a multitude of concepts that have been promoted by many software developers over the past three decades. One of the first implementations can be found in the Boundary-Control-Entity model (Border-Management-Essence), proposed by Ivar Jacobkson in his masterpiece “Object-Oriented Software Engineering: Precedento Oriented Approach”, published in 1992. But Martin also lists other more recent versions of this architecture.

I will not repeat here what he has already explained, (besides, I can’t do it better than him), I’ll just post links to some resources that you can have a look at to familiarize yourself with these concepts:

• "Pure Architecture" - a post by Robert Martin, who succinctly describes the purpose of architecture. It also lists resources with similar architectures.
• The principle of openness-closeness - an article by Robert Martin, not much related to the concept of Pure Architecture, but important for understanding the concept of separation.
• Hakka Labs: Robert “Uncle Bob” Martin - Architecture: The Lost Years video by Robert Martin from Hakka Labs.
• DDD and Testing Strategies by Lauri Taimila
• (coming soon) Robert Martin's book Pure Architecture , published by Prentice Hall.

The purpose of this article is to show how to create web services in Python from scratch using pure architecture. One of the main advantages of this multi-layered design is testability, so I will program by following the TDD approach. The project was originally developed from scratch in about 3 hours. Given the toy nature of the project, some decisions have been made to simplify the final code. I will point out these simplifications and explain them if this seems necessary.

If you want to learn more about TDD in Python, read the articles in this category .

Project Overview

The goal of the project "Rent-O-Matic" (fans of the "Tentacle Day" can catch the link) is to create a simple search engine on top of a data set of objects described by some quantities. This search engine should also allow you to set filters to refine your search.

Data records are warehouses for rent, described by the following values:

• Unique identificator
• Area in square meters
• Rental price in euro / day
• Coordinates (latitude and longitude)

Pure architecture involves the separation of different layers of the system. The architecture itself is described by four layers, which can be implemented by more than four actual code modules. I will give a brief description of these layers.

Entities

This is the level at which domain models are described. Here you should create a class that represents storage rooms, the data about which are stored in the database. Also this class represents information that, in my opinion, is necessary for the execution of business processes.

It is important to understand that the models in this layer are different from the usual models in such frameworks as Django. These models are not related to the storage system, they can not be directly stored or obtained using the methods of these classes. They contain helper methods related to business rules.

Scenarios

This layer contains scripts implemented in the system. In this simple example, there will be only one scenario in which a list of storage rooms is displayed in accordance with the specified filters. Also here you can place a script that shows the detailed information of the warehouse, or any other necessary business process, for example, booking the warehouse, filling it with goods, etc.

Interface adapters

This layer corresponds to the boundary between business logic and external systems and implements the APIs used to exchange data with them. The storage system, as well as the user interface, are external systems for this layer, which need to exchange data with scripts. And this layer provides an interface for a similar data stream. In this project, communication with the view is provided via a JSON serializer, on which an external web service can be built. The storage adapter describes a common API with which different storage systems will work.

External interfaces

This part of the architecture consists of external systems that implement the interfaces defined in the previous layer. Here, for example, you can find a web server that implements (REST) ​​entry points that have access to data (as part of scripting) through a JSON serializer. Also here are implementations of data storage systems, for example, for MongoDB.

API and shades

The word API is extremely important in pure architecture. Each layer can be accessed using an API, which is a fixed set of entry points (methods or objects). By "fixed" is meant "the same for each implementation." Obviously, the API may change over time. Each presentation method will refer to the same scenarios and the same methods to obtain the domain models resulting from the work of this particular scenario. They go up to the presentation layer, where this data will be formatted according to specific information representations, for example, in HTML, PDF, images, etc. If you understand extensibility- based architectures, then you should already understand the basic concept of a split, API-managed component (or layer).

The same concept is valid for the storage layer. Each storage implementation must provide the same methods. When dealing with scripts, you don’t need to think about which storage system is used: it can be a local MongoDB, a cloud storage system or a trivial memory table.

The separation between the layers, as well as the content of each layer, can not always remain unchanged and fixed. A well-designed system must deal with problems such as performance, or other specific requirements. When designing an architecture, it is important to know "what, where and why." This is especially important to know when you "bend" the rules for themselves. Not all questions can be answered unequivocally "black" or "white." Many solutions are “shades of gray”, that is, you yourself have to determine what, why and where you should place.

But, the main thing is not to violate the structure of pure architecture. In particular, you must be firm in the matter of data flow (see the “Crossing the Border” section in Robert Martin’s article). If you break the data stream, you break the whole structure. Let me stress again: never disrupt the flow of data . An example of a data flow violation would be issuing a Python class by some script instead of representing this class (for example, as a JSON string).

Project structure

Let's look at the project structure.

Cookiecutter was used to build the project structure. Quickly go through this part. The rentomatic directory contains the following subdirectories:

• domain
• repositories
• REST
• serializers use_cases

These directories reflect the layered structure presented in the previous section, as well as the directory structure with the tests created so that the tests can be easily found.

Source

You can find the source code in this GitHub repository . Make forks from it, experiment, change it, look for better ways to solve the problem, which I discuss in this article. The source code contains commits with tags, which will allow you to follow the creation process as described in the article. In the article itself, the <Git tag: tag > labels are placed under the headings. A label is just a link to a commit with a GitHub tag, which you can use to go to a githab to see the code without cloning it.

Project Initialization

Git tag: step01

Update: this cookiecutter package creates an environment exactly the same as in this section. The following description explains how to manage dependencies and configurations, take a closer look at this tool, maybe you use it in your next project.

I usually like to deploy a virtual Python environment inside a project, so I will create a temporary virtual environment to set the cookiecutter, create a project, and remove virtualenv. Cookiecutter will ask you a few questions about you and the project to provide the initial file structure. We are going to create our own testing environment, so it’s better to answer, “no” to use_pytest . This is a demonstration project in which we don’t need to publish, so you can answer “no” and use_pypi_deployment_with_travis . The project does not have a command line interface, and you can safely create an author file and use any license.

 virtualenv venv3 -p python3 source venv3/bin/activate pip install cookiecutter cookiecutter https://github.com/audreyr/cookiecutter-pypackage 

Now answer the questions, and then complete the project creation with the following code

 deactivate rm -fR venv3 cd rentomatic virtualenv venv3 -p python3 source venv3/bin/activate 

Get rid of the requirements_dev.txt file that Cookiecutter created for you. I usually store virtualenv dependencies in different hierarchical files to separate production, development, and test environments, so let's create a requirements directory and corresponding files:

 mkdir requirements touch requirements/prod.txt touch requirements/dev.txt touch requirements/test.txt 

The test.txt file will contain specific packages used for testing the project. Since to test the project, you need to install packages for the development environment, you must include the development dependencies in the file.

 -r prod.txt pytest tox coverage pytest-cov 

The dev.txt file will contain the packages used in the development process, as well as install packages for testing and production.

 -r test.txt pip wheel flake8 Sphinx 

(using the fact that test.txt already includes prod.txt ).

Last, the main requirements.txt file will simply import requirements/prod.txt

 -r prod.txt 

Obviously, you can find a project structure that best suits your needs or preferences. This structure is suitable for this project, but I do not force it to be used for all your next projects.

This separation allows you to install a complete development environment on your computer, install only testing tools in a testing environment like Travis, and also reduce the number of dependencies to be installed on production.

As you can see, I do not use tag versions in dependency files. This is because this project will not run on production, so the environment can not be frozen.

Do not forget to install the dependencies inside virtualenv

 $ pip install -r requirements/dev.txt 

Different configurations

The pytest testing pytest must be configured. This is done via the pytest.ini file, which you can create in the root directory (where setup.py is located)

 [pytest] minversion = 2.0 norecursedirs = .git .tox venv* requirements* python_files = test*.py 

To run tests during project development, just run

 $ py.test -sv 

If you want to check the coverage, i.e. the amount of code that runs through tests run

 $ py.test --cov-report term-missing --cov=rentomatic 

If you want to know more about test coverage, have a look at the official documentation of the Coverage.py and pytest-cov packages.

I highly recommend using the flake8 package to check your Python code for PEP8 compatibility. Here are the flake8 settings that need to be placed in the setup.cfg file:

 [flake8] ignore = D203 exclude = .git, venv*, docs max-complexity = 10 

To verify that the code complies with the PEP8 standard, run

 $ flake8 

Flake8 documentation is available here .

Please note that each step in this post creates a test code and coverage of 100%. One of the advantages of pure architecture is the separation between layers, which guarantees a greater degree of testability. However, it should be noted that in this tutorial, in particular in the REST sections, some tests will be omitted in favor of a simpler description of the architecture.

To be continued in Part 2 .