Channels Philosophy

Much time has passed since my last post about Channels, and at the same time a lot of things happened - the API was developed and stabilized, functionality like “overpressure control” was added, the situation with backends looks much better, especially after the local interaction layer and remote machines became a bit more mature.

On the other hand, however, there was a misunderstanding and concern about the direction in which the Channels develops; Directions that this project sets for Django and Python. When developing the Channels, I had to touch and even fight with my own feelings about this direction, to choose the right set of compromises - sometimes even from two equally correct options.

I did not publicly discuss my rationale and development vision for the Channels as much as I could; I hope this post clarifies my point a little. Let me identify a certain set of problems that I try to solve, explain why I chose the design of architecture that I chose, and talk about the next steps in development.

This is not only about WebSocket

For many people, the reaction to Channels is twofold: firstly, for some reason everyone thinks Channels are only a way to get WebSocket support in Django (this is a thing that happened during development, and not the only reason for creating a project - I’ll explain in more detail later), and -second, then everyone says that the implementation of WebSocket-s through a distributed message processing system is gun shooting at sparrows.

Here they are right. Python's asynchronous capabilities are getting better, and you can simply file a WebSocket server in a few hours using existing libraries (for example, Autobahn). You may have to standardize the interface so that you can communicate between this server and the rest of your project, but this is not difficult.

This was the first path I took, and that was how early versions of the Channels worked (then they were also called django-onair). However, as development and thinking over how to run everything on a serious scale, the real problem became clear.

Understand, handling of the WebSocket-protocol IMHO not a problem; the problem is using these sockets in a larger project. Most scripts use event-oriented sockets: you send data to a socket when something happens outside, for example, the model is saved, the external system changes, or another message is received in another WebSocket.

All these different sources of events live in different places of the expanded project. If you go along the trodden path of launching heaps of servers, each with a web server and a Python code, you will quickly realize that you need a way to communicate between the servers. WebSocket support is one of the tasks, but sending a message to groups of sockets when something happens is really WHAT you write when developing applications.

Imagine a large-scale chat server where different people are logged on to different physical machines. How will you send incoming chat messages to the other people in the same chat, on all other servers? Where will you keep track of who is in the chat? What will you do with foul sessions?

What about the system where you send notifications to users when someone views their profile? These views will probably happen on different servers, so how do you get this event through the server on which the user's WebSocket just fell off?

This is a very complex problem, which Channels is designed to solve; this is not just processing the WebSocket protocol, but the task of building complex applications around WebSockets. Processing messages in distributed systems is really difficult, and I believe that this is one of those tasks that benefit much more from a few common polished solutions than from blunt instructions on how to make a bunch of asynchronous code work together.

Encouraging asynchrony

One of the things Channels does is launch Django in a synchronous manner, which allows you to write all message handlers in the same way. Channels simply executes this code in a fast loop, discouraging you from doing blocking operations in this loop.

The problem is that everyone seems to believe that this is the only way to write code for the Channels. This is not so: it is understood that Channels makes communication between synchronous and asynchronous programs easier in general, offering you the choice of the best tool for the task (and I would argue that for a huge number of simple business scenarios you might want a synchronous code, because it is much easier to write and maintain).

In fact, Channels make writing completely asynchronous applications that communicate with the rest of the Python project easier than ever. In the end, this is all a server with Daphne's WebSocket interfaces. Want asynchronous URL requests? Communication with the Internet of Things? Outgoing sockets? You can write asynchronous code as usual, and then, thanks to the Channels, leave the rest of the code in a familiar synchronous framework and communicate both ways with your specialized code.

When you have a project running on Channels, it becomes easier than ever to add more asynchronous code as a separate process to perform some new task, and have a clear working solution for interacting with other asynchronous and synchronous processes. Great experience comes from community experience and documentation, articles on the use and disassembled examples of others who have already passed all this - after all, all this is done on a single common platform in a single style.

More protocols

Of course, all this brings us back to the idea that Channels are not only about WebSockets. This is a univarsal cross-process event system for Django (and, I hope, generally for Python). WebSockets are one of the protocols whose support is listed in the Channels, but work is already underway on Slack front-end servers (allowing you to add chat integration to the server cluster) and mail (which will allow writing handlers for incoming messages next to your HTTP and WebSocket- code).

Message format specifications also allow making alternative implementations; just as there are many WSGI servers, message formats allow a large number of ASGI-compatible HTTP and WebSocket servers to exist, even working together on the same system.

Some protocols do not require the functionality of broadcast messages, like WebSockets, especially if they do not have stateful connections, but a good channel layer architecture allows them to be routed to the same server. Although the channel layer was designed to be cross-process and network independent, this does not mean that each message should be routed through a central point. The Channels structure was designed to distinguish messages that can be processed locally from those that need to be sent elsewhere.

In fact, with the recent addition of RedisLocalChannelLayer to the asgi_redis package, you can start the server in standard mode as a master worker (in the original socket-terminator and worker pair), and the channel level will try to leave as many messages as possible on one machine, transmitting data on network only when it needs to find a specific socket in order to send something to another user, or send a group message.

Distributed systems are hard

In its core, Channels solves the problem of communication and broadcast messages in distributed systems. Distributed systems are an area where there are no perfect solutions; You will always have to make compromises. One of the options is at-least-once and at-most-once logic (guaranteed delivery and guaranteed absence of duplicates). CAP-theorem about distributed databases is a side effect of other trade-offs.

Channels go to a specific set of trade-offs, with the goal of being the best solution for scripts and protocols that are widely used on the Web, especially for Web Sockets. For example, packet loss and socket closure are preferred over duplicate packets; the client can reconnect, but it is wildly difficult to file a distributed deduplication system for actions, unless you make absolutely all actions idempotent. I hope to write a separate post about which particular compromises I have chosen and which alternatives were there; Believe me, each of them was chosen for a specific reason.

Channels will never be suitable for any occasion; this is an unattainable goal. Instead, a solution is assumed for about 90% of the cases - something that is not always perfect, but generally does what you want; a solution where compromises overlap with the enormous advantages of a single common platform and community, which is formed around the project. In many ways, it’s like Django, which may not be the perfect web framework; it is simply impossible to solve every problem of every developer, but we can solve 90% of the remaining problems that arise constantly among developers, having standards and architecture, the consequence of which is code reuse and ease of use.

The ASGI API, on the basis of which the Channels are written, is intentionally made very flexible. It specifies the minimum required for work, so you can get a fully working environment for different backends, leaving much to the discretion of a particular backend, and thereby gaining greater flexibility in the message passing mechanism. As you grow, your needs will change and be refined; this will help abstraction layer channels, allowing you to grow without going beyond the limits of abstraction, being flexible and at the same time retaining the same basic API that you worked with when you started; channels, groups, sending and receiving.

I do not expect the appearance of sites from Top-100 that would work with the unmodified layer of ASGI channels, in the same way as they would not work with the default delivery of Django. As you grow and refine requirements, you will need a solution that leaves you able to slowly and carefully replace it, and my goal in developing ASGI was that even when you remove all the Channels code, you will be left with an abstraction and architecture that works with much more specialized distributed systems and events. As well as the core of Django, ASGI allows you to deeply modify and replace parts of the code as it grows, and gets out of your way when you no longer need it.

In the end, this is the philosophy of Channels - a solution that is offered not as a panacea, but as a common base to simplify the development of applications that use multiple servers and work with statful protocols like WebSockets. Small teams, web-studios, and medium-sized sites can use it without change; large projects are likely to require finishing the backend of the channel layer, and maybe some of the basic handlers, but can still benefit from the introduction of developers with the abstraction and Channels patterns.

Looking ahead

With all this in mind, what is the route ahead for the Channels and ASGI? WebSocket projects themselves are still in their infancy: very few are deployed on a sufficient scale; not to mention the use of Channels - so we have to go through a period of maturation no matter what. Sites that already use Channels in production, and the feedback I get about it, were mostly positive, so here we are on the right path to adulthood.

Daphne itself is largely based on Twisted code to handle HTTP, and Autobahn for WebSockets are two libraries with a long history of stability — while ASGI is based on our experience and research on scalable event systems in Eventbrite, my previous experiments with distributed messaging, research and examples from IT, and discussions of similar problems with colleagues. ASGI is as durable as possible in a situation where there is no successful open-source example solution.

The feedback I received during the discussion on the inclusion of Channels in Django-1.10 was very valuable (it was not possible to include in the release before the deadline, but the development of the package will continue as an external application for Django-1.10). Some recent changes and developments, such as "backpressure" and "local-and-remote Redis backend", are based on the feedback on the proposal, and I assume that more improvements will be added as more and more projects on Channels come out. in production.

At the same time, I believe that the fundamental abstraction of the architecture, "distributed message transfer", is a fairly robust and competent API for building Django applications of the future, with the needs and complexity of Web applications much more than just processing requests-responses. This is where Django and Python have the opportunity to set an example in the architecture and operation of this type of application.

I’m also interested in getting PEP status for the message formats and the ASGI interface standard, but before that I’m going to try other web frameworks to make sure that this standard really works independently of the framework, as always expected. It is also necessary to test and correct potential problems in actual use scenarios.

I'm not sure what the future holds for the Channels - in an ideal world, this project would allow Django and Python to be a solution for a much larger class of problems than those for which they are used now; Bringing the best of language, framework, and community to a growing audience of developers confronted with writing large systems with stateful protocols. It is also possible that the project will remain a WebSocket management layer for Django, but even for this purpose, it is important that its architecture becomes perfect.

I hope this post has highlighted some context and plans for Channels; Community feedback is very important for the development of the project, and if the Channels helped you, or you have more questions, stay in touch and let me know. It is important that everyone understands both the implementation and the context of the problem being solved - one means nothing without the other. And I hope that in perspective we will have a clear idea of ​​how important they are together.