On the intricacies of the "encrypted pipeline" in the process of developing an IMAP client on Scala + Akka + Spray
Most recently, I switched from my beloved object-oriented C ++ to a new and not yet completely understandable functional Scala. The reasons for the transition - a completely separate story. But one of them was the presence of a fairly good, judging by the reviews, support for the model of actors - with the help of the Akka library. I have long dreamed of trying out on my own experience all the described advantages of this technology, and the existing implementations in C ++ (CAF_C ++ and Theron), which I turned a little into small tests, turned out to be quite raw for my needs. The most canonical (in my opinion) solution of the model of actors - Erlang, I dismissed, because I considered that I would need too much time to master it, and it’s not a fact that I can find third-party libraries I need for this far from universal language. Therefore, as a result, my choice fell on Scala in conjunction with Akka, especially since I once began to study Scala long ago, but abandoned it because of inexpediency. However, as it turned out, this time I chose not the best time for my experiment, which I was convinced only after a sufficiently substantial part of the project was already completed.
Development from the very beginning went at an accelerated pace: almost all the necessary functionality that was needed for my application was available in abundance on the Internet as third-party libraries. And the lack of what is not written specifically for Scala has been more than compensated for by the rich variety of different components for Java. However, the problem, as always, arose in the most unexpected place.
The fact is that at some point my application needs to connect to the IMAP server to read and process the received mail messages. And since the model of actors implies asynchronous work with the network, I needed a library that could connect to the mail server and receive mail asynchronously in order to fit in beautifully with my new application structure. After a brief search, I came across the akka-camel module, which allows you to use the apache-camel library as a message channel for actors. And camel, as it turned out, is able to connect, among other things, to mail servers. In addition, when specifying the required connection parameters, camel can read only fresh (with the \ Recent flag) messages, delete read messages, or copy / move them into a specially created folder. I couldn’t dream of more. To start working in SBT, all that was needed was to mention in the dependencies akka-camel, camel-core and camel-mail.
Creating an actor and connecting it to the IMAP server took literally a few lines of code. And here in the application log fell out the text of the message, which I myself sent to the mail for the test. I started rubbing my hands in satisfaction and thinking about the next task, but I decided to try to connect to the workbox, which would receive mail for processing as a result, just in case. And here my actor threw an exception and "fell." As it turned out, he could not parse the server response correctly. On the Internet, I did not find any information about this error and possible solutions. And a bit sad. I didn’t really want to spend time studying protocol specifications and writing my client. And I, reluctantly, in the name of saving time, decided to retreat from the intended course of complete asynchrony and use the synchronous blocking library JavaMail. However, in the same place with the same exception, this library also fell. After that, I had already firmly decided that giving up ideals was a way for weaklings and lazy people, but I would still write my own IMAP client, with asynchrony and actors. Moreover, I did not need to implement the entire IMAP, the functionality required was very limited: authorization, selecting the INBOX folder, getting a list of messages, reading a specific message, copying the message to another folder and deleting.
')
I didn't particularly have to choose where to start. As you know, Akka developers at some point abandoned Netty for network I / O in favor of Spray. Further, the development of Akka and Spray is so closely intertwined that even their documentation mutually refers to each other, and pieces of code from spray.io smoothly migrated to akka.io. And it was here that the main trick was waiting for me: once, during the development of version 2.x, Akka adopted the idea of ​​channels used in Spray (they are also “pipelines”, they are also pipelines in English), which allow ( according to the authors, it is easy to create network protocols that support back pressure — that is, the ability to “screw the valve” so that the “pipes do not clog” in case the recipient does not have time to process the data stream from the sender, filter, divide, multiply the data and what else with them not to do. But something in these "pipelines" went wrong, and they, without leaving the experimental stage, were declared deprecated. The last announced innovation from Akka, whose goal is to fully replace the channels, is “reactive streams” (reactive streams), which have already been written on Habré. But since this innovation is still in the announcement stage, in the latest version of akka 2.3.6 it is not there yet, but there are no channels anymore. The channels remained in the spray, but all the documentation on them leads to the outdated documentation Akka Version 2.2.0-RC1, which no longer reflects the entire current reality. And the new Akka documentation says that the channels remained in the Spray. In general, the first version of my email client turned out to be approximately similar to the long-suffering child of Frankenstein - collected from different pieces of fragmented and sometimes contradictory documentation. I immediately decided to abandon the “pipeline” in view of the seemingly transcendental complexity of this concept, and therefore my client worked directly with the stream of characters from the server in the form of a ByteString. More precisely, with the scraps of this stream, since no one guarantees that the answer of interest will come in one single piece, or two answers will not blindly form together. In some miraculous way, through a bunch of mathews poured into the monitor and rewritten pieces of code, I managed to tie SSL / TLS encryption to my actor with the help of several pieces of code found in various places. The code I found only in some (very obsolete) of a specific of many versions of official documentation refused to work.
With the implementation of each subsequent stage of its modest functionality, my client became more and more monstrous. In the end, after the next iteration, at 3 am, I realized that it was impossible to continue to live like this, wrote to TODO with annoyance to try all the blocking JavaMail, but through POP3, without the ability to move messages through folders, and went to sleep.
However, being a stubborn scumbag, the next morning (or rather, at lunch), instead of trying to tame JavaMail, I first got into the Spray source code on the github. I spent a couple of days studying them and adapting the information received to fit my needs, but the time spent paid off handsomely. First of all, in the source, I came across the ConnectionHandler class, which is not described anywhere in the documentation, which made my life and creation much easier for me and my creation, putting a lot of things in their places. It was precisely by studying the use of this class in spray-can that I understood how and where these pipelines could be used, of which I found out from the documentation only what tasks they are meant to solve, but not how they do it. In the same place, in source codes, I found out how it is possible to “connect pipes” - that is, to combine several “pipes” (stages) of the pipeline (PipelineStage) into one “pipeline”, to which this leads, and how it is supposed to be used. And I also found out why and how exactly SSL encryption, which I had bolted on the eve, works, which until that moment had remained for me a black box that “just works and doesn’t have to go”.
For those who are interested in the details: the “pipeline” is made up of parts, in the original they are called “stages” or “stages” (English stages), but I will call them “pipes” to maintain figurativeness. These "pipes" are combined in the code with the help of the operator >>, the order matters. The first are the "pipe", the most "close" to the client, the last - to the server. That is, everything that goes from the client goes through the "pipeline" from left to right, from the server - on the contrary, from right to left. For example, the “pipe” performing the encryption is indicated last, therefore everything that the client sends to the “pipeline” first goes through all the necessary transformations, and only then is encrypted and the encrypted data is sent to the server. And vice versa, everything that the server sends is first decrypted, then transformed by the remaining part of the “pipeline”. Why do we need all this plumbing? For a wide variety of things. For example, to filter the data sent or received. Or for the transformation of some entities into others, which is useful when implementing protocols. Let's say there is a certain case class DeleteMessage (id: String). The client sends an instance of DeleteMessage (“23”) to the “pipeline”, and at one of the stages (in one of the “pipes”) this class converts the command “a001 STORE 23 + FLAGS.SILENT (\ Deleted)” into an understandable server. More "pipes" may delay the delivery of data, for example, if the response from the server is incomplete, and the addition is expected.
The main point, which at first completely confused me, is the presence of the conceptual concepts “event” (Event) and “command” (Command), and the corresponding splitting of the pipeline into two: the event pipeline and the command pipeline within single class: PipelineStage. It was these very concepts that I did not understand at the very beginning (well, the manuals, especially such scattered and incomprehensible, only the losers read to the end, the normal guys immediately go ahead and fill the cones) made me think about the bad pipelines and decide for myself that this is too difficult and not worth the time spent. It seemed to me that this was somehow connected with the very “back pressure”, which I would have to take into account and implement, although I don’t need it at all. And this is in addition to the fact that at first I didn’t understand where to “stick” one “pipe” and how to put something into it so that it would reach the server. And how then from her pull the answer. And then there were two more pipes. On the other hand, if it were not for this misunderstanding - I would not have fully felt the full power of the "sanitary" approach after the invention of my little monster. In fact, the idea turned out to be so simple that it even made me laugh: Event is what comes from the server to the client, Command is what goes from the client to the server. As a result, the pipe turned out to be one, just inside it separates two oncoming flows for itself, so as not to get confused, where does it go from where.
In general, as a result of my research, I have a new class that is responsible for connecting to the IMAP server, which has taken such a brief and concise look:
I got this class by simplifying the class spray.can.client.HttpClientConnection. It is inherited from spray.io.ConnectionHandler, and that, in turn, from akka.Actor. That is, it is an ordinary actor. In essence, this is a plumber, let's call him Stanislav. Stanislav is responsible for the pipeline from the client to the server and the delivery of data through this pipeline, back and forth. It pushes this pipeline when the actor (ie, itself) is initialized with the standard context.actorOf (...). The pipeline property is the pipeline assembled in this case from three pipes - the frontend, the response parser from the server and the SSL / TLS cryptographer. And now, all the data sent by the plumber to Stanislav (the usual message to him, as an actor, a message by the operator!, The tell method or by any other available means), he will carefully put into the pipe and send it to the server. And everything that comes in response from the server will also carefully pull it out of the pipe and send it to the client. Such is he, our hard-working guy Stasik.
As for the "pipes" that I used.
SslTlsSupport is a standard SSL / TLS encryption feature implemented in Spray. It requires a special context (returned by the pipelineContext method), and also requires keeping the client side of the connection open, even after the server closes the connection from its side (the so-called half-open connection).
ResponseParsing is an object written by me with the apply () function, which returns an instance of the “pipe” responsible for parsing - parsing the stream of characters from the server (in the form of “raw” Tcp.Received messages) to case classes of specific answers understood and processed already target actor (which is my IMAP client). The parser is also responsible for monitoring the integrity of the returned data: wait for additional data if the response from the server is not complete, and separate several responses from each other if they came in one piece. This greatly unloaded the code of my client, who has now turned from a terrible patchwork monster into a simple, understandable, direct and uncomplicated guy Vasily, a bosom friend of our neat Stasik (they wouldn’t be friends, Stas just came and did a lot of dirty work). The mass of tests required to maintain Vasina's performance also decreased significantly.
Finally, eventFrontend is a function that returns a pipe instance — PipelineStage, the essence of which is one thing: to transmit all “events” (that is, data that has passed through the entire pipeline from the server and that have already undergone all necessary changes) to the client, that is, Vasya the address of which Stanislav knows due to the variable passed in the constructor of the class.
I did not do any special command rendering, for lack of such necessity. All commands to the server are sent using simple Tcp.Write.
Here, in fact, all the plumbing. As an epilogue, I can say that the client itself is a finite state machine (Finite-state machine), based on Akka.FSM. I just fell in love with the implementation of this concept in Akka, as the writing of the automaton and unit tests for him is such an exciting mini-game.
Start
Development from the very beginning went at an accelerated pace: almost all the necessary functionality that was needed for my application was available in abundance on the Internet as third-party libraries. And the lack of what is not written specifically for Scala has been more than compensated for by the rich variety of different components for Java. However, the problem, as always, arose in the most unexpected place.
The fact is that at some point my application needs to connect to the IMAP server to read and process the received mail messages. And since the model of actors implies asynchronous work with the network, I needed a library that could connect to the mail server and receive mail asynchronously in order to fit in beautifully with my new application structure. After a brief search, I came across the akka-camel module, which allows you to use the apache-camel library as a message channel for actors. And camel, as it turned out, is able to connect, among other things, to mail servers. In addition, when specifying the required connection parameters, camel can read only fresh (with the \ Recent flag) messages, delete read messages, or copy / move them into a specially created folder. I couldn’t dream of more. To start working in SBT, all that was needed was to mention in the dependencies akka-camel, camel-core and camel-mail.
First try
Creating an actor and connecting it to the IMAP server took literally a few lines of code. And here in the application log fell out the text of the message, which I myself sent to the mail for the test. I started rubbing my hands in satisfaction and thinking about the next task, but I decided to try to connect to the workbox, which would receive mail for processing as a result, just in case. And here my actor threw an exception and "fell." As it turned out, he could not parse the server response correctly. On the Internet, I did not find any information about this error and possible solutions. And a bit sad. I didn’t really want to spend time studying protocol specifications and writing my client. And I, reluctantly, in the name of saving time, decided to retreat from the intended course of complete asynchrony and use the synchronous blocking library JavaMail. However, in the same place with the same exception, this library also fell. After that, I had already firmly decided that giving up ideals was a way for weaklings and lazy people, but I would still write my own IMAP client, with asynchrony and actors. Moreover, I did not need to implement the entire IMAP, the functionality required was very limited: authorization, selecting the INBOX folder, getting a list of messages, reading a specific message, copying the message to another folder and deleting.
')
Second try
I didn't particularly have to choose where to start. As you know, Akka developers at some point abandoned Netty for network I / O in favor of Spray. Further, the development of Akka and Spray is so closely intertwined that even their documentation mutually refers to each other, and pieces of code from spray.io smoothly migrated to akka.io. And it was here that the main trick was waiting for me: once, during the development of version 2.x, Akka adopted the idea of ​​channels used in Spray (they are also “pipelines”, they are also pipelines in English), which allow ( according to the authors, it is easy to create network protocols that support back pressure — that is, the ability to “screw the valve” so that the “pipes do not clog” in case the recipient does not have time to process the data stream from the sender, filter, divide, multiply the data and what else with them not to do. But something in these "pipelines" went wrong, and they, without leaving the experimental stage, were declared deprecated. The last announced innovation from Akka, whose goal is to fully replace the channels, is “reactive streams” (reactive streams), which have already been written on Habré. But since this innovation is still in the announcement stage, in the latest version of akka 2.3.6 it is not there yet, but there are no channels anymore. The channels remained in the spray, but all the documentation on them leads to the outdated documentation Akka Version 2.2.0-RC1, which no longer reflects the entire current reality. And the new Akka documentation says that the channels remained in the Spray. In general, the first version of my email client turned out to be approximately similar to the long-suffering child of Frankenstein - collected from different pieces of fragmented and sometimes contradictory documentation. I immediately decided to abandon the “pipeline” in view of the seemingly transcendental complexity of this concept, and therefore my client worked directly with the stream of characters from the server in the form of a ByteString. More precisely, with the scraps of this stream, since no one guarantees that the answer of interest will come in one single piece, or two answers will not blindly form together. In some miraculous way, through a bunch of mathews poured into the monitor and rewritten pieces of code, I managed to tie SSL / TLS encryption to my actor with the help of several pieces of code found in various places. The code I found only in some (very obsolete) of a specific of many versions of official documentation refused to work.
With the implementation of each subsequent stage of its modest functionality, my client became more and more monstrous. In the end, after the next iteration, at 3 am, I realized that it was impossible to continue to live like this, wrote to TODO with annoyance to try all the blocking JavaMail, but through POP3, without the ability to move messages through folders, and went to sleep.
Third attempt
However, being a stubborn scumbag, the next morning (or rather, at lunch), instead of trying to tame JavaMail, I first got into the Spray source code on the github. I spent a couple of days studying them and adapting the information received to fit my needs, but the time spent paid off handsomely. First of all, in the source, I came across the ConnectionHandler class, which is not described anywhere in the documentation, which made my life and creation much easier for me and my creation, putting a lot of things in their places. It was precisely by studying the use of this class in spray-can that I understood how and where these pipelines could be used, of which I found out from the documentation only what tasks they are meant to solve, but not how they do it. In the same place, in source codes, I found out how it is possible to “connect pipes” - that is, to combine several “pipes” (stages) of the pipeline (PipelineStage) into one “pipeline”, to which this leads, and how it is supposed to be used. And I also found out why and how exactly SSL encryption, which I had bolted on the eve, works, which until that moment had remained for me a black box that “just works and doesn’t have to go”.
Enlightenment
For those who are interested in the details: the “pipeline” is made up of parts, in the original they are called “stages” or “stages” (English stages), but I will call them “pipes” to maintain figurativeness. These "pipes" are combined in the code with the help of the operator >>, the order matters. The first are the "pipe", the most "close" to the client, the last - to the server. That is, everything that goes from the client goes through the "pipeline" from left to right, from the server - on the contrary, from right to left. For example, the “pipe” performing the encryption is indicated last, therefore everything that the client sends to the “pipeline” first goes through all the necessary transformations, and only then is encrypted and the encrypted data is sent to the server. And vice versa, everything that the server sends is first decrypted, then transformed by the remaining part of the “pipeline”. Why do we need all this plumbing? For a wide variety of things. For example, to filter the data sent or received. Or for the transformation of some entities into others, which is useful when implementing protocols. Let's say there is a certain case class DeleteMessage (id: String). The client sends an instance of DeleteMessage (“23”) to the “pipeline”, and at one of the stages (in one of the “pipes”) this class converts the command “a001 STORE 23 + FLAGS.SILENT (\ Deleted)” into an understandable server. More "pipes" may delay the delivery of data, for example, if the response from the server is incomplete, and the addition is expected.
The main point, which at first completely confused me, is the presence of the conceptual concepts “event” (Event) and “command” (Command), and the corresponding splitting of the pipeline into two: the event pipeline and the command pipeline within single class: PipelineStage. It was these very concepts that I did not understand at the very beginning (well, the manuals, especially such scattered and incomprehensible, only the losers read to the end, the normal guys immediately go ahead and fill the cones) made me think about the bad pipelines and decide for myself that this is too difficult and not worth the time spent. It seemed to me that this was somehow connected with the very “back pressure”, which I would have to take into account and implement, although I don’t need it at all. And this is in addition to the fact that at first I didn’t understand where to “stick” one “pipe” and how to put something into it so that it would reach the server. And how then from her pull the answer. And then there were two more pipes. On the other hand, if it were not for this misunderstanding - I would not have fully felt the full power of the "sanitary" approach after the invention of my little monster. In fact, the idea turned out to be so simple that it even made me laugh: Event is what comes from the server to the client, Command is what goes from the client to the server. As a result, the pipe turned out to be one, just inside it separates two oncoming flows for itself, so as not to get confused, where does it go from where.
Result
In general, as a result of my research, I have a new class that is responsible for connecting to the IMAP server, which has taken such a brief and concise look:
class Connection(client: ActorRef, remoteAddress: InetSocketAddress, sslEncryption: Boolean, connectTimeout: Duration)(implicit sslEngineProvider: ClientSSLEngineProvider) extends ConnectionHandler { actor => override def supervisorStrategy = SupervisorStrategy.stoppingStrategy def tcp = IO(Tcp)(context.system) log.debug("Attempting connection to {}", remoteAddress) tcp ! Tcp.Connect(remoteAddress)//, timeout = Some(Duration(connectTimeout, TimeUnit.SECONDS))) context.setReceiveTimeout(connectTimeout) val pipeline = eventFrontend >> ResponseParsing() >> SslTlsSupport(512, publishSslSessionInfo = false) override def receive: Receive = { case connected: Tcp.Connected => val connection = sender() connection ! Tcp.Register(self, keepOpenOnPeerClosed = sslEncryption) client ! connected context.watch(connection) context.become(running(connection, pipeline, pipelineContext(connected))) case Tcp.CommandFailed(_: Tcp.Connect) => throw new ConnectionFailure(1, "Failed to connect to IMAP server") case ReceiveTimeout => log.warning("Connect timed out after {}", connectTimeout) throw new ConnectionFailure(2, "Connect timed out") } def eventFrontend = new PipelineStage { def apply(context: PipelineContext, commandPL: CPL, eventPL: EPL): Pipelines = new Pipelines { val commandPipeline: CPL = commandPL val eventPipeline: EPL = { case event => client ! event } } } def pipelineContext(connected: Tcp.Connected) = new SslTlsContext { def actorContext = context def remoteAddress = connected.remoteAddress def localAddress = connected.localAddress def log = actor.log def sslEngine = if (sslEncryption) sslEngineProvider(this) else None } } I got this class by simplifying the class spray.can.client.HttpClientConnection. It is inherited from spray.io.ConnectionHandler, and that, in turn, from akka.Actor. That is, it is an ordinary actor. In essence, this is a plumber, let's call him Stanislav. Stanislav is responsible for the pipeline from the client to the server and the delivery of data through this pipeline, back and forth. It pushes this pipeline when the actor (ie, itself) is initialized with the standard context.actorOf (...). The pipeline property is the pipeline assembled in this case from three pipes - the frontend, the response parser from the server and the SSL / TLS cryptographer. And now, all the data sent by the plumber to Stanislav (the usual message to him, as an actor, a message by the operator!, The tell method or by any other available means), he will carefully put into the pipe and send it to the server. And everything that comes in response from the server will also carefully pull it out of the pipe and send it to the client. Such is he, our hard-working guy Stasik.
As for the "pipes" that I used.
SslTlsSupport is a standard SSL / TLS encryption feature implemented in Spray. It requires a special context (returned by the pipelineContext method), and also requires keeping the client side of the connection open, even after the server closes the connection from its side (the so-called half-open connection).
ResponseParsing is an object written by me with the apply () function, which returns an instance of the “pipe” responsible for parsing - parsing the stream of characters from the server (in the form of “raw” Tcp.Received messages) to case classes of specific answers understood and processed already target actor (which is my IMAP client). The parser is also responsible for monitoring the integrity of the returned data: wait for additional data if the response from the server is not complete, and separate several responses from each other if they came in one piece. This greatly unloaded the code of my client, who has now turned from a terrible patchwork monster into a simple, understandable, direct and uncomplicated guy Vasily, a bosom friend of our neat Stasik (they wouldn’t be friends, Stas just came and did a lot of dirty work). The mass of tests required to maintain Vasina's performance also decreased significantly.
Finally, eventFrontend is a function that returns a pipe instance — PipelineStage, the essence of which is one thing: to transmit all “events” (that is, data that has passed through the entire pipeline from the server and that have already undergone all necessary changes) to the client, that is, Vasya the address of which Stanislav knows due to the variable passed in the constructor of the class.
I did not do any special command rendering, for lack of such necessity. All commands to the server are sent using simple Tcp.Write.
Epilogue
Here, in fact, all the plumbing. As an epilogue, I can say that the client itself is a finite state machine (Finite-state machine), based on Akka.FSM. I just fell in love with the implementation of this concept in Akka, as the writing of the automaton and unit tests for him is such an exciting mini-game.
Source: https://habr.com/ru/post/237427/
All Articles