Julia. Web services

We continue to consider Julia technology. And today we will talk about packages designed to build web services. It's no secret that the main niche of the Julia language is high-performance computing. Therefore, a rather logical step is the direct creation of web services capable of performing these calculations on demand. Of course, web services are not the only way to communicate in a networked environment. But, since they are the most widely used in distributed systems now, we will consider the creation of services serving HTTP requests.

Note that due to Julia’s youth, there is a set of competing packages. Therefore, we will try to figure out how and why to use them. Along the way, let's compare the implementation of the same JSON web service with their help.

Infrastructure Julia is actively developing in the last year or two. And, in this case, it is not just a duty phrase, inscribed for a beautiful beginning of the text, but an emphasis on the fact that everything is changing rapidly, but what was relevant a couple of years ago is now outdated. However, we will try to isolate stable packages and give recommendations on how to implement web services with their help. For definiteness, we will create a web service that receives a POST request with JSON data of the following format:

{ "title": "something", "body": "something" } 

We assume that the service we create is not RESTful. Our main task is to look at how to describe routes and request handlers.

HTTP.jl package

This package is the main implementation of the HTTP protocol in Julia and gradually acquires new features. In addition to implementing typical structures and functions for performing client HTTP requests, this package implements functions for creating HTTP servers. At the same time, as it develops, the package received functions that make it quite comfortable for the programmer to register handlers and, thus, build typical services. Also, in the latest versions, there is built-in support for the WebSocket protocol, the implementation of which was previously done as part of a separate package WebSocket.jl. That is, HTTP.jl, currently, can satisfy most of the needs of the programmer. Consider a couple of examples in more detail.

HTTP client

We start the implementation with the client code, which we will use to test the performance.

 #!/usr/bin/env julia --project=@. import HTTP import JSON.json const PORT = "8080" const HOST = "127.0.0.1" const NAME = "Jemand" #    struct Document title::String body::String end #         Base.show(r::HTTP.Messages.Response) = println(r.status == 200 ? String(r.body) : "Error: " * r.status) #    r = HTTP.get("http://$(HOST):$(PORT)") show(r) #   /user/:name r = HTTP.get("http://$(HOST):$(PORT)/user/$(NAME)"; verbose=1) show(r) #  JSON- POST- doc = Document("Some document", "Test document with some content.") r = HTTP.post( "http://$(HOST):$(PORT)/resource/process", [("Content-Type" => "application/json")], json(doc); verbose=3) show(r) 

The HTTP package provides methods that correspond to the names of the HTTP protocol commands. In this case, use get and post . The optional named argument verbose allows you to set the amount of debugging output. So, for example, verbose=1 returns:

 GET /user/Jemand HTTP/1.1 HTTP/1.1 200 OK <= (GET /user/Jemand HTTP/1.1) 

And in the case of verbose=3 we already get the full set of transmitted and received data:

 DEBUG: 2019-04-21T22:40:40.961 eb4f ️-> "POST /resource/process HTTP/1.1\r\n" (write) DEBUG: 2019-04-21T22:40:40.961 eb4f ️-> "Content-Type: application/json\r\n" (write) DEBUG: 2019-04-21T22:40:40.961 eb4f ️-> "Host: 127.0.0.1\r\n" (write) DEBUG: 2019-04-21T22:40:40.961 eb4f ️-> "Content-Length: 67\r\n" (write) DEBUG: 2019-04-21T22:40:40.961 eb4f ️-> "\r\n" (write) DEBUG: 2019-04-21T22:40:40.961 e1c6 ️-> "{\"title\":\"Some document\",\"body\":\"Test document with some content.\"}" (unsafe_write) DEBUG: 2019-04-21T22:40:40.963 eb4f ️<- "HTTP/1.1 200 OK\r\n" (readuntil) DEBUG: "Content-Type: application/json\r\n" DEBUG: "Transfer-Encoding: chunked\r\n" DEBUG: "\r\n" DEBUG: 2019-04-21T22:40:40.963 eb4f ️<- "5d\r\n" (readuntil) DEBUG: 2019-04-21T22:40:40.963 eb4f ️<- "{\"body\":\"Test document with some content.\",\"server_mark\":\"confirmed\",\"title\":\"Some document\"}" (unsafe_read) DEBUG: 2019-04-21T22:40:40.968 eb4f ️<- "\r\n" (readuntil) DEBUG: "0\r\n" DEBUG: 2019-04-21T22:40:40.968 eb4f ️<- "\r\n" (readuntil) 

In the future, we will use only verbose=1 in order to see only minimal information about what is happening.

A few comments on the code.

 doc = Document("Some document", "Test document with some content.") 

Since we previously declared the Document structure (and, immutable), a constructor is available for it by default, whose arguments correspond to the declared fields of the structure. In order to convert it to JSON, we use the JSON.jl package and its json(doc) method.
Pay attention to the fragment:

 r = HTTP.post( "http://$(HOST):$(PORT)/resource/process", [("Content-Type" => "application/json")], json(doc); verbose=3) 

Since we are transmitting JSON, you must explicitly specify the type application/json in the Content-Type header. Headers are transferred to the HTTP.post method (as well as to all others) using an array (of the Vector type, but not Dict) containing the name of the title-value pairs.

For the performance test, we will perform three queries:

• GET request to root path;
• GET request in the format / user / name, where name is the transmitted name;
• POST request / resource / process with transfer of JSON object. We expect to receive the same document, but with the added server_mark field.

We will use this client code to test all server implementation options.

HTTP server

After you have dealt with the client, it's time to start implementing the server. To begin with, we will make the service only with the help of HTTP.jl , in order to keep it as a basic variant, which does not require installation of other packages. We remind you that all other packages use HTTP.jl

 #!/usr/bin/env julia --project=@. import Sockets import HTTP import JSON #    #    index(req::HTTP.Request) = HTTP.Response(200, "Hello World") #     function welcome_user(req::HTTP.Request) # dump(req) user = "" if (m = match( r".*/user/([[:alpha:]]+)", req.target)) != nothing user = m[1] end return HTTP.Response(200, "Hello " * user) end #  JSON function process_resource(req::HTTP.Request) # dump(req) message = JSON.parse(String(req.body)) @info message message["server_mark"] = "confirmed" return HTTP.Response(200, JSON.json(message)) end #      const ROUTER = HTTP.Router() HTTP.@register(ROUTER, "GET", "/", index) HTTP.@register(ROUTER, "GET", "/user/*", welcome_user) HTTP.@register(ROUTER, "POST", "/resource/process", process_resource) HTTP.serve(ROUTER, Sockets.localhost, 8080) 

In the example, pay attention to the following code:

 dump(req) 

print to the console all that is known for the object. Includes data types, values, and all nested fields and their values. This method is useful for both library research and debugging.

Line

 (m = match( r".*/user/([[:alpha:]]+)", req.target)) 

is a regular expression that parses the route to which the handler is registered. HTTP.jl does not provide automatic ways to detect patterns in the route.

Inside the process_resource handler, we are parsing JSON, which is accepted by the service.

 message = JSON.parse(String(req.body)) 

Data access is performed via the req.body field. Note that the data comes in a byte array format. Therefore, to work with them as with a string, an explicit conversion to a string is performed. The JSON.parse method is a JSON.jl package method that performs JSON.jl data and builds an object. Since the object in this case is an associative array (Dict), we can easily add a new key to it. Line

 message["server_mark"] = "confirmed" 

adds the server_mark key with the confirmed value.

The service HTTP.serve(ROUTER, Sockets.localhost, 8080) when the HTTP.serve(ROUTER, Sockets.localhost, 8080) line is HTTP.serve(ROUTER, Sockets.localhost, 8080) .

Control response for the HTTP.jl-based service (obtained when launching client code with verbose=1 ):

 GET / HTTP/1.1 HTTP/1.1 200 OK <= (GET / HTTP/1.1) Hello World GET /user/Jemand HTTP/1.1 HTTP/1.1 200 OK <= (GET /user/Jemand HTTP/1.1) Hello Jemand POST /resource/process HTTP/1.1 HTTP/1.1 200 OK <= (POST /resource/process HTTP/1.1) {"body":"Test document with some content.","server_mark":"confirmed","title":"Some document"} 

Against the background of debug information with verbose=1 , the lines are clearly visible: Hello World , Hello Jemand , "server_mark":"confirmed" .

After viewing the service code, a natural question arises - why do we need all the other packages, if everything is so simple in HTTP. There is a very simple answer to this. HTTP - allows you to register dynamic handlers, but even the elementary implementation of reading a static image file from a directory requires a separate implementation. Therefore, we also consider packages that are focused on creating web applications.

Mux.jl package

This package is positioned as an intermediate layer for web applications implemented on Julia. Its implementation is very lightweight. The main purpose is to provide an easy way to describe handlers. It cannot be said that the project is not developing, but it is developing slowly. However, look at the code of our service, serving the same routes.

 #!/usr/bin/env julia --project=@. using Mux using JSON @app test = ( Mux.defaults, page(respond("<h1>Hello World!</h1>")), page("/user/:user", req -> "<h1>Hello, $(req[:params][:user])!</h1>"), route("/resource/process", req -> begin message = JSON.parse(String(req[:data])) @info message message["server_mark"] = "confirmed" return Dict( :body => JSON.json(message), :headers => [("Content-Type" => "application/json")] ) end), Mux.notfound() ) serve(test, 8080) Base.JLOptions().isinteractive == 0 && wait() 

Routes are described here using the page method. The web application is declared using the @app macro. The arguments to the page method are the route and handler. A handler can be defined as a function that accepts a request as an input, or it can be specified as a lambda function in place. Of the additional useful functions, there is Mux.notfound() for sending the specified Not found response. And the result that should be sent to the client does not need to be packaged in HTTP.Response , as we did in the previous example, since Mux will do it by itself. However, parsing JSON still has to be done by yourself, as is the serialization of the object for the response - JSON.json(message) .

  message = JSON.parse(String(req[:data])) message["server_mark"] = "confirmed" return Dict( :body => JSON.json(message), :headers => [("Content-Type" => "application/json")] ) 

The answer is sent as an associative array with fields :body :headers .

Starting the server using the serve(test, 8080) method is asynchronous, so one of the options in Julia to organize waiting for completion is to call the code:

 Base.JLOptions().isinteractive == 0 && wait() 

Otherwise, the service does the same as the previous version on HTTP.jl

Service control response:

 GET / HTTP/1.1 HTTP/1.1 200 OK <= (GET / HTTP/1.1) <h1>Hello World!</h1> GET /user/Jemand HTTP/1.1 HTTP/1.1 200 OK <= (GET /user/Jemand HTTP/1.1) <h1>Hello, Jemand!</h1> POST /resource/process HTTP/1.1 HTTP/1.1 200 OK <= (POST /resource/process HTTP/1.1) {"body":"Test document with some content.","server_mark":"confirmed","title":"Some document"} 

Bukdu.jl package

The package was developed under the influence of the Phoenix framework, which, in turn, is implemented on Elixir and is the implementation of the ideas of web-building from the Ruby-community in projection on Elixir. The project is developing quite actively and is positioned as a tool for creating RESTful API and lightweight web applications. There are functions that simplify JSON serialization and deserialization. This is missing from HTTP.jl and Mux.jl Let's look at the implementation of our web service.

 #!/usr/bin/env julia --project=@. using Bukdu using JSON #   struct WelcomeController <: ApplicationController conn::Conn end #   index(c::WelcomeController) = render(JSON, "Hello World") welcome_user(c::WelcomeController) = render(JSON, "Hello " * c.params.user) function process_resource(c::WelcomeController) message = JSON.parse(String(c.conn.request.body)) @info message message["server_mark"] = "confirmed" render(JSON, message) end #   routes() do get("/", WelcomeController, index) get("/user/:user", WelcomeController, welcome_user, :user => String) post("/resource/process", WelcomeController, process_resource) end #   Bukdu.start(8080) Base.JLOptions().isinteractive == 0 && wait() 

The first thing you should pay attention to is the declaration of the structure for storing the state of the controller.

 struct WelcomeController <: ApplicationController conn::Conn end 

In this case, it is a concrete type created as a descendant of the abstract type ApplicationController .

Methods for the controller are declared in a similar way with respect to previous implementations. There is a slight difference in the handler of our JSON object.

 function process_resource(c::WelcomeController) message = JSON.parse(String(c.conn.request.body)) @info message message["server_mark"] = "confirmed" render(JSON, message) end 

As you can see, deserialization is also performed independently using the JSON.parse method, but to serialize the response, the render(JSON, message) method render(JSON, message) provided by Bukdu is used.

The declaration of the routes is carried out in the traditional style for rubists, including the use of the do...end block.

 routes() do get("/", WelcomeController, index) get("/user/:user", WelcomeController, welcome_user, :user => String) post("/resource/process", WelcomeController, process_resource) end 

Also, in the traditional way for rubists, a segment is declared in the route string /user/:user . In other words, the variable part of the expression, which can be accessed by the name specified in the template. Syntactically designated as a representative of type Symbol . By the way, for Julia, the Symbol type means, in essence, the same as for Ruby - it is an immutable string, represented in memory by a single instance.

Accordingly, after we declared a route with a variable part, and also indicated the type of this variable part, we can refer to the already parsed data by the assigned name. In the method that handles the request, we simply access the field through a dot in the form c.params.user .

 welcome_user(c::WelcomeController) = render(JSON, "Hello " * c.params.user) 

Service control response:

 GET / HTTP/1.1 HTTP/1.1 200 OK <= (GET / HTTP/1.1) "Hello World" GET /user/Jemand HTTP/1.1 HTTP/1.1 200 OK <= (GET /user/Jemand HTTP/1.1) "Hello Jemand" POST /resource/process HTTP/1.1 HTTP/1.1 200 OK <= (POST /resource/process HTTP/1.1) {"body":"Test document with some content.","server_mark":"confirmed","title":"Some document"} 

Conclusion of service in the console:

 >./bukdu_json.jl INFO: Bukdu Listening on 127.0.0.1:8080 INFO: GET WelcomeController index 200 / INFO: GET WelcomeController welcome_user 200 /user/Jemand INFO: Dict{String,Any}("body"=>"Test document with some content.","title"=>"Some document") INFO: POST WelcomeController process_resource200 /resource/process 

Package Genie.jl

An ambitious project, positioned as MVC web framework. In his approach, the Rails on Julia are quite clearly visible, including the directory structure created by the generator. The project is developing, however, for unknown reasons, this package is not part of the Julia package repository. That is, its installation is possible only from the git repository with the command:

 julia>] # switch to pkg> mode pkg> add https://github.com/essenciary/Genie.jl 

The code of our service in Genie is as follows (do not use generators):

 #!/usr/bin/env julia --project=@. #     import Genie import Genie.Router: route, @params, POST import Genie.Requests: jsonpayload, rawpayload import Genie.Renderer: json! #      route("/") do "Hello World!" end route("/user/:user") do "Hello " * @params(:user) end route("/resource/process", method = POST) do message = jsonpayload() # if message == nothing # dump(Genie.Requests.rawpayload()) # end message["server_mark"] = "confirmed" return message |> json! end #   Genie.AppServer.startup(8080) Base.JLOptions().isinteractive == 0 && wait() 

Here you should pay attention to the format of the declaration.

 route("/") do "Hello World!" end 

This code is very familiar to Ruby programmers. Block do...end as a handler and route as an argument to the method. Note that for Julia this code can be rewritten in the form:

 route(req -> "Hello World!", "/") 

That is, the handler function is in the first place, the route is in the second. But for our case, leave the Ruby-style.

Genie automatically packs the execution result into an HTTP response. In the minimal case, we only need to return the result of the correct type, for example, String. Of the additional facilities, implemented an automatic check of the input format and its analysis. For example, for JSON you just need to call the jsonpayload() method.

 route("/resource/process", method = POST) do message = jsonpayload() # if message == nothing # dump(Genie.Requests.rawpayload()) # end message["server_mark"] = "confirmed" return message |> json! end 

Notice the code snippet commented out here. The jsonpayload() method returns nothing if, for some reason, the input format is not recognized as JSON. Note that for this purpose, the header [("Content-Type" => "application/json")] added to our HTTP client, since otherwise Genie will not even begin to parse the data as JSON. In case something incomprehensible has come, it is useful to look at rawpayload() for what it is. However, since this is just a debug stage, you should not leave it in the code.

Also, you should pay attention to the return result in the format message |> json! . Method json!(str) here put the last in the pipeline. It provides serialization of data in JSON format, and also ensures that Genie will add the correct Content-Type . Also, note that the word return in most cases in the examples above is redundant. Julia, like Ruby, for example, always returns the result of the last operation or the value of the last specified expression. That is, the word return is optional.

Genie's capabilities don't end there, but we don't need them to implement a web service.

Service control response:

 GET / HTTP/1.1 HTTP/1.1 200 OK <= (GET / HTTP/1.1) Hello World! GET /user/Jemand HTTP/1.1 HTTP/1.1 200 OK <= (GET /user/Jemand HTTP/1.1) Hello Jemand POST /resource/process HTTP/1.1 HTTP/1.1 200 OK <= (POST /resource/process HTTP/1.1) {"body":"Test document with some content.","server_mark":"confirmed","title":"Some document"} 

Conclusion of service in the console:

 >./genie_json.jl [ Info: Ready! 2019-04-24 17:18:51:DEBUG:Main: Web Server starting at http://127.0.0.1:8080 2019-04-24 17:18:51:DEBUG:Main: Web Server running at http://127.0.0.1:8080 2019-04-24 17:19:21:INFO:Main: / 200 2019-04-24 17:19:21:INFO:Main: /user/Jemand 200 2019-04-24 17:19:22:INFO:Main: /resource/process 200 

JuliaWebAPI.jl package

This package was positioned as an intermediate layer for creating web applications at the time when HTTP.jl was just a library implementing the protocol. The author of this package also implemented a server code generator based on the Swagger specification (OpenAPI and http://editor.swagger.io/ ) - see the project https://github.com/JuliaComputing/Swagger.jl , and this generator used JuliaWebAPI .jl. However, the problem with JuliaWebAPI.jl is that the ability to process the request body (for example, JSON) sent to the server via a POST request is not implemented. The author believed that the transfer of parameters in the key-value format is suitable for all occasions ... The future of this package is not clear. All its functions are already implemented in many other packages, including HTTP.jl. The Swagger.jl package also no longer uses it.

WebSockets.jl

Early implementation of the WebSocket protocol. The package has been used for a long time as the main implementation of this protocol, however, currently, its implementation is included in the HTTP.jl package. The WebSockets.jl package itself uses HTTP.jl to establish a connection, but now you should not use it in new developments. It should be considered as a package for compatibility.

Conclusion

The following review demonstrates various ways to implement a web service on Julia. The easiest and most universal way is to directly use the HTTP.jl package. Also, the Bukdu.jl and Genie.jl packages are very useful. At a minimum, their development should be monitored. With regards to the Mux.jl package, its merits are now dissolving against the background of HTTP.jl. Therefore, personal opinion is not to use it. Genie.jl is a very promising framework. However, before you start using it, you should at least understand why the author does not register it as an official package.

Note that the JSON deserialization code in the examples was used without error handling. In all cases, except Genie, it is necessary to handle parsing errors and inform the user about it. An example of such a code for HTTP.jl:

  local message = nothing local body = IOBuffer(HTTP.payload(req)) try message = JSON.parse(body) catch err @error err.msg return HTTP.Response(400, string(err.msg)) end 

In general, we can say that Julia already has enough tools for creating web services. That is, "reinventing the wheel" for their writing is not necessary. The next step is to assess how Julia can handle the current benchmarks, if someone is ready to take it. However, for now let's stop on this review.

Links

• https://github.com/JuliaWeb/HTTP.jl
• https://github.com/JuliaWeb/Mux.jl
• https://github.com/wookay/Bukdu.jl
• https://github.com/essenciary/Genie.jl
• https://github.com/JuliaComputing/Swagger.jl