Writing a web server in Common Lisp part one
Not so long ago, I took up the study of Common Lisp. As it may seem, the study of a new programming language is not a very simple matter, especially if it is completely different from all the languages ​​that have been encountered before. So I decided to start with the Land Of Lisp book. The book is very good, with interesting pictures and is very well suited for beginners. One of the chapters described how to create a Common Lisp web server. I decided to slightly develop this topic, and as a result I didn’t get exactly what was described in this chapter, but a very interesting web server. Source codes can be viewed here .
To write it we need Linux with emacs, sbcl, slime and quicklisp installed. Describe how to install it all, configure it and how to use it, I will not - there are many articles on the Internet about it. Our entire web server will be in one package called myweb. Create a folder with this name, and in it create two folders, log and web. The log folder will contain the web server log file. In the web folder will be html-pages and images that the web server will give to clients. The entire web server consists of seven files.
We start with the file that declares the package, and the asd file that describes the package itself.
Create a package.lisp file:
As you can see, our web server consists of three packages:
The in-package function is usually placed at the beginning of the file and indicates the name of the package in which we declare variables and functions. In this case, since we declare the packages, we must declare them in the main package: cl-user.
Pay attention to the directives : use and : export in the package advertisement. : use allows us to use functions from other packages without specifying the name of the packages at the beginning of the function name, thereby reducing the amount of typed text. : export sets the names of those functions that can be used outside the package. As you can see, in our package: myweb has functions: start-http and: stop-http. While in the: cl-user package, we will not be able to call them through myweb: start-http unless we first declare them using the: export directive.
')
We already have the package advertisement, now it remains to write the source code of these packages. Create the web.lisp, util.lisp and handler.lisp files and add an in-package call to each of them. For web.lisp - (in-package: myweb), for util.lisp (in-package: myweb.util), etc. We will also need to create a log.lisp file with a call (in-package: cl-log). This file is needed to run and configure the cl-log logging system.
The final touch to creating the file structure for the web server will be the creation of the file myweb.asd, which describes what files the asdf system should download so that everything works.
Key: serial t indicates that asdf downloads files in the same order in which they are listed.
Now you need to write a file load.lisp, which will load our package and run the swank server for slime.
To continue development, we need to run swank and load all the necessary libraries using quicklisp. To do this, run sbcl, located in the myweb directory, and call the function (quicklisp: quickload "swank"). After installing swank, start the swank server by calling (swank: create-server) from the sbcl command line.
Using slime-connect from emacs, connect to running sbcl and call all other functions with quicklisp from load.lisp using slime-mode in emacs and the key combination ctrl-e. If you did everything correctly, then quicklisp will download all the necessary libraries and load them with the help of asdf for you. Everything is ready to start developing.
Let's start with the web server itself. For him, we need sockets. I decided to implement the work with sockets using the widespread usocket library. We also need threads (threads), for which we will use bordeaux-threads . But first, I would like to talk about the HTTP request handling model that we are going to create. Each request will be processed in a separate thread. We will have worker threads that will be created depending on the number of requests. Among them, we will have separate idle streams, which, after the completion of the processing of the request, will switch to the condition-wait state, waiting for new requests. Thereby, it is possible to reduce the burden of creating new worker-streams. It turns out a kind of thread pool mechanism for processing http requests.
Let's start with the declaration of sockets and variables for mutexes in the web.lisp file:
To accept and distribute requests by threads, we will use a separate thread, a pointer to which will be stored in * listen-thread *. Let's start with the start-http method:
This is a simple function to start a distribution flow, which in turn will call the http-acceptor function. We also have two keys - this is worker-limit - the maximum number of workers, and idle-workers - the number of idle workers.
Let's write the request distribution function itself:
The first thing we do is socket-listen to the specified address and port. Further in the loop, we do the socket-accept, resulting in a socket on the connected client, which we must process in the worker. Plus we assign request-id to request. At this stage, we must decide what to do with the request and how to handle it. First of all, we check the number of idle threads. If all our workers are busy, we add the request to the queue for processing. If we have a free idle worker, then again we add the request to the queue, but this time we call (condition-notify (caar * idle-workers *))). And in the third case, we simply create a new worker and pass it a request that will be processed in the worker-thread function. Everything is quite simple. It remains only to write the function of processing worker-flow:
If we had a call with request-id, then we need to process the request first. We simply call the helper function http-worker and pass the client socket to it. Next, we check that there are more processing requests: we simply remove the first request from the queue and pass it to worker-thread for processing, thereby calling the worker-thread function recursively. A question may arise: “Will the recursion limit happen if the stack overflows at some point, for example, when there are a large number of requests in the queue?” Since, after calling worker-thread recursively, nothing is called in our function, recursion limit will not happen. Almost all modern Common Lisp implementations support this optimization. Well, if the queue is empty, then we have to check the number of idle workers. If everything is all right with us, then we simply complete the request and remove the worker from the list of workers. If not, then we do a condition-wait, and thus the worker becomes an idle worker.
If you notice, we also call list-workers. This auxiliary function which simply clears a sheet of workers from dead threads.
It remains to write the http-worker function:
Here we create a socket-stream, parse the request and pass it to myweb.handler: process-request (we'll talk about these functions in the second part). list-workers simply returns us a list of workers, having previously cleared it of dead threads. We call this function in worker-thread before condition-wait.
The last thing we need to do is write the stop-http function, which will stop our web server:
As you can see, everything is simple here - we stop the flow of the distributor, kill all workers and reset the lists.
And so, everything is ready to process our requests. We will talk about this in the second part .
Thank you for your attention!
PS Thank you ertaquo for help with spelling and layout
To write it we need Linux with emacs, sbcl, slime and quicklisp installed. Describe how to install it all, configure it and how to use it, I will not - there are many articles on the Internet about it. Our entire web server will be in one package called myweb. Create a folder with this name, and in it create two folders, log and web. The log folder will contain the web server log file. In the web folder will be html-pages and images that the web server will give to clients. The entire web server consists of seven files.
We start with the file that declares the package, and the asd file that describes the package itself.
Create a package.lisp file:
(in-package :cl-user) (defpackage :myweb (:use :cl :usocket :bordeaux-threads) (:export :start-http :stop-http :list-workers :list-requests)) (defpackage :myweb.util (:use :cl :local-time) (:export :parse-request :read-utf-8-string :response-write :get-param :get-header :http-response :file-response :html-template :log-info :log-warning :log-error)) (defpackage :myweb.handler (:use :cl) (:export :process-request)) As you can see, our web server consists of three packages:
- myweb - will contain functions for starting and stopping the web server
- myweb.util - will contain functions that help process requests
- myweb.handler - will contain the request processing code itself
The in-package function is usually placed at the beginning of the file and indicates the name of the package in which we declare variables and functions. In this case, since we declare the packages, we must declare them in the main package: cl-user.
Pay attention to the directives : use and : export in the package advertisement. : use allows us to use functions from other packages without specifying the name of the packages at the beginning of the function name, thereby reducing the amount of typed text. : export sets the names of those functions that can be used outside the package. As you can see, in our package: myweb has functions: start-http and: stop-http. While in the: cl-user package, we will not be able to call them through myweb: start-http unless we first declare them using the: export directive.
')
We already have the package advertisement, now it remains to write the source code of these packages. Create the web.lisp, util.lisp and handler.lisp files and add an in-package call to each of them. For web.lisp - (in-package: myweb), for util.lisp (in-package: myweb.util), etc. We will also need to create a log.lisp file with a call (in-package: cl-log). This file is needed to run and configure the cl-log logging system.
The final touch to creating the file structure for the web server will be the creation of the file myweb.asd, which describes what files the asdf system should download so that everything works.
;; myweb.asd (asdf:defsystem #:myweb :serial t :components ((:file "package") (:file "log") (:file "util") (:file "web") (:file "handler"))) Key: serial t indicates that asdf downloads files in the same order in which they are listed.
Now you need to write a file load.lisp, which will load our package and run the swank server for slime.
(in-package :cl-user) (quicklisp:quickload "swank") (quicklisp:quickload "usocket") (quicklisp:quickload "bordeaux-threads") (quicklisp:quickload "trivial-utf-8") (quicklisp:quickload "cl-log") (quicklisp:quickload "local-time") (pushnew '*default-pathname-defaults* asdf:*central-registry*) (asdf:load-system 'myweb) (swank:create-server) To continue development, we need to run swank and load all the necessary libraries using quicklisp. To do this, run sbcl, located in the myweb directory, and call the function (quicklisp: quickload "swank"). After installing swank, start the swank server by calling (swank: create-server) from the sbcl command line.
Using slime-connect from emacs, connect to running sbcl and call all other functions with quicklisp from load.lisp using slime-mode in emacs and the key combination ctrl-e. If you did everything correctly, then quicklisp will download all the necessary libraries and load them with the help of asdf for you. Everything is ready to start developing.
Let's start with the web server itself. For him, we need sockets. I decided to implement the work with sockets using the widespread usocket library. We also need threads (threads), for which we will use bordeaux-threads . But first, I would like to talk about the HTTP request handling model that we are going to create. Each request will be processed in a separate thread. We will have worker threads that will be created depending on the number of requests. Among them, we will have separate idle streams, which, after the completion of the processing of the request, will switch to the condition-wait state, waiting for new requests. Thereby, it is possible to reduce the burden of creating new worker-streams. It turns out a kind of thread pool mechanism for processing http requests.
Let's start with the declaration of sockets and variables for mutexes in the web.lisp file:
(defvar *listen-socket* nil) (defvar *listen-thread* nil) (defvar *request-mutex* (make-lock "request-mutex")) (defvar *request-threads* (list)) (defvar *worker-mutex* (make-lock "worker-mutex")) (defvar *workers* (list)) (defvar *worker-num* 0) (defvar *idle-workers* (list)) (defvar *idle-workers-num* 0) (defvar *request-queue* (list)) To accept and distribute requests by threads, we will use a separate thread, a pointer to which will be stored in * listen-thread *. Let's start with the start-http method:
(defun start-http (host port &key (worker-limit 10) (idle-workers 1)) (if (not *listen-socket*) (setq *listen-thread* (make-thread (lambda () (http-acceptor host port worker-limit idle-workers)) :name "socket-acceptor")) "http server already started")) This is a simple function to start a distribution flow, which in turn will call the http-acceptor function. We also have two keys - this is worker-limit - the maximum number of workers, and idle-workers - the number of idle workers.
Let's write the request distribution function itself:
(defun http-acceptor (host port worker-limit idle-workers) (setq *listen-socket* (socket-listen host port :reuse-address t :element-type '(unsigned-byte 8) :backlog (* worker-limit 2))) (let ((request-id 0) (worker-id 0)) (loop while *listen-thread* do (let* ((socket (socket-accept *listen-socket* :element-type '(unsigned-byte 8)))) (progn (setq request-id (1+ request-id)) (acquire-lock *worker-mutex*) (if (>= *worker-num* worker-limit) (push (cons request-id socket) *request-queue*) ;; Get worker from idle workers (if (> *idle-workers-num* 0) (progn (push (cons request-id socket) *request-queue*) (condition-notify (caar *idle-workers*))) ;; Add new Worker (progn (setq worker-id (1+ worker-id)) (setq *worker-num* (1+ *worker-num*)) (setq *workers* (cons (make-thread (lambda () (worker-thread request-id socket idle-workers)) :name (concatenate 'string "socket-worker-" (prin1-to-string worker-id))) *workers*))))) (release-lock *worker-mutex*) t))))) The first thing we do is socket-listen to the specified address and port. Further in the loop, we do the socket-accept, resulting in a socket on the connected client, which we must process in the worker. Plus we assign request-id to request. At this stage, we must decide what to do with the request and how to handle it. First of all, we check the number of idle threads. If all our workers are busy, we add the request to the queue for processing. If we have a free idle worker, then again we add the request to the queue, but this time we call (condition-notify (caar * idle-workers *))). And in the third case, we simply create a new worker and pass it a request that will be processed in the worker-thread function. Everything is quite simple. It remains only to write the function of processing worker-flow:
(defun worker-thread (request-id socket idle-workers) (if request-id ;; Process request if it is not nil (progn (with-lock-held (*request-mutex*) (setq *request-threads* (cons (cons request-id (current-thread)) *request-threads*)) ) (http-worker socket) (with-lock-held (*request-mutex*) (setq *request-threads* (remove-if (lambda (x) (eq (car x) request-id)) *request-threads*)) ) )) (acquire-lock *worker-mutex*) (if *request-queue* (let ((request nil)) (setq request (car *request-queue*)) (setq *request-queue* (cdr *request-queue*)) (release-lock *worker-mutex*) (worker-thread (car request) (cdr request) idle-workers)) (if (< *idle-workers-num* idle-workers) (let ((condition (make-condition-variable)) (idle-lock (make-lock)) (request nil)) (push (cons condition (current-thread)) *idle-workers*) (setq *idle-workers-num* (1+ *idle-workers-num*)) (release-lock *worker-mutex*) (list-workers) (with-lock-held (idle-lock) (condition-wait condition idle-lock) ) (with-lock-held (*worker-mutex*) (setq *idle-workers* (cdr *idle-workers*)) (setq *idle-workers-num* (1- *idle-workers-num*)) (setq request (car *request-queue*)) (setq *request-queue* (cdr *request-queue*)) ) (worker-thread (car request) (cdr request) idle-workers)) (progn (setq *workers* (remove (current-thread) *workers*)) (setq *worker-num* (1- *worker-num*)) (release-lock *worker-mutex*))))) If we had a call with request-id, then we need to process the request first. We simply call the helper function http-worker and pass the client socket to it. Next, we check that there are more processing requests: we simply remove the first request from the queue and pass it to worker-thread for processing, thereby calling the worker-thread function recursively. A question may arise: “Will the recursion limit happen if the stack overflows at some point, for example, when there are a large number of requests in the queue?” Since, after calling worker-thread recursively, nothing is called in our function, recursion limit will not happen. Almost all modern Common Lisp implementations support this optimization. Well, if the queue is empty, then we have to check the number of idle workers. If everything is all right with us, then we simply complete the request and remove the worker from the list of workers. If not, then we do a condition-wait, and thus the worker becomes an idle worker.
If you notice, we also call list-workers. This auxiliary function which simply clears a sheet of workers from dead threads.
It remains to write the http-worker function:
(defun http-worker (socket) (let* ((stream (socket-stream socket)) (request (myweb.util:parse-request stream))) (myweb.handler:process-request request stream) (finish-output stream) (socket-close socket))) (defun list-workers () (with-lock-held (*worker-mutex*) (setq *workers* (remove-if (lambda (w) (not (thread-alive-p w))) *workers*)) (setq *worker-num* (length *workers*)) *workers*)) Here we create a socket-stream, parse the request and pass it to myweb.handler: process-request (we'll talk about these functions in the second part). list-workers simply returns us a list of workers, having previously cleared it of dead threads. We call this function in worker-thread before condition-wait.
The last thing we need to do is write the stop-http function, which will stop our web server:
(defun stop-http () (if *listen-socket* (progn (stop-thread) (socket-close *listen-socket*) (setq *listen-socket* nil) (setq *request-queue* nil) (setq *worker-num* 0) (setq *workers* nil) (mapcar (lambda (i) (destroy-thread (cdr i))) *idle-workers*) (setq *idle-workers-num* 0) (setq *idle-workers* nil) (release-lock *worker-mutex*) (setq *request-threads* nil) (release-lock *request-mutex*) (setq *request-mutex* (make-lock "request-mutex")) (setq *worker-mutex* (make-lock "worker-mutex"))))) (defun stop-thread () (if (and *listen-thread* (thread-alive-p *listen-thread*)) (destroy-thread *listen-thread*))) As you can see, everything is simple here - we stop the flow of the distributor, kill all workers and reset the lists.
And so, everything is ready to process our requests. We will talk about this in the second part .
Thank you for your attention!
PS Thank you ertaquo for help with spelling and layout
Source: https://habr.com/ru/post/131586/
All Articles