From the translator:
I have often heard that Common Lisp has libraries from the 80s and no other, and many of its users are 3.5 professors working on artificial intelligence in the DARPA catacombs and not even having a close understanding of the everyday tasks of an ordinary programmer. This article shows well that even though you may stumble upon old libraries, you only need to use modern analogues, that existing libraries are developing and new ones are constantly appearing.
The author of this article, Fernando Borretti , is an active contributor to the Common Lisp ecosystem, the author of more than 30 libraries, most of which are for web development.
A passing reader will get a general idea of ​​the state of affairs in Common Lisp, an interested person will be able to understand what he needs in order to try to write code and which libraries to take for a test task, and an experienced developer will learn about the latest developments, understand in which directions he can help the community and get some tips on how to answer newbie questions so as not to kill their interest in this great technology. Go!')
Here is a description of the Common Lisp ecosystem as of August 2015 from a user and contributor point of view.
The purpose of this article is to provide an overview of the ecosystem and help the introduction of
unification in each area.
For each application area, recommendations are given on the unification of this part of the ecosystem, as well as interesting directions for future development.
Application Development
Web development
Backend
Clack , an analogue of WSGI / Rack, has existed since 2009 and is well tested and works in production. Three web frameworks -
Caveman2 ,
Ningle and
Lucerne are based on it.
Clack is an HTTP server abstraction that allows a user to write web frameworks (or better web application frameworks) without reference to a specific server implementation.
The importance of using Clack should not be underestimated: if you make an application directly, say, Hunchentoot, you are tied to Hunchentoot, and if a new, faster server such as
Woo appears, you will have to rewrite the entire application to use it. If you write a plugin for Clack, for example
clack-errors , it can automatically be used by all applications, regardless of the frameworks made on Clack, reducing the useless duplication of code.
With Clack, switching from Hunchentoot to Woo and the joy of tremendous acceleration is just a matter of installing
libev and changing one key argument.
Conclusion:Stop using Hunchentoot directly . Use Clack, or better yet, one of the frameworks based on it.
Future work:The main part is finished, it is time to write high-level things. An extensible Clack application administration framework like in Django would be a good example.
Frontend
Everything is connected with the ability of Common Lisp to compile into JavaScript. Common Lisp is a fairly large language, with few options:
- Parenscript : a DSL that compiles a subset of Common Lisp into idiomatic JavaScript
- JSCL : self-developing compiler from CL to JS. There is no support for CLOS, format and loop. ( unlike Parenscript is an interpreter in the browser - comment of the translator )
Conclusion:The best way to unify is to develop one of the existing implementations of CL-in-JS.
Future work:Something like CFFI, but for the implementation of CL-in-JS, so that you don’t have to re-write JavaScript libraries to bind when new versions of libraries are released.
A utility to compile Common Lisp to JavaScript independent of the specific implementation, to facilitate the transition from Parenscript to JSCL or other implementations.
Command line
For many years, various utilities have appeared in this area, the last of them, the one that received the biggest impulse -
Roswell - the implementation of the manager / installer and the script launcher. One of its great features is the easy compilation of small scripts into executable files, for example, for
generating documentation .
It is also used to install implementations (CL) in
Travis to bypass some of the problems with
cl-travis .
Conclusion:Kill cl-launch, use
Roswell .
Future work:More Roswell scripts.
Graphical interface
For a long time, the lack of a complete GUI solution has been of great concern. Now we have it:
CommonQt plus
Qtools . The first is used in real applications for years, and the second is a layer that allows you to make everything easier.
The biggest problem in using CommonQt is the requirement for
Smoke to work, and getting libraries to work can be difficult, especially on a system other than Linux. This is solved by Qtools, which depends on the
qt-libs library. It downloads Smoke libraries for the current platform and makes customizing and deploying applications much easier.
Conclusion:Focus on CommonQt and help improve
cl-cffi-gtk , while other libraries are considered obsolete.
CLIM is interesting and was the last attempt to explore ways to create user interfaces, but it is not a viable option in 2015.
Future work:More tutorials and examples of using CommonQt and Qtools.
Machine learning
CLML is quite a multifunctional solution. It was developed by
Mathematical Systems Inc. Japanese company.
Mike Maul subsequently transferred it to github and cleaned up the code a bit. Available
tutorial on time series .
Another candidate in this field is
mgl , used by
its author to win the
Higgs Boson Machine Learning Challenge .
In the area of ​​numerical computation, I have always been interested in the
Antik library, but unfortunately it depends on the
GNU Scientific Library , which makes it spread under the GPL. There is also
mgl-mat and
LLA .
Conclusion:Combining parts for calculations from
LLA and
mgl-mat and machine learning parts from
CLML and
mgl will solve most problems in this area.
Future work:CLML probably contains a lot of code for working with calculations that can be moved to a separate library of the
SciPy and
NumPy type .
Database
cl-dbi provides a generic interface to various database-specific libraries (cl-postgres, cl-mysql, etc.).
SxQL provides DSL to build secure,
automatically parameterizable SQL queries.
There are two equally full ORMs:
Crane from the author of this review, and
Integral from the author
cl-dbi .
Conclusion:Prevent the use of anything other than cl-dbi.
Future work:Binding for other databases, such as Oracle, exist. Writing
cl-dbi drivers will be the best way to promote and help unify.
Graphics
I have never programmed graphics, so I have little knowledge in this area. There is
CEPL and sister project
Varjo , with an excellent collection of
video tutorials . Of course, there are such low-level libraries as
cl-opengl and
cl-sdl2 .
Conclusion:Promote the use of CEPL, because it is almost complete.
Future work:A high-level OpenGL library, like
pg , would be great.
More libraries in this area, especially for manipulating various meshes and other 3D formats.
Competitiveness
cl-async is probably the most complete solution for all things related to competitiveness. And it is built on top of
libuv , the library behind
Node.js.Other notable libraries in this area:
- STMX : provides support for software transactional memory, and is very impressive.
- lparallel : a complete framework for parallel programming.
Legion libraries simplify concurrency for certain cases.
Conclusion:Future work:In this area there is a lot of space for new ideas.
File formats
Common Lisp has libraries for all popular file formats:
New to JSON libraries is
Jonathan , a very fast JSON generator and parser.
Conclusion:There are too many XML and JSON libraries, this leads to a paralysis of choice.
Future work:YAML parser so that
cl-yaml does not depend on libyaml, it will make the distribution much easier.
Interaction with the system
The UIOP and compatibility level from ASDF contain a huge set of utilities for porting any functionality from querying a host name to running external programs and manipulating environment variables.
Conclusion:Use UIOP for everything. Do not use
cl-fad . Do not use
OSICAT .
Future work:An interesting project in this area would be DSL to generate LLVM IR, either as a string or using the LLVM API bindings. This would allow using Common Lisp as an assembler language with an incredibly powerful macro system. For example, to JIT-compile a highly optimized SIMD-vectorized numerical code.
miscellanea
Benchmarks
Creating a framework like
Criterion in Haskell would be a big step forward.
Parsing
Here I usually choose
esrap , which generates parsers from Lisp-like grammar descriptions. Need more variety in this area. Recently there was a release of
proc-parse , a fast vector parsing library.
Logging
Not much can be said about this, except that Common Lisp has good logging utilities, such as
log4cl and
vom .
Development
Type system
Again, there’s nothing to say, except that Common Lisp has a pretty good type system that is far from being used to its fullest.
Conclusion:CLOS has no competitors among the surviving versions of Flavors, and this is practically the only implementation of OOP in Common Lisp.
Future work:I often look at
trivial-types , a library containing many simple type specifiers. A larger library including
this CDR would be good.
Testing
There are many frameworks for testing, the main ones are
FiveAM and the newer
Prove .
From existing libraries, Prove is a good candidate. Extensible report generators are great, and the general approach to extensibility makes it a good solution to the problem of test frameworks.
Other, older libraries depend on frameworks such as rt or lisp-unit, but they are no longer capable.
Conclusion:Prevent the use of test frameworks other than FiveAM and Prove.
Future work:Writing new test frameworks will be counter productive in the current situation. Work should be focused on the unification of existing ones.
Cloud Testing
Common Lisp supports well the use of services like
Travis and
Coveralls for testing code and tracking coverage in the cloud, respectively. The two main libraries for this are
cl-travis and
cl-coveralls, and both are very easy to use. In my experience, testing and tracking Common Lisp project coverage is definitely simpler than the same thing in a Python project.
Conclusion:cl-travis has several problems, for example, it needs sudo to install implementations, because of this it cannot use the new Docker-based infrastructure in Travis.
Roswell solves this problem, so Roswell is recommended to be used with cl-travis.
Here you can find a tutorial on using Roswell with Travis.
Future work:Supporting many services, such as
Circle CI , is always helpful. Tutorials will also be a good addition.
Documentation
For online, automatically updated documentation in the style of
Read the Docs, there are
Quickdocs , which gets the project API descriptions using
docparser .
There are unexpectedly few documentation generators. I use
Codex to generate documentation. It is written in
CommonDoc , a library that provides a format-independent internal presentation of documents. Codex is designed to be similar to Sphinx: documentation is written in prose, and you insert automatically retrieved API documentation (functions, macros, classes, etc.) into the text using macros.
Conclusion:Promote the use of Quickdocs by referring to it from the README, advising new users, etc.
Future work:I will continue to work on Codex and CommonDoc, and any improvements to CommonDoc (mostly new output formats and macros) will be reflected in Codex.
Unicode
This is essentially a solved problem in most implementations. Sometimes you can stumble upon a problem if some encoding other than UTF-8 is installed, but this is easily fixed, for example in SBCL:
(setf sb-impl::*default-external-format* :utf-8)
I have never encountered other problems with Unicode, even when I interacted with the C libraries.
Conclusion:Some implementations are lagging behind in unicode support, for example ABCL.
Future work:You can fix problems with unicode support by improving the
cl-unicode library.
Package management
Quicklisp is the de facto package manager for Common Lisp. And this is quite enough. Do not destroy what works.
Conclusion:All is well.
Future work:Quicklisp based utilities such as
Quickdocs and
qlot .
Assembly system
ASDF is the de facto assembly system for Common Lisp. Everyone uses it.
Each project has an .asd file, it is called a system definition, indicates the project metadata (author, maintainer, home page, etc.) and components.
For me, this is one of the main attractive features of Common Lisp. In languages ​​like Python, each file imports to itself what it needs, and the project becomes a massive graph of file cross dependencies. In ASDF, you simply list the files of your project in the order in which they are defined. Or you can specify the dependencies between the files and let ASDF calculate the order yourself. The bottom line is that the dependencies are set to clearly and clearly visible.
But enough agitation. The main thing is that ASDF has no competitors, not because no one wants to create an analog, but because ASDF destroyed all alternatives several years ago. And this is good.
Conclusion:All is well.
Future work:More components to ASDF, for example for assembling C / C ++ files. A platform-independent batch manager for loading external C libraries required by Lisp libraries would be incredibly useful.
Development environments
SLIME is the development environment for Common Lisp based on Emacs and the most widespread. The fact that it is done on Emacs is a problem, because telling new users that they must learn Emacs to use SLIME (which is not entirely true) is a significant artificial barrier to learning CL.
Conclusion:Reduce the input barrier to use SLIME.
Future work:Environments other than Emacs cannot harm. A plugin for
Atom that would work with a Swank server would be very useful.
Conclusion
Specific problems
Communication
The number of people creating web applications in Common Lisp and not knowing about Clack is too high. Obviously, there is a problem in informing potential users about good libraries.
Paralysis of choice
When someone asks which library to use for writing code in a particular area, only the best library from that area should be recommended. Saying “you can also take X, Y and Z” only worsens the paradox of choice.
If someone asks for a GUI library, answer "
CommonQt with
Qtools ", do not add "but you can also take
cl-cffi-gtk or
LTK ..."
If you are asked what to use for web applications, share a link to one of the Clack frameworks. Do not talk about Hunchentoot or Wookie or Woo or any other server.
If someone asks which development environment to use, point them to
SLIME . Do not tell about some muddy abandoned project from 2005 that works only with
CLISP .
And most importantly: when you are asked what implementation to use - just answer
SBCL or
CCL . Do not say "well, you can also get an
ECL for embedding or
ABCL to work on a JVM". They are for use in narrower cases, not to give a quick start to a newbie.
Development
Quicklisp downloads
Below is the total number of downloads for the hundred most popular projects from
Quicklisp , between January and July:

Repositories
I subscribed to
this feed on new Lisbon repositories on Github. I also wrote some code for querying the
Newsbeuter base:
(ql:quickload (list :dbi :yason :local-time :group-by)) (defvar *connection* (dbi:connect :sqlite3 :database-name (merge-pathnames #p".newsbeuter/cache.db" (user-homedir-pathname)))) (let* ((output (merge-pathnames #p"lisp-github.json" (user-homedir-pathname))) (query (dbi:prepare *connection* "SELECT pubDate FROM rss_item WHERE feedurl = ?")) (results (mapcar #'(lambda (result) (local-time:unix-to-timestamp (getf result :|pubDate|))) (dbi:fetch-all (dbi:execute query "http://planet.lisp.org/github.atom"))))) (with-open-file (stream output :direction :output :if-exists :supersede :if-does-not-exist :create) (yason:encode (mapcar #'(lambda (group) (list (local-time:format-timestring nil (first group) :format (list :year #\- :month #\- :day)) (1- (length group)))) (group-by:group-by-repeated results :keys (list #'identity) :tests (list #'(lambda (ab) (= (local-time:day-of a) (local-time:day-of b)))))) stream)))
From March 23, 2015 to August 20, 2015, 882 repositories were created.

Thanks
Thanks to
Javier Olaechea for his comments on Unicode support,
Eitaro Fukamachi and
François-René Rideau , and
Gabriel Gonzalez for the original article on the
state of the Haskell ecosystem .