📜 ⬆️ ⬇️

The state of the Common Lisp ecosystem for 2015

From the translator:
I have often heard that Common Lisp has libraries from the 80s and no other, and many of its users are 3.5 professors working on artificial intelligence in the DARPA catacombs and not even having a close understanding of the everyday tasks of an ordinary programmer. This article shows well that even though you may stumble upon old libraries, you only need to use modern analogues, that existing libraries are developing and new ones are constantly appearing.

The author of this article, Fernando Borretti , is an active contributor to the Common Lisp ecosystem, the author of more than 30 libraries, most of which are for web development.

A passing reader will get a general idea of ​​the state of affairs in Common Lisp, an interested person will be able to understand what he needs in order to try to write code and which libraries to take for a test task, and an experienced developer will learn about the latest developments, understand in which directions he can help the community and get some tips on how to answer newbie questions so as not to kill their interest in this great technology. Go!
')


Here is a description of the Common Lisp ecosystem as of August 2015 from a user and contributor point of view.

The purpose of this article is to provide an overview of the ecosystem and help the introduction of unification in each area.

For each application area, recommendations are given on the unification of this part of the ecosystem, as well as interesting directions for future development.

Application Development



Web development



Backend



Clack , an analogue of WSGI / Rack, has existed since 2009 and is well tested and works in production. Three web frameworks - Caveman2 , Ningle and Lucerne are based on it.

Clack is an HTTP server abstraction that allows a user to write web frameworks (or better web application frameworks) without reference to a specific server implementation.

The importance of using Clack should not be underestimated: if you make an application directly, say, Hunchentoot, you are tied to Hunchentoot, and if a new, faster server such as Woo appears, you will have to rewrite the entire application to use it. If you write a plugin for Clack, for example clack-errors , it can automatically be used by all applications, regardless of the frameworks made on Clack, reducing the useless duplication of code.

With Clack, switching from Hunchentoot to Woo and the joy of tremendous acceleration is just a matter of installing libev and changing one key argument.

Conclusion:

Stop using Hunchentoot directly . Use Clack, or better yet, one of the frameworks based on it.

Future work:
The main part is finished, it is time to write high-level things. An extensible Clack application administration framework like in Django would be a good example.

Frontend



Everything is connected with the ability of Common Lisp to compile into JavaScript. Common Lisp is a fairly large language, with few options:



Conclusion:

The best way to unify is to develop one of the existing implementations of CL-in-JS.

Future work:

Something like CFFI, but for the implementation of CL-in-JS, so that you don’t have to re-write JavaScript libraries to bind when new versions of libraries are released.

A utility to compile Common Lisp to JavaScript independent of the specific implementation, to facilitate the transition from Parenscript to JSCL or other implementations.

Command line



For many years, various utilities have appeared in this area, the last of them, the one that received the biggest impulse - Roswell - the implementation of the manager / installer and the script launcher. One of its great features is the easy compilation of small scripts into executable files, for example, for generating documentation .

It is also used to install implementations (CL) in Travis to bypass some of the problems with cl-travis .

Conclusion:

Kill cl-launch, use Roswell .

Future work:

More Roswell scripts.

Graphical interface



For a long time, the lack of a complete GUI solution has been of great concern. Now we have it: CommonQt plus Qtools . The first is used in real applications for years, and the second is a layer that allows you to make everything easier.

The biggest problem in using CommonQt is the requirement for Smoke to work, and getting libraries to work can be difficult, especially on a system other than Linux. This is solved by Qtools, which depends on the qt-libs library. It downloads Smoke libraries for the current platform and makes customizing and deploying applications much easier.

Conclusion:

Focus on CommonQt and help improve cl-cffi-gtk , while other libraries are considered obsolete.

CLIM is interesting and was the last attempt to explore ways to create user interfaces, but it is not a viable option in 2015.

Future work:

More tutorials and examples of using CommonQt and Qtools.

Machine learning



CLML is quite a multifunctional solution. It was developed by Mathematical Systems Inc. Japanese company. Mike Maul subsequently transferred it to github and cleaned up the code a bit. Available tutorial on time series .

Another candidate in this field is mgl , used by its author to win the Higgs Boson Machine Learning Challenge .

In the area of ​​numerical computation, I have always been interested in the Antik library, but unfortunately it depends on the GNU Scientific Library , which makes it spread under the GPL. There is also mgl-mat and LLA .

Conclusion:

Combining parts for calculations from LLA and mgl-mat and machine learning parts from CLML and mgl will solve most problems in this area.

Future work:

CLML probably contains a lot of code for working with calculations that can be moved to a separate library of the SciPy and NumPy type .

Database



cl-dbi provides a generic interface to various database-specific libraries (cl-postgres, cl-mysql, etc.). SxQL provides DSL to build secure, automatically parameterizable SQL queries.

There are two equally full ORMs: Crane from the author of this review, and Integral from the author cl-dbi .

Conclusion:

Prevent the use of anything other than cl-dbi.

Future work:

Binding for other databases, such as Oracle, exist. Writing cl-dbi drivers will be the best way to promote and help unify.

Graphics



I have never programmed graphics, so I have little knowledge in this area. There is CEPL and sister project Varjo , with an excellent collection of video tutorials . Of course, there are such low-level libraries as cl-opengl and cl-sdl2 .

Conclusion:

Promote the use of CEPL, because it is almost complete.

Future work:

A high-level OpenGL library, like pg , would be great.

More libraries in this area, especially for manipulating various meshes and other 3D formats.

Competitiveness



cl-async is probably the most complete solution for all things related to competitiveness. And it is built on top of libuv , the library behind Node.js.

Other notable libraries in this area:



Legion libraries simplify concurrency for certain cases.

Conclusion:

Future work:

In this area there is a lot of space for new ideas.

File formats



Common Lisp has libraries for all popular file formats:



New to JSON libraries is Jonathan , a very fast JSON generator and parser.

Conclusion:

There are too many XML and JSON libraries, this leads to a paralysis of choice.

Future work:

YAML parser so that cl-yaml does not depend on libyaml, it will make the distribution much easier.

Interaction with the system



The UIOP and compatibility level from ASDF contain a huge set of utilities for porting any functionality from querying a host name to running external programs and manipulating environment variables.

Conclusion:

Use UIOP for everything. Do not use cl-fad . Do not use OSICAT .

Future work:

An interesting project in this area would be DSL to generate LLVM IR, either as a string or using the LLVM API bindings. This would allow using Common Lisp as an assembler language with an incredibly powerful macro system. For example, to JIT-compile a highly optimized SIMD-vectorized numerical code.

miscellanea



Benchmarks



Creating a framework like Criterion in Haskell would be a big step forward.

Parsing



Here I usually choose esrap , which generates parsers from Lisp-like grammar descriptions. Need more variety in this area. Recently there was a release of proc-parse , a fast vector parsing library.

Logging



Not much can be said about this, except that Common Lisp has good logging utilities, such as log4cl and vom .

Development



Type system



Again, there’s nothing to say, except that Common Lisp has a pretty good type system that is far from being used to its fullest.

Conclusion:

CLOS has no competitors among the surviving versions of Flavors, and this is practically the only implementation of OOP in Common Lisp.

Future work:

I often look at trivial-types , a library containing many simple type specifiers. A larger library including this CDR would be good.

Testing



There are many frameworks for testing, the main ones are FiveAM and the newer Prove .

From existing libraries, Prove is a good candidate. Extensible report generators are great, and the general approach to extensibility makes it a good solution to the problem of test frameworks.

Other, older libraries depend on frameworks such as rt or lisp-unit, but they are no longer capable.

Conclusion:

Prevent the use of test frameworks other than FiveAM and Prove.

Future work:

Writing new test frameworks will be counter productive in the current situation. Work should be focused on the unification of existing ones.

Cloud Testing



Common Lisp supports well the use of services like Travis and Coveralls for testing code and tracking coverage in the cloud, respectively. The two main libraries for this are cl-travis and cl-coveralls, and both are very easy to use. In my experience, testing and tracking Common Lisp project coverage is definitely simpler than the same thing in a Python project.

Conclusion:

cl-travis has several problems, for example, it needs sudo to install implementations, because of this it cannot use the new Docker-based infrastructure in Travis.

Roswell solves this problem, so Roswell is recommended to be used with cl-travis. Here you can find a tutorial on using Roswell with Travis.

Future work:

Supporting many services, such as Circle CI , is always helpful. Tutorials will also be a good addition.

Documentation



For online, automatically updated documentation in the style of Read the Docs, there are Quickdocs , which gets the project API descriptions using docparser .

There are unexpectedly few documentation generators. I use Codex to generate documentation. It is written in CommonDoc , a library that provides a format-independent internal presentation of documents. Codex is designed to be similar to Sphinx: documentation is written in prose, and you insert automatically retrieved API documentation (functions, macros, classes, etc.) into the text using macros.

Conclusion:

Promote the use of Quickdocs by referring to it from the README, advising new users, etc.

Future work:

I will continue to work on Codex and CommonDoc, and any improvements to CommonDoc (mostly new output formats and macros) will be reflected in Codex.

Unicode



This is essentially a solved problem in most implementations. Sometimes you can stumble upon a problem if some encoding other than UTF-8 is installed, but this is easily fixed, for example in SBCL:

(setf sb-impl::*default-external-format* :utf-8) 


I have never encountered other problems with Unicode, even when I interacted with the C libraries.

Conclusion:

Some implementations are lagging behind in unicode support, for example ABCL.

Future work:

You can fix problems with unicode support by improving the cl-unicode library.

Package management



Quicklisp is the de facto package manager for Common Lisp. And this is quite enough. Do not destroy what works.

Conclusion:

All is well.

Future work:

Quicklisp based utilities such as Quickdocs and qlot .

Assembly system



ASDF is the de facto assembly system for Common Lisp. Everyone uses it.

Each project has an .asd file, it is called a system definition, indicates the project metadata (author, maintainer, home page, etc.) and components.

For me, this is one of the main attractive features of Common Lisp. In languages ​​like Python, each file imports to itself what it needs, and the project becomes a massive graph of file cross dependencies. In ASDF, you simply list the files of your project in the order in which they are defined. Or you can specify the dependencies between the files and let ASDF calculate the order yourself. The bottom line is that the dependencies are set to clearly and clearly visible.

But enough agitation. The main thing is that ASDF has no competitors, not because no one wants to create an analog, but because ASDF destroyed all alternatives several years ago. And this is good.

Conclusion:

All is well.

Future work:

More components to ASDF, for example for assembling C / C ++ files. A platform-independent batch manager for loading external C libraries required by Lisp libraries would be incredibly useful.

Development environments



SLIME is the development environment for Common Lisp based on Emacs and the most widespread. The fact that it is done on Emacs is a problem, because telling new users that they must learn Emacs to use SLIME (which is not entirely true) is a significant artificial barrier to learning CL.

Conclusion:

Reduce the input barrier to use SLIME.

Future work:

Environments other than Emacs cannot harm. A plugin for Atom that would work with a Swank server would be very useful.

Conclusion



Specific problems



Communication



The number of people creating web applications in Common Lisp and not knowing about Clack is too high. Obviously, there is a problem in informing potential users about good libraries.

Paralysis of choice



When someone asks which library to use for writing code in a particular area, only the best library from that area should be recommended. Saying “you can also take X, Y and Z” only worsens the paradox of choice.

If someone asks for a GUI library, answer " CommonQt with Qtools ", do not add "but you can also take cl-cffi-gtk or LTK ..."

If you are asked what to use for web applications, share a link to one of the Clack frameworks. Do not talk about Hunchentoot or Wookie or Woo or any other server.

If someone asks which development environment to use, point them to SLIME . Do not tell about some muddy abandoned project from 2005 that works only with CLISP .

And most importantly: when you are asked what implementation to use - just answer SBCL or CCL . Do not say "well, you can also get an ECL for embedding or ABCL to work on a JVM". They are for use in narrower cases, not to give a quick start to a newbie.

Development



Quicklisp downloads



Below is the total number of downloads for the hundred most popular projects from Quicklisp , between January and July:



Repositories



I subscribed to this feed on new Lisbon repositories on Github. I also wrote some code for querying the Newsbeuter base:

 (ql:quickload (list :dbi :yason :local-time :group-by)) (defvar *connection* (dbi:connect :sqlite3 :database-name (merge-pathnames #p".newsbeuter/cache.db" (user-homedir-pathname)))) (let* ((output (merge-pathnames #p"lisp-github.json" (user-homedir-pathname))) (query (dbi:prepare *connection* "SELECT pubDate FROM rss_item WHERE feedurl = ?")) (results (mapcar #'(lambda (result) (local-time:unix-to-timestamp (getf result :|pubDate|))) (dbi:fetch-all (dbi:execute query "http://planet.lisp.org/github.atom"))))) (with-open-file (stream output :direction :output :if-exists :supersede :if-does-not-exist :create) (yason:encode (mapcar #'(lambda (group) (list (local-time:format-timestring nil (first group) :format (list :year #\- :month #\- :day)) (1- (length group)))) (group-by:group-by-repeated results :keys (list #'identity) :tests (list #'(lambda (ab) (= (local-time:day-of a) (local-time:day-of b)))))) stream))) 


From March 23, 2015 to August 20, 2015, 882 repositories were created.



Thanks



Thanks to Javier Olaechea for his comments on Unicode support, Eitaro Fukamachi and François-René Rideau , and Gabriel Gonzalez for the original article on the state of the Haskell ecosystem .

Source: https://habr.com/ru/post/265589/


All Articles