Packages, systems, modules, libraries - WHAT?

According to my observations, at least once a week in the cll list or another Lisp list, “newbies” are confused about what is associated with packages. They talk about package “loading”, “requiring” (requiring) a package, they are surprised that after loading the system one needs to use package markers , etc. It annoys me, I also think that this may be one of the reasons why beginners think that using libraries in Lisp is more difficult than it actually is.



I usually stop trying to write a useful explanation, and, of course, this explanation is very simple. I created this page so that next time I could just send here, instead of explaining the same thing over and over again.



First of all, you should have a clear head. The term “package” is heavily overloaded. In Linux distributions like Debian or Gentoo, there are “packages”, “packages” are in the programming languages ​​Java, Perl or Python. It is likely that you came to Lisp with a preconceived opinion about what a “package” is or what it should be.

')

Packages

A package in Common Lisp is a full-fledged element of a language whose semantics is clearly defined by the standard. Moreover, of all the terms discussed on this page, this is the only one that has (in the context of Common Lisp) an unambiguous definition. Packages are, strictly speaking, containers for characters. We can say that they are needed to help organize the individual namespaces in your programs.



Common Lisp has functions and macros for creating, modifying, exploring, and deleting packages. A very good introduction to the packages (and symbols) can be found in chapter 21 of Peter Sibel's magnificent book Practical Common Lisp . The definition of the term is found in Chapter 11 (online versions) of the ANSI Common Lisp specification .



In general, about packages, that's all. Technically speaking, you do not download packages. You can download (with the help of LOAD ) code, which in turn will create a package, and this is a significant difference.



In addition, if your Lisp complains that it cannot find a package, it means that there is no package as a Lisp object in the image (i.e. FIND-PACKAGE returns NIL ) because no one has created it yet. This does not mean that the Lisp machine looked in the file system and found nothing. (A common cause of such a failure is that events occur in the wrong order. About this below.)

Systems

Systems, unlike packages, are not even mentioned in the standard . However, experienced Lisp programmers know this term because they will need to know and use some kind of system definition tool. The most visible today is ASDF (used by most open source Lisp libraries); Another well-known tool for determining systems that are much older than ASDF is MK: DEFSYSTEM . Some developers also supply their system definition tools along with distributions, see for example Common Defsystem for LispWorks.



In this key, the system, strictly speaking, is a set of code plus instructions for its processing, for example, dependencies on other systems, what should be loaded / compiled first and foremost, etc. In other words, the system definition tool is similar in purpose to make or Ant .



In addition, the system definition tool can usually be much more - Common Defsystem can, for example, integrate COM type library files , ASDF is fully extensible and used, among other things, to compile C files . It is also often used to define test suites of the described system.



Although ASDF is quite popular, it is not omnipresent. It comes pre-installed with many Lisp systems like SBCL, OpenMCL or AllegroCL, most likely it will load in other Lisp systems, but this fact does not make it part of Common Lisp. This is a set of code without explicit specifications and with different versions that are incompatible with each other.

Come understand ...

Modules

The standard defines modules only superficially. There are two things you need to know about REQUIRE , PROVIDE and *MODULES* - this functionality is not recommended (deprecated) and depends on the implementation. Do not be disturbed by the fact that this functionality is not recommended. All distributions today contain the specified functions, and the likelihood that a new ANSI standard will appear and all implementations will suddenly remove them, of course, is small. What you need to worry about is that REQUIRE can be a convenient, but not portable method (if you are, of course, concerned about the mechanisms of portability).



For example, in LispWorks you can use

 (require "foreign-parser") 

to load a parser capable of reading definitions in C , but this does not work on OpenMCL. You can also call

 (require :asdf) 

to load ASDF on OpenMCL, but not in LispWorks.



Some distributions offer hooks to customize how REQUIRE works, and there are extensions like common-lisp-controller that connect REQUIRE with ASDF, but in general a module is a thing that depends on the implementation and should not be confused with systems (ASDF), and , especially with packages.

Libraries

Most likely you will not find a clear definition of what a library is. Most people think of this as a collection of code designed to perform one or more specific tasks and distributed as a whole, usually in the form of a compressed archive that can be downloaded from somewhere. In fact, this vague definition is, I think, most appropriate when talking about programs written in Lisp. Most Lisp libraries today include system definition (ASDF), but this is not at all necessary. Perhaps, depending on the method of receipt, it will be a module in your Lisp system, but this is also not necessary. In addition, the library usually defines one or more packages, and may not define one.



And, by agreement, or perhaps due to a lack of imagination, a situation may arise and often happens when the Ku library goes with the definition of the Ku system, which can be downloaded as a Ku module. After downloading the code, get a new package called “Ku”. Four different entities with the same name! I admit that this is confusing, but I hope that a few previous paragraphs have helped to clarify the situation slightly.

But still nothing works for me!

Often people complain that they cannot compile a file containing code like this:

 ;;        (require :cl-ppcre) (asdf:oos 'asdf:load-op :cl-ppcre) (defun my-simple-number-scanner (string) (cl-ppcre:scan "^[0-9]+$" string)) 

Why is that? Why can I download this file, but I cannot compile it? And why can I compile it after downloading? Isn't it strange?



No, not strange. The compiler reads the first form (which is the instruction to compile - if necessary - and load the CL-PPCRE system, but not to execute it. In the end, the compiler is only interested in compiling the code. After executing the first form, it proceeds to the second form, to the definition of the function. An error message is possible here, since a Lisp scanner trying to read this form will detect the “cl-ppcre: scan” character sequence, which should denote an external character from the CL-PPCRE package, but the CL-PPCRE package itself is not yet. CL-PPCRE system boot, among other things, the CL-PPCRE package is being created, but this has not happened yet. Read chapter 3 of the CLHS.



You can use EVAL-WHEN to tell the compiler to load CL-PPCRE before reading the second form. It should, however, find another way to organize your code. The first form is simply the announcement that your code depends on the CL-PPCRE system. This should not be in the same file as the Lisp code. Write a system definition for your program and place the dependencies there.