Linux kernel documentation goes to Python Sphinx
The Linux kernel today is the most dynamic, complex, large open source project . How are things going with his documentation? There is a direct connection: the better and more accessible the project documentation, the easier it is for outsiders to learn the basics of the case, get comfortable and become a full participant.
At the Kernel Recipies seminar, the maintainer of the Linux documentation of the kernel, Jonathan Corbet, spoke about the current state of documentation and how the transition from anarchy to order will be made. The first success in this endeavor is already there . Some documents were recently converted to ReStructuredText using the Python Sphinx. About how it was told inside.
What does the Documentation folder contain?
Documentation is the only Linux directory that starts with a capital, it underlines its special role and position, but it will break a leg inside the devil.
- 2264 files
- 228 directories
- 23 MB of documentation, not including
Documentation/devicetree
Documentation / * is a gigantic mess, currently based on where passers-by put things down last.
- Rob Landley, July, 2007.
All this exists in two ways: ordinary txt files and DocBook templates, which have something to tell. Simple txt files are not related to each other , written at different times by different people and not all of them are equally useful. Some of them are known to all kernel developers. These are ManagementStyle , written by Linus or CodingStyle , from which all volunteers begin. I think these documents should be read by all developers, Linux users and sympathizers.
Btw, when talking about the "kernel manager", it’s If you sign, you’ll not be a kernel manager. May not apply to you.
If you sign spending orders and are good at budgeting, then you are definitely not a senior kernel developer.
Lost relevance docks apparently no different from the rest. For example, Documentation/applying-patches.txt instructs how to download patches via ftp and apply using the patch command. It is clear that today no one does.
# moving from 2.6.11 to 2.6.12 $ cd ~/linux-2.6.11 # change to kernel source dir $ patch -p1 < ../patch-2.6.12 # apply the 2.6.12 patch $ cd .. $ mv linux-2.6.11 linux-2.6.12 # rename source dir # moving from 2.6.11.1 to 2.6.12 $ cd ~/linux-2.6.11.1 # change to kernel source dir $ patch -p1 -R < ../patch-2.6.11.1 # revert the 2.6.11.1 patch # source dir is now 2.6.11 $ patch -p1 < ../patch-2.6.12 # apply new 2.6.12 patch $ cd .. $ mv linux-2.6.11.1 linux-2.6.12 # rename source dir Excavations sometimes produce surprising results. It turns out that Linux has been supporting Klingon writing since 1996. We look in Documentation/unicode.txt .
U+F8D0 KLINGON LETTER A U+F8D1 KLINGON LETTER B U+F8D2 KLINGON LETTER CH U+F8D3 KLINGON LETTER D U+F8D4 KLINGON LETTER E U+F8D5 KLINGON LETTER GH U+F8D6 KLINGON LETTER H U+F8D7 KLINGON LETTER I U+F8D8 KLINGON LETTER J U+F8D9 KLINGON LETTER L U+F8DA KLINGON LETTER M U+F8DB KLINGON LETTER N U+F8DC KLINGON LETTER NG U+F8DD KLINGON LETTER O U+F8DE KLINGON LETTER P U+F8DF KLINGON LETTER Q The document Documentation/zorro.txt fraught with the secrets of programming drivers for the PC family Amiga. Not surprisingly, the dock was last ruled in 2003.
Some texts are so furious that few people will understand. Jonathan took Documentation/memory-barriers.txt . Those who understand what they are talking about, he says, think better than him and many others.
There is also a lot of junk in the catalogs of the lower level, this document is probably one of the most dense. It's funny, a hundred years ago, I also had an email on usa.net, but after the collapse of the DotCom market, it became paid and I moved to Yahoo.
(5:521)$ less Documentation/sound/oss/MultiSound #! /bin/sh # # Turtle Beach MultiSound Driver Notes # -- Andrew Veliath <andrewtv@usa.net> # # Last update: September 10, 1998 # Corresponding msnd driver: 0.8.3 It is noteworthy that the sound subsystem OSS from the kernel is almost completely drunk, but the documentation is still there.
Now, from the pure anarchy of plaintext files, we move on to the second, much more complicated form of kernel documentation.
Docbook and guys
These 34 XML templates with all the processing infrastructure are the DocBook. Unlike plain text, DocBook formatting allows you to structure the description of the API, functions, and get the output html, pdf or man page, interconnected by cross-references.
(5:522)$ ll Documentation/DocBook/*.tmpl |wc -l 34 The source code contains special inserts - kerneldoc comments . Such inserts in the code over 55 thousand.
/** * list_add - add a new entry * @new: new entry to be added * @head: list head to add it after * * Insert a new entry after the specified head. * This is good for implementing stacks. */ The DocBook template contains instructions on how to unload documentation from sources .
/* * Parse file, calling action specific functions for: * 1) Lines containing !E * 2) Lines containing !I * 3) Lines containing !D * 4) Lines containing !F * 5) Lines containing !P * 6) Lines containing !C * 7) Default lines - lines not matching the above */ An illustration from the Documentation/DocBook/networking.tmpl file.
<sect1><title>Socket Buffer Functions</title> !Iinclude/linux/skbuff.h !Iinclude/net/sock.h !Enet/socket.c !Enet/core/skbuff.c !Enet/core/sock.c !Enet/core/datagram.c !Enet/core/stream.c </sect1> After the user types the make htmldocs the scripts/docproc parses the templates in the documentation export instruction and performs the following actions for each instruction.
- Finds in the specified files the instructions
EXPORT_SYMBOL(), parses them and adds the names of the functions to the list of exported symbols. - It causes the Perl script
scripts/kernel-doc, which runs through functions, structures, and everything else that it finds in C code in order to get the necessary definitions out. - It
scripts/kernel-docover the second round and it reads the same files again, but this time it creates the documentation of those same functions and packages the result into DocBook snippets.
Then docproc stuffs the created files with DocBook formatting into templates. For HTML, the scripts/kernel-doc-xml-ref script is invoked, which creates cross-references, but only for a single template. Finally, xmlto produces the documentation in the specified format, but you can also specify an alternative program in the Makefile .
This jumble of scripts, non-dull XML templates, hellish regulars [1] was unreliable and inhibited [2] , and most importantly - did not produce pleasing to the eye, coherent documentation.
Markdown, Asciidoc, Sphinx
It is clear that this situation did not suit anyone, and Jonathan and his colleagues were looking for a way out of the situation. Having reviewed and tried several options, at the end they stopped at a bunch of ReStructuredTtext and Sphinx .
Principles were first formulated. The new documentation system must meet the requirements.
- Eliminate the mess.
- Improve formatting and readability.
- Plaintext.
- Rationally organized toolkit.
The use of lightweight markup languages suggested itself, and the first attempt was Markdown . This made it possible to solve several problems at once. First , comments written in the program itself will be updated along with it. The idea is generally correct, albeit optimistic. Secondly , the documentation, which is easier to concoct, has a better chance of being written, since the text MarkDown has a much lower threshold of entry than DocBook-XML .
Soon, however, it became clear that MarkDown could not be used by itself, but only as an add-on to the current set of programs. I had to add the MarkDown support to the regular kernel-doc MarkDown , which required the additional conversion utility pandoc (and then asciidoc ). The tools were clearly out of tune with each other, the markup floated, debugging became more complicated. The brakes became more, and the toolkit became even more capricious.
Then Jonathan realized that the way out of the situation must be sought, moving in the opposite direction. Instead of complicating the chain of formatting and mutual transformations, you should take and throw out something superfluous from there . This was superfluous DocBook and everything around it. What if everything is transferred to AsciiDoc and from it directly create the necessary html, pdf and man pages, bypassing the DocBook infrastructure?
Easy to say, hard to implement. AsciiDoc pulled Ruby's dependencies, badly contacted xmlto and still complemented DocBook rather than replacing it, and the linking of documents still didn’t grow. However, it was decided to move forward instead of waiting for something unclear. At the last moment, Jonathan took a little time out, once again weighed the pros and cons and suggested adopting the Sphinx python system of documentation , based on the lightweight RestructuredText markup.
Why sphinx ?
- Designed to document software. For example, he knows what a function is.
- Good for large documents scattered across multiple files.
- Often used and has good support. [3] .
- Documents are finally linked.
- Tulchan simplified, faster assembly.
In order to convince colleagues, Jonathan wrote a crooked Proof of Concept on his knee, and then Jani Nikula polished it and brought it to mind. Pretty soon a consensus emerged amongst the documentalists on this issue and everything began to turn out. By the way, the trick with PoC, always works according to Andrew Morton, the main container of the mm kernel branch. This is how Linux appeared, and so often controversial moments are solved when it is necessary to replace a terrible solution with an ideal one.
What has been done and what remains to be done
The changes are already partially visible, the Sphinx settings are stored in Documentation/Makefile.sphinx and Documentation/conf.py There is also an index file index.rst and kernel-documentation.rst . The developers of gpu and media ahead of the rest and have already translated all documentation Documentation/gpu/ into Sphinx .
The transition to Sphinx will take place without shock therapy, i.e. plaintext files and DocBook templates will be gradually converted to a new format in coordination with the rest of the maintainers. So while DocBook, will not go anywhere, however for new rst documents docproc no longer needed.
Version 4.8 is the first stable Linux kernel with a new documentation system. Linus commented on these changes in the letter, where he was surprised to say that 20 percent of the patch is documentation.
It wasn’t much of a window that has been noticed.
Linux Torvalds (4.8-rc1 release)
A good example is the HSI document. Until now, it has existed in two ways: plaintext and DocBook. Documents are not related to each other.
Documentation/hsi.txtDocumentation/DocBook/device-drivers.tmpl
Starting with Linux 4.9, this will be a single document Documentation/drivers-api/hsi.rst . And this is only the beginning of a big way!
A lot of work ahead. The long-term goal is to completely get rid of DocBook and the twenty-year Perl magic kernel-doc . You should convert more than two thousand text files into ReStructuredText , set a strict and understandable file system hierarchy in Documentation , so that there is the same order as in the kernel source space.
If you wish to participate in cleaning augia stables Reform documentation Linux kernels, dare, special knowledge is not required for this, although the knowledge of the subject area, of course, not superfluous. It is equally important to learn new things, interact with all interested parties and adhere to healthy conservatism and the principle of gradualness.
Used materials
- The present and future of formatted kernel documentation
- Kernel documentation with Sphinx, part 1: how we got here
- Kernel documentation with Sphinx, part 2: how it works
')
Source: https://habr.com/ru/post/316758/
All Articles