How to start and do not stop writing OS

Reading Habr during the last two years, I have seen only a few attempts to develop the OS (if specifically: from pehat and iley users (postponed indefinitely) and Igor1024 (not abandoned, but so far more like the description of the work of protected mode x86-compatible processors, which undoubtedly also needs to be known for writing an OS under x86; and a description of the finished system from alman (though not from scratch, although there is nothing bad about it, maybe even vice versa)). For some reason I think that almost all system programmers (and some of them are applied) at least once, but thought about writing their own operating system. In this connection, 3 OS from a large community of this resource seems a ridiculous number. Apparently, most people who think about their own OS do not go further than that, and a small part stops after writing the bootloader, few write kernel pieces, and only hopelessly hard-nosed ones create something remotely resembling an OS (if compared with something like Windows / Linux) . There are many reasons for this, but the main thing in my opinion is that people abandon the development (some even fail to start) due to the small number of descriptions of the process of writing and debugging the OS, which is quite different from what happens during development application software.

I would like to show this small note that, if I start correctly, there is nothing particularly difficult in developing my own OS. Under the cut is a brief and fairly general guide to action on writing the OS from scratch.

How not to start

Please do not take the following text as an obvious criticism of someone's articles or manuals on writing the OS. It’s just too often in such articles under loud headlines the emphasis is on the implementation of some minimal workpiece, and it is served as a prototype of the core. In fact, you should think about the structure of the kernel and the interaction of parts of the OS as a whole, and consider that prototype as a standard “Hello, World!” - an application in the world of application software. As a small excuse for these remarks, it should be said that below there is a subsection “Hello, World!”, Which in this case is given exactly as much attention as needed, and no more.
')
Do not write a bootloader. Smart people came up with the Multiboot Specification , implemented and described in detail what it is and how to use it. I do not want to repeat myself, I just say that it works, makes life easier, and it should be used. Specification, by the way, it is better to read completely, it is small and even contains examples.

No need to write the OS completely in assembler. It's not that bad, rather the opposite - fast and small programs will always be in high esteem. Just because this language requires much more development, the use of assembler will only lead to a decrease in enthusiasm and, as a result, to throwing the OS sources into a long box.

No need to load a custom font in the video memory and display something in Russian. No sense from this. It is much easier and more versatile to use English, and leave the font change for later, loading it from the hard disk through the file system driver (at the same time there will be an additional incentive to do more than just start).

Training

To begin with, as always, you should familiarize yourself with the general theory in order to have some ideas about the forthcoming amount of work. Good sources on this issue are the books of E. Tanenbaum, which have already been mentioned in other articles on writing OS on Habré. There are also articles describing existing systems, and there are various guides / newsletters / articles / examples / sites with a bias in OS development, links to some of which are given at the end of the article.

After the initial educational program, you need to decide on the main issues:

• The target architecture is x86 (real / protected / long mode), PowerPC, ARM, ...
• kernel architecture / OS - monolith, modular monolith, microkernel, exo-kernel, various hybrids
• language and its compiler - C, C ++, ...
• kernel file format - elf, a.out, coff, binary, ...
• development environment (yes, it also plays an important role) - IDE, vim, emacs, ...

Further it is necessary to deepen knowledge according to the chosen and in the following areas:

• video memory and work with it - the conclusion as proof of work is necessary from the very beginning
• HAL (Hardware Abstraction layer) - even if the support of several hardware architectures does not plan to properly separate the lowest-level parts of the kernel from the implementation of such abstract things as processes, semaphores, and so on, it will not be superfluous
• memory management - physical and virtual
• performance management - processes and flows, their planning
• device management - drivers
• virtual file systems - to provide a single interface to the contents of various filesystems
• API (Application Programming Interface) - how exactly applications will access the kernel
• IPC (Interprocess Communication) - sooner or later, processes will have to interact

Instruments

Considering the chosen language and development tools, you should choose such a set of utilities and their settings, which in future will allow, by writing scripts, to simplify and speed up the assembly, preparing the image and launching the virtual machine with the project as much as possible. Let us dwell in more detail on each of these points:

• Any standard tools such as make, cmake, ... are suitable for building. Scripts for the linker and (specially written) utilities for adding a Multiboot header, checksums or for any other purposes can be used here.
• the preparation of the image means its mounting and copying files. Accordingly, the image file format must be selected so that it is supported by both the mount / copy utility and the virtual machine. Naturally, no one forbids to perform actions from this point either as the final part of the assembly, or as preparation for the launch of the emulator. It all depends on the specific means and selected options for their use.
• starting a virtual machine of labor does not represent, but you need to remember to unmount the image first (unmounting at this point, since there is no real sense in this operation before starting the virtual machine). Also, the script for running the emulator in debug mode (if one is available) will not be superfluous.

Hello, World!

If all the previous steps are completed, you should write a minimal program that will boot as a kernel and display something on the screen. In case of inconvenience or shortcomings of the chosen means, it is necessary to eliminate them (shortcomings), or, at worst, take them for granted.

In this step, you need to test as many features of the development tools that you plan to use in the future. For example, loading modules into GRUB or using a virtual machine with a physical disk / partition / flash drive instead of an image.

After this stage is successful, the real development begins.

Providing run-time support

Since it is proposed to write in high-level languages, care should be taken to provide support for some of the language tools that are usually implemented by the authors of the compiler package. For example for C ++, these include:

• function to dynamically allocate a block of data on the stack
• work with heap
• data block copy function (memcpy)
• function-entry point to the program
• calls to constructors and global object destructors
• a number of functions for working with exceptions
• stub for unrealized pure-virtual functions
• ...

When writing “Hello, World!”, The absence of these functions may not be felt in any way, but as you add code, the linker will start complaining about unmet dependencies.

Naturally, you should immediately mention the standard library. A full implementation is not necessary, but the main subset of functions is worth realizing. Then writing code will be much more familiar and faster.

Debugging

Do not look at debugging at the end of the article. In fact, this is a very serious and difficult question in the development of the OS, since the usual means are not applicable here (with some exceptions).

Can advise the following:

• a matter of course debugging output
• assert with immediate debugger access (see next paragraph)
• some semblance of a console debugger
• check if the emulator does not allow to connect a debugger, symbol tables or something else

Without the debugger built into the kernel, finding errors has a very real chance of becoming a nightmare. So from its writing at some stage of development just can not escape. And since this is unavoidable, it is better to start writing it in advance and thus significantly facilitate development and save a lot of time. It is important to be able to implement the debugger in a manner independent of the kernel, so that debugging has a minimal effect on the normal operation of the system. Here are some types of commands that may be helpful:

• part of the standard debugging operations: breakpoints, call stack, output values, print dump, ...
• commands for outputting various useful information, such as the execution queue of the scheduler or various statistics (it is not as useless as it may seem at first)
• commands for checking the consistency of the state of different structures: free / used memory lists, heap or message queue

Development

Next, you need to write and debug the basic elements of the OS, which at the moment should ensure its stable operation, and in the future - easy extensibility and flexibility. In addition to memory managers / processes / (something else), the interface of drivers and file systems is very important. Their design should be approached with great care, taking into account the variety of types of devices / FS. Over time, they can of course be changed, but this is a very painful and error-prone process (and debugging the kernel is not an easy task), so just remember - think about these interfaces at least ten times before you take up their implementation.

Similarity SDK

As the project develops, new drivers and programs should be added to it. Most likely, some common features (directory structure, assembly management files, specification of dependencies between modules, duplicate code in main or in system request handlers (for example, if the drivers themselves check their compatibility with the device )). If so, then this is a sign of the need to develop templates for various types of programs for your OS.

There is no need for documentation describing the process of writing one or another type of program. But it is worth making a blank of standard elements. This will not only make it easier to add programs (which you can do by copying existing programs and then changing them, but it will take more time), but it will also make it easier to update them with changes in interfaces, formats or something else. It is clear that such changes ideally should not be, but since the development of the OS is not a typical thing, there are a lot of places for potentially wrong decisions. But the understanding of the fallacy of the decisions taken will always come some time after their implementation.

Next steps

In short, then: read about the operating systems (and first of all, about their device), develop your system (the pace is not really important, the main thing is not to stop at all and return to the project from time to time with new forces and ideas) and it is natural to correct errors in it (in order to find which it is sometimes necessary to start the system and “play” with it). Over time, the development process will become easier and easier, mistakes will occur less frequently, and you will be enrolled in the list of "hopelessly hard-nosed", those few who, despite some absurdity in the idea of ​​developing their own OS, have done so.

useful links

On Habré:
Writing Your OS: Release 1
Writing Your OS: Release 2
We continue to write OSes. Step by step
What is Protected Mode and what it is eaten with
We learn system of paging addressing and interrupt handling
We start talking about multitasking
Memory and tasks

Other sources with a lot of information on OS development:
wasm.ru
sysbin.com
wiki on osdev.org
iakovlev.org
Bona fide os