How to create an operating system

According to the report of Ivan Ruzanov “Windows - how the operating system is created” from the conference “ Platform 2009 ”.

As already wrote antonms , records of reports from the Platform became available. I looked at a few of them and want to share the ones you like the most. Considering the employment of an ordinary habra-user and the duration of the initial recording of the report (1 hour 20 minutes), I give a detailed presentation of the material below.

• Windows history
• Windows today
• Windows development
• Windows 7 kernel changes
• Maintenance of Windows, updates / patches

Windows history

Few people know when the development of Windows NT began - more than 20 years ago, in November 1988. Rumor has it that Microsoft bought / stole Windows NT. Of course, this is not the case. To work on the new operating system, Microsoft invited Dave Cutler, who worked for DEC and was one of the developers of the DEC VAX VMS operating system. In 1988, the development team of the new operating system was only 12 people, among which, along with Dave Cutler, were also former employees of DEC, who participated in the development of DEC VAX VMS. And all the code for the new Windows NT was written to Microsoft.
The abbreviation "NT" is decoded by marketing as "New Technologies", but in the project documentation, it meant something completely different. The fact is that Windows NT was developed for a new, not released in 1988, Intel i860 processor . Its code name was “N10” ( NT en).
The first version, Windows NT 3.1, was released 5 years later, in 1993. At this point in the team was already 250 developers.

Windows today

• 1 billion users
• 140 million lines of code (including test code and toolkit)
  Windows code is very different. Some parts were written 20 years ago, some appeared only in the current version. For example, the Web Services on Devices (WSD) code in Windows Vista exists in its first version, the GDI code is in the final stages of its development and almost does not change, the DirectX code is already well developed, but it is actively changing now.
• 8000 developers
• 36 localization languages
• 20 years of development

Windows development

20-30 years ago, only one programming methodology, Waterfall, was used. It is a sequence:
Specifications → Design → Implementation → Testing → Supply.
But this methodology works only for small projects. For a product like Windows today, other methodologies are needed:

• Product Cycle Model
• Team Software Process
• “ Extreme Programming ”

All of these methodologies have advantages and disadvantages. Depending on the size of the team and the development stage of the component, different Windows development groups apply different development methodologies.
For Windows, as a product as a whole, the Product Cycle Model is used:

• Periods of 3-4 months
• Inside the period - "waterfall"

The biggest challenge in developing a product of this magnitude is that development takes time. At the initial stage, those problems that exist in the current time and existing means are solved. But the only thing that is constant is that everything will change. Over the years of development:

• Requirements will change
• Opportunities will change
• The schedule will change
• The project will change
• Users will change

Despite the fact that different teams are developing differently, there are "universal" rules:

• Release of intermediate versions (milestones, beta, CTP) for broad masses of testers
• Production of internal assemblies with short cycles (1 day)
• Simplicity and reliability of design
• Personal and team code reading
• Unit tests
• Build Verification Tests
• Any intermediate assembly should be of high quality (what is written should work)

From myself, I note that for the month of working with Windows 7 build 6801 as the main OS on the home computer, I have a positive impression about this build.
')
The entire Windows development process is built around a daily build:

• This is the pulse of the product.
• Development never stops
• Daily Auto Test
• Early Integration
• Developer Responsibility
• Obvious condition of the product

Once there used to be only one source code branch, and all developers made changes directly to it. Now the development team is so big that it does not work. Supported by many branches, among which is the main - WinMain. Each laboratory has its own local development branch into which changes are integrated. Proven changes over time integrate into WinMain.
Daily development cycle:

• 3:00 pm - Admitted to integration changes to the source code control system
• Build 6 versions (Free / Checked - x86, x64, IA64)
• 18:00 - New versions are available for testing.
• The new version is installed on several thousand workstations and servers for testing.
• Automated stress test
• 05:00 - Test protocols are analyzed, failures are diagnosed
• 09:00 - Summary reports automatically sent to teams
• 09:30 - Consolidated meeting of team leaders to set goals

All project participants, including the most senior managers, use intermediate versions on their work (and usually home) computers.

What is Windows written on?

• C, C ++, C #, Assembler (x86, x64, IA64)
  Assemblers are used to a rather limited extent in situations where it cannot be avoided.
• Visual Studio, Source Insight, build, nmake
• Source Depot - source control system
• WinDbg, KD, NTSD - debuggers

Many internal tools, such as build, can be downloaded from microsoft.com/whdc/devtools .

Windows 7 kernel changes

The Windows 7 kernel has undergone the following changes:

• Refactoring
  Why is it impossible to delete the graphics subsystem in Windows?
  From a technical point of view, the answer to this question is that the graphics subsystem in Windows is not independent, it is part of the Win32 subsystem.
  In Windows 7, many low-level components were refactored to break dependencies. It will not be noticeable to users, only new Dll will appear, for example, kernel32.dll is divided into kernel32.dll and kernelbase.dll .
  This split made it possible to isolate a minimal kernel called MinWin (20 megabytes per disk).
• EFI support for x86 and x64 (as in Vista SP1)
  Many manufacturers are trying to get rid of the BIOS in favor of EFI.
• Boot from VHD (virtual hard disk)
• Parallel device initialization and start of services
  When booting Windows, it takes quite a long time to build a tree of devices. The PNP manager must interrogate the bus drivers (PCI, USB, FireWire, etc.) for what devices they have. And most of the time, the processor waits for devices to respond (or not). After all, in order to determine the devices on the bus you need to interview them. If they are, they will answer, and if not, then you have to wait, and the processor is idle. Parallel execution of these tasks reduces load time.
• Remove Dispatcher lock from scheduler and PFN lock from memory manager
  The last few years, the processor clock speeds do not grow, and the development is moving towards an increase in the number of concurrently running instructions both at the level of a single core and at the system level (multicore). In this regard, a lot of work has been done to improve scaling.
  The two hottest locks that were in the core were the Dispatcher lock and the PFN lock.
  Dispatcher lock was used by the scheduler when the state of threads changes. This lock has been removed, and the “waiting” flow state has been divided into several:
  • Waiting: In process
  • Waiting: Completed
  • Waiting: Canceled
  PFN lock was used when changing the attributes of physical memory pages. In a multiprocessor system, each processor requested access to this lock, which led to a lot of time.
• Support for 256 logical processors
  Earlier in Windows, a machine word was used as an affinity mask. This was done because it was so easy to find free processors — every bit is a processor. Accordingly, 32 logical processors were supported in a 32-bit system, and 64 in a 64-bit one.
  In Windows 7, as a result of the transition to the affinity mask segment model, support for 256 logical processors became possible. Processors began to be grouped into groups / segments. Each group can contain up to 64 processors. As a result, backward compatibility is obtained, old programs “see” only processors in one group, and new programs that use new interfaces work with all processors in the system.
• Improved power saving: turning off processor sockets Today, there is a serious problem of power saving not only for owners of laptops, but also for owners of data centers. In the US, 2% of the electricity consumed by computer data centers. Many of them turn off some of their servers during low user activity (weekends).
  It was found that it is much more profitable to disconnect the entire processor socket than one core over several, since in this case, you can disable the entire socket support infrastructure (memory controller).

Windows maintenance, updates

Previously, updates were often cumulative (accumulative). This meant that if the erroneous code was contained in an early update of the component, then later versions will contain this code. But not all users need all the updates, they have a different configuration.
Now after release (RTM) in Windows, there are 2 versions of the source code:

• RTM GDR (General Distribution Release)
  Includes those few changes that are meant for everyone. Mostly security fixes.
• RTM LDR (Limited Distribution Release)
  During the installation of the update, the Windows Update client selects the branch it needs and installs the code from it.

Create security update

The job of creating a security update begins with a vulnerability discovery. There are lots of different detection methods — internal security teams, security partners, developers. When a vulnerability is discovered, 2 parallel processes begin:

• Development fixes for all platforms
• Search for "options"
  Large-scale search for similar vulnerability options on all platforms. Search not identical code, but similar.

After the development of the fix, begin checking its code. When they complete, the fix is ​​integrated into the build, and the build is sent for testing:

• Manual and automatic component testing
• Automatic testing of file format distortions, network components, etc. (more than a million options)
• System testing in general, including backward compatibility testing

Only corrections that meet all quality criteria are allowed to be released to Windows Update and Download Center.

Thanks to everyone who read to the end =)