Operating System Migration History
From the translator: I present to your attention the translation of the article by Michael Steil . I have long wanted to prepare a similar review of how to use virtualization for compatibility tasks. I even posted some notes on this topic: in the simulation tutorial, Chapter 1 , and on Habré in a post about system VMs . However, I was not able to open the question as deeply as it is presented in this work. So I decided to share the translation with the readers.
Operating system vendors face this problem once or twice a decade: they need to transfer their user base from the old operating system to their very different new OS, or they need to switch from one CPU architecture to another while maintaining the ability to run old applications without modifications. and also help third-party developers port their applications to the new OS.
Let's look at how this happened in the past 30 years, using the examples of MS DOS / Windows, Apple Macintosh, Amiga OS and Palm OS.
')
CP / M is an 8-bit OS from Digital Research that worked on all versions of systems based on the Intel 8080. 86-DOS from Seattle Computer Products, later known as MS-DOS (also called PC-DOS on IBM machines), was a clone of CP / M for later Intel 8086; The situation was similar with CP / M-86 from DR itself (this OS was later called DR-DOS).
Not being binary compatible, the Intel 8086 was “assembly level compatible” from the 8080. This meant that it was easy to convert an 8-bit program written in 8080 or Z80 assembler to an 8086 assembler program, since these two architectures were very similar (backward-compatible memory model, easily mapped register sets), only machine coding of commands differed.
Since MS-DOS used the same ABI conventions and a memory card, this OS was source-level compatible with CP / M. On CP / M systems with access to a maximum of 64 kB of memory, the address range from 0x0000 to 0x0100 (zero page) was reserved for the needs of the OS and contained, among other things, command line arguments. The executable application was located starting at address 0x0100 and above, and the OS was located at the top of the memory; The application stack grew down, starting just below the start of the OS. The 8086 memory model divides the memory into (overlapping) contiguous 64-kbyte segments, so each of these segments is actually sufficient for the 8080 virtual machine. Therefore, the MS-DOS .COM executable files are up to 64 kB in size and loaded into memory starting at 0x100. The range 0x0 - 0x100 in MS-DOS is referred to as Program Segment Prefix and is very similar to the CP / M zero page. Due to the high degree of compatibility of these processors and their ABI program transfer from CP / M to DOS was relatively painless. However, such a port could use a maximum of 64 kbytes of memory (examples - 1982 Wordstar 3.0). But this made it possible to have a single code base for both operating systems with just a few macros and two different assembler programs.
MS-DOS 2.0 brought new, more powerful API conventions (file handles instead of FCB structures, subdirectories, .EXE format) and turned most of the APIs from CP / M into outdated. But DOS was compatible with CP / M up to the latest version.
Microsoft Windows was originally designed as a graphical shell on top of MS-DOS. All work with devices and the file system took place through DOS API calls. Therefore, all MS-DOS drivers were executed directly, and Windows could use them. DOS applications could still run after exiting Windows to pure DOS.
In Windows / 386 2.1, the model has been changed. This was already the real OS kernel, running many virtual machines in virtual 8086 (V86) mode, side by side; one was allocated for the MS-DOS OS session, and one was allocated for each Windows application. The DOS virtual machine was used by Windows to access the device and file system drivers, so it was practically a driver compatibility layer that was executed inside the VM. A user could run any number of additional DOS VMs to work with DOS applications; and each contained its own copy of DOS. Windows intercepted calls from the screen's memory, as well as some system calls, and redirected them from the Windows graphics driver or to the first “main” VM DOS.
Windows 3.x started using native Windows drivers, which replaced inward DOS VMs, and also started redirecting some DOS calls to devices inside Windows. The standard installation of Windows 95 did not at all use an appeal to the DOS VM for the needs of working with drivers and the file system, however, if necessary, such a mode could still be used.
DOS was not only a compatibility environment for old drivers and applications, it also provided the Windows command line, so when Windows 95 came up with support for long file names, it was necessary to intercept DOS API calls to provide new functionality for command line utilities.
The Windows NT family of systems were not based on DOS; nevertheless, from the very first version they allowed to run MS-DOS applications. As in the case of the non-NT version of Windows, in this case the DOS program worked in the V86 mode of the processor. However, instead of using a full copy of MS-DOS inside the VM, NT placed only application code in it, intercepted all system calls and device calls, and wrapped them in NT API calls. In fact, this method is not a pure VM: the V86 mode was used only to create the memory model needed to support DOS applications.
There is a widespread misconception that the Windows NT command line is “DOS shell”. In fact, the interpreter and auxiliary utilities were native NT applications, and the NTDVM subsystem (the DOS virtual machine) did not start at all until it was really necessary to start the DOS program from the command line.
Beginning with the release of Windows NT 3.1 in 1993, it became clear that it would eventually replace the classic Windows. Despite the fact that NT had the same Win16 user interface and good compatibility with DOS, each newer version usually required a fairly powerful computer to work. Therefore, the migration from Windows to Windows NT was the opposite: each version of Windows became more and more like NT until they became quite similar. By that time, even the weaker computers of that era had become powerful enough to work under NT.
A big step towards bringing Windows and Windows NT closer is support for the native Win32 API for the NT. The first step was a free update of “Win32S” to Windows 3.1, which provided a subset (hence “S” - subset - in the name) of the Win32 API on classic Windows. Win32S expanded the Windows kernel with the ability to create 32-bit address spaces for all 32-bit applications (in NT, a separate space was created for each application). It also made available versions of some libraries from NT (for example, RICHED32.DLL), as well as 32-bit DLLs that accepted calls from low-level Win32API calls (the “GDI” and “USER” subsystems) were automatically translated into calls to the Win16 subsystem (“thunking ").
In Windows 95, this functionality was available by default. She also used isolated address spaces for 32-bit applications, and supported a larger subset of the Win32 API. Some basic applications have become 32-bit (for example, Windows Explorer), but much of the basic system was still 16-bit. With the advent of Windows 95, most developers switched to creating 32-bit applications, which also made it possible to compile them on Windows NT systems.
The second step of migration from 16-bit Windows to Windows NT occurred when switching from Windows ME to Windows XP (NT-family) in 2001. Windows NT (2000, XP ...) was a completely 32-bit OS with Win32 API, but also allowed to execute 16-bit Windows applications through the translation of Win16 calls to Win32 (thunking).
The Windows NT 3.1 / 3.5 / 4.0 driver model (Windows NT Driver Model) differed from the classical Windows (VxD) approach, therefore Windows 98 (heir to Windows 95) and Windows 2000 (heir to Windows 4.0) both supported the new “Windows Driver Model”. The same driver could now work in both OSs, but each continued to maintain its own original mode for old drivers.
When Microsoft transferred home users to NT systems, most applications, games, and drivers were made on Windows XP. The only thing that had to be rewritten is system utilities.
The transition from 32-bit Windows to 64-bit is currently taking place. Windows XP was the first Microsoft OS available for Intel 64 / AMD64 architecture, all subsequent systems (Vista, 7, 8, 10) are available in 32 and 64 bit versions. In 64-bit editions, the kernel works in 64-bit mode, like all system libraries and most applications. The 32-bit API is supported through the WOW64 subsystem (Windows-on-Windows 64). A 32-bit application is linked to 32-bit libraries, however low-level API calls are translated via WOW64 into analogues for 64-bit DLLs.
Since the drivers are executed in the same address space as the kernel, it was difficult to ensure the support of 32-bit drivers, so it is not there. Support for DOS and Win16 applications in 64-bit versions of the OS has also been discontinued.
Apple transferred its computers from Motorola 68k processors to Motorola / IBM PowerPC from 1994 to 1996. Since the Macintosh operating system, System 7, was mainly written in 68k assembler, it could not be easily converted to PowerPC OS. Instead, most of it was played through a simulation. The new “nano-core” received and processed interrupts and performed minimal memory management, abstracting the PowerPC features, and the integrated 68k emulator executed the old OS, which was modified to bind to the nano-core. Thus, System 7.1.2 for PowerPC was actually paravirtualized and executed on top of a very thin layer of the hypervisor.
The first Mac OS for PowerPC version executed most of the system code inside the emulator for 68k, including drivers. Part of the code that is critical for performance, was rewritten for direct execution. The executable loader could detect programs with PowerPC and could execute them directly. Most of the communications with the OS still went through the emulator. In later versions of Mac OS, an increasing proportion of the code corresponded from 68k to PowerPC.
Just as Microsoft moved users from Windows to Windows NT, Apple switched from classic Mac OS to Mac OS X. While classic Mac OS was a hackish operating system with cooperative multitasking, without memory protection, it still played the part system code inside the 68k emulator, Mac OS X was based on NEXTSTEP, a modern UNIX-like OS with a completely different API.
When Apple decided to migrate, they ported the classic Mac OS system libraries (“Toolbox”), while removing calls that could not be supported on a modern OS, replacing them with alternatives, and called the new API “Carbon”. The same API became available in classic Mac OS 8.1 in 1998, so that developers were able to update their applications to work in Mac OS X, while maintaining compatibility with classic Mac OS. When Mac OS X became publicly available in 2004, applications that were “carbonized” by that time could work on both operating systems without recompiling. This is reminiscent of Microsoft’s approach to bringing Windows NT closer to Windows NT.
But since it was not expected that all applications would be converted into a “carbonized” form by the time Mac OS X appeared, the new OS also contained a virtual machine “Classic” or “Blue Box”, in which unmodified Mac OS 9 was executed and could support arbitrary number of old applications. Requests to the network and file system inside the VM were intercepted and redirected to the host OS, and integration with the window manager allowed the two desktop environments to look almost seamless.
In 2005, Apple announced a second transition from one CPU architecture to another, this time from PowerPC to Intel IA-32. Since this OS was mostly written in C and Objective C, it could easily be ported to IA-32. Apple claims that they have always supported the OS version for IA-32, starting from the very first release.
In order to run old applications that have not yet been ported to IA-32, Apple has included the Rosetta emulator in the new OS. This time it was not tightly integrated with the core, as was the case with the transition from 68k to PowerPC. The kernel provided only the ability to run an external recompilator for the application whenever it turned out to be intended for PowerPC. Rosetta translated all application code and library code and provided an interface for interacting with the OS kernel.
The next transition for Apple was to migrate from the 32-bit Intel architecture to its 64-bit version. It occurred on Mac OS X 10.4 in 2006. Although the entire OS could have been ported to 64 bits, as was done in the case of Windows, Apple decided to use an approach more similar to switching to Windows 95: the kernel remained 32-bit, but received support for 64-bit applications. All system applications and drivers remained 32-bit, but some libraries also received 64-bit variants. Thus, the 64-bit application linked to 64-bit libraries and made 64-bit system calls, which were converted from 32-bit inside the kernel.
Mac OS X 10.5 began shipping all the libraries in the 64-bit version, the kernel remained 32-bit. In version 10.6, the kernel was translated to 64 bits, and the use of 64-bit drivers became mandatory.
The Amiga platform used the same OS for the 68k CPU in the Commodore era from 1985 to 1994. However, since 1997, there has been an upgrade to the board with the PowerPC architecture CPU. This operating system did not provide its source code for third-party, so it could not be ported to PowerPC. For this reason, Amiga OS itself continued to work on a dedicated 68k chip, and the extended program loader detected the PowerPC code and transferred it to the second CPU. All system calls were routed through the library layer back to 68k.
AmigaOS 4 (2006) is a true AmigaOS port on PowerPC. This transition required a lot of effort, since a lot of OS source code had to be first converted from BCPL into C language. Application support for 68k is done through binary code emulation and call API forwarding.
Palm switched from 68k to ARM in Palm OS 5 in 2002. The OS was ported to ARM, and the emulator for the 68k Palm Application Compatibility Environment (PACE) was included in its composition for running old applications. However, Palm did not recommend developers to switch to writing applications for ARM and did not even provide an environment for the execution of such applications under OS control. It was argued that most applications for Palm OS were spent most of the time in the (already optimized “native”) code of the operating system, and therefore they would not have received a tangible gain in speed when porting.
However, for applications intensively using the CPU or containing code to compress or encrypt data, Palm has provided a method for executing small pieces of ARM code inside the application for 68k. Similar ARMlets (later called PNOlets, "Palm Native Object") could be called from code for 68k. They provided a minimal data interface (one integer per input and one output). Developers themselves had to take care of transferring large structures in memory and solve problems of the order of bytes (endianness) and their alignment. Procedures from the ARM code could neither call code for 68k, nor make API calls to the operating system.
Binding to the 68k architecture (which is not native to the hardware used) as the basis for most applications actually meant using a virtual architecture of user applications, in the manner of Java or .NET. The transition to ARM was almost imperceptible to third-party developers, and in theory, this approach allowed Palm to re-change the host architecture if necessary in the future at minimal cost.
Let's go over the considered transitions between the OS and the CPU and how different vendors approached the migration task.
The transition to the new mode of the processor is the easiest for the OS, since old applications can continue to be executed directly, and API calls can be translated. At the same time, the OS itself can either continue to use the old mode, converting system calls from new applications to the old format, or switch to the new bit depth and convert calls to old applications. There are also two ways to intercept calls themselves. You can intercept high-level API calls (such as creating a GUI window), but this is difficult to accomplish, since there are many such calls and it is difficult to write the correct converter for so many functions.
Alternatively, the OS can convert low-level system calls. In this case, the intercepted interface is very narrow. But, since old applications are linked with old libraries, and new ones with new libraries, their storage and use consumes twice as much space (memory) in cases where the user executes old and new applications at the same time.
In the case of transition from 16 to 32 bits in Windows, the OS remained 16-bit, and calls were converted from a 32-bit format to a 16-bit API. When Windows NT was translated from 32 to 64 bits, the entire OS became 64-bit, and the old system calls were converted to new ones. The same transition was made differently in the case of Mac OS X: OS remained 32-bit, and 64-bit system calls were broadcast at the kernel level.
The approaches used for Windows NT and Mac OS X are similar, since in both cases 32-bit applications continued to use 32-bit libraries, and 64-bit applications used new 64-bit libraries. The difference was in the mode of the new core. In the case of Windows, there was an advantage in the form of the possibility of using more than 4 GB of memory in kernel mode and some acceleration from using the new 64-bit register set. In the case of Mac OS X, it remains possible to continue to use the old 32-bit drivers without modifications. And yes, the second step Mac OS X later completely switched to the 64-bit kernel and drivers.
It is more difficult to change the used processor, since the old machine code can no longer be executed on the new architecture. Some operating systems were not so easy to migrate.
Mac OS X for Intel and Palm OS for ARM were written in a sufficiently platform-independent manner to be portable to the new architecture. Both included recompilators that executed the old code. This is the simplest approach. Amiga OS could not be ported in this way, as its source code was not available. Therefore, the system included both processors - the original OS and its code was executed on the old processor, and the new processor worked on the new code, sending system calls to the old CPU.
For classic Macintosh (from 68k to PowerPC), the source code was available, but still could not be easily changed. Therefore, the transition was made similar to the situation with the Amiga, but on the same CPU. Most of the OS was executed inside the emulator, new applications were launched directly, causing system calls back from the emulator.
MS-DOS was the implementation of the old OS, made by another company. She did not support the execution of the old binary code. Instead, developers were encouraged to recompile their applications.
The transition to the new OS with the preservation of all users and programs - the most difficult of those considered.
The approach chosen depends on the plans to support the old API from the old OS. If the API was good enough to be worth supporting in the new OS, the new OS should simply support the same API. This was done for transitions from CP / M to DOS and from Windows 9x to Windows NT. In a sense, this was done to switch from classic Mac OS to Mac OS X, but in this case Carbon was not the main API of the OS, but only one of the three supported (Carbon, Cocoa, Java). Virtually everything except Cocoa is currently deprecated.
If the old API is not good enough for a new OS, but it is necessary to support the work of old applications, it makes sense to keep the old OS inside the VM along with the applications for it. This was used in the case of Windows to run DOS applications, and Mac OS X for older Mac OS applications.
If the interface of the old OS is relatively small or detailed repetition of the semantics of calls is optional, emulation of the API may be the best solution: intercept system calls of the old application with mapping them to new ones. This was used to enable DOS applications to work in Windows NT; while the level of compatibility was average.
Unexpectedly, to see how different ways the migration was performed for all the OSs considered, no approach completely repeated the other. The reason for this, perhaps, is that the conditions for the transition in each case were slightly different, and a lot of time was spent on developing the best solution for each specific case.
But there are general patterns. Over time, OSs became more modern and less lurid; in addition, migrations went through many small steps, avoiding big jumps. Modern operating systems, Windows NT and Mac OS X, can be ported to new architectures relatively easily, emulators help execute old applications, and system call translations can be used to display them in their native OS calls. Thanks to the abstractions used by the systems you use, the OS can be ported to a new architecture or bit depth in steps, and a part of it can remain in the old environment, and a part can be transferred to a new one. These same abstractions allow developers to replace entire subsystems and rework parts of the OS without a strong effect on users. These processes are becoming more streamlined and well-developed - and, alas, less exciting.
Unfortunately, some links of the original post lead nowhere, and I could not find the documents.
1. Pat Villani. FreeDOS Kernel: An MS-DOS Emulator for Platform Independence & Embedded System Development
2. Broken reference to Apple’s PowerPC documentation
3. Brian Maas. Palm OS 5 ARM Programming
4. PNOLet Samples
Operating system vendors face this problem once or twice a decade: they need to transfer their user base from the old operating system to their very different new OS, or they need to switch from one CPU architecture to another while maintaining the ability to run old applications without modifications. and also help third-party developers port their applications to the new OS.
Let's look at how this happened in the past 30 years, using the examples of MS DOS / Windows, Apple Macintosh, Amiga OS and Palm OS.
')
From CP / M to PC-DOS / MS-DOS
CP / M is an 8-bit OS from Digital Research that worked on all versions of systems based on the Intel 8080. 86-DOS from Seattle Computer Products, later known as MS-DOS (also called PC-DOS on IBM machines), was a clone of CP / M for later Intel 8086; The situation was similar with CP / M-86 from DR itself (this OS was later called DR-DOS).
Not being binary compatible, the Intel 8086 was “assembly level compatible” from the 8080. This meant that it was easy to convert an 8-bit program written in 8080 or Z80 assembler to an 8086 assembler program, since these two architectures were very similar (backward-compatible memory model, easily mapped register sets), only machine coding of commands differed.
Since MS-DOS used the same ABI conventions and a memory card, this OS was source-level compatible with CP / M. On CP / M systems with access to a maximum of 64 kB of memory, the address range from 0x0000 to 0x0100 (zero page) was reserved for the needs of the OS and contained, among other things, command line arguments. The executable application was located starting at address 0x0100 and above, and the OS was located at the top of the memory; The application stack grew down, starting just below the start of the OS. The 8086 memory model divides the memory into (overlapping) contiguous 64-kbyte segments, so each of these segments is actually sufficient for the 8080 virtual machine. Therefore, the MS-DOS .COM executable files are up to 64 kB in size and loaded into memory starting at 0x100. The range 0x0 - 0x100 in MS-DOS is referred to as Program Segment Prefix and is very similar to the CP / M zero page. Due to the high degree of compatibility of these processors and their ABI program transfer from CP / M to DOS was relatively painless. However, such a port could use a maximum of 64 kbytes of memory (examples - 1982 Wordstar 3.0). But this made it possible to have a single code base for both operating systems with just a few macros and two different assembler programs.
MS-DOS 2.0 brought new, more powerful API conventions (file handles instead of FCB structures, subdirectories, .EXE format) and turned most of the APIs from CP / M into outdated. But DOS was compatible with CP / M up to the latest version.
| From CP / M to PC-DOS / MS-DOS | |
| Change | New CPU New OS code base |
| Execution of new applications | Direct |
| Execution of old applications (without modification) | Not supported |
| Execution of old drivers | Not supported |
| Tools for porting applications | High source and ABI compatibility |
From DOS to Windows
Microsoft Windows was originally designed as a graphical shell on top of MS-DOS. All work with devices and the file system took place through DOS API calls. Therefore, all MS-DOS drivers were executed directly, and Windows could use them. DOS applications could still run after exiting Windows to pure DOS.
In Windows / 386 2.1, the model has been changed. This was already the real OS kernel, running many virtual machines in virtual 8086 (V86) mode, side by side; one was allocated for the MS-DOS OS session, and one was allocated for each Windows application. The DOS virtual machine was used by Windows to access the device and file system drivers, so it was practically a driver compatibility layer that was executed inside the VM. A user could run any number of additional DOS VMs to work with DOS applications; and each contained its own copy of DOS. Windows intercepted calls from the screen's memory, as well as some system calls, and redirected them from the Windows graphics driver or to the first “main” VM DOS.
Windows 3.x started using native Windows drivers, which replaced inward DOS VMs, and also started redirecting some DOS calls to devices inside Windows. The standard installation of Windows 95 did not at all use an appeal to the DOS VM for the needs of working with drivers and the file system, however, if necessary, such a mode could still be used.
DOS was not only a compatibility environment for old drivers and applications, it also provided the Windows command line, so when Windows 95 came up with support for long file names, it was necessary to intercept DOS API calls to provide new functionality for command line utilities.
| From MS-DOS to Windows | |
| Change | New OS |
| Execution of new applications | Direct |
| Execution of old applications | Virtual machine with the old OS |
| Execution of old drivers | Virtual machine with the old OS |
| Tools for porting applications | Not provided |
From DOS to Windows NT
The Windows NT family of systems were not based on DOS; nevertheless, from the very first version they allowed to run MS-DOS applications. As in the case of the non-NT version of Windows, in this case the DOS program worked in the V86 mode of the processor. However, instead of using a full copy of MS-DOS inside the VM, NT placed only application code in it, intercepted all system calls and device calls, and wrapped them in NT API calls. In fact, this method is not a pure VM: the V86 mode was used only to create the memory model needed to support DOS applications.
There is a widespread misconception that the Windows NT command line is “DOS shell”. In fact, the interpreter and auxiliary utilities were native NT applications, and the NTDVM subsystem (the DOS virtual machine) did not start at all until it was really necessary to start the DOS program from the command line.
| From DOS to Windows NT | |
| Change | New OS |
| Execution of new applications | Direct |
| Execution of old applications | Compatible APIs |
| Execution of old drivers | Not supported |
| Tools for porting applications | Not provided |
From Windows 3.1 (Win16) to Windows 95 (Win32)
Beginning with the release of Windows NT 3.1 in 1993, it became clear that it would eventually replace the classic Windows. Despite the fact that NT had the same Win16 user interface and good compatibility with DOS, each newer version usually required a fairly powerful computer to work. Therefore, the migration from Windows to Windows NT was the opposite: each version of Windows became more and more like NT until they became quite similar. By that time, even the weaker computers of that era had become powerful enough to work under NT.
A big step towards bringing Windows and Windows NT closer is support for the native Win32 API for the NT. The first step was a free update of “Win32S” to Windows 3.1, which provided a subset (hence “S” - subset - in the name) of the Win32 API on classic Windows. Win32S expanded the Windows kernel with the ability to create 32-bit address spaces for all 32-bit applications (in NT, a separate space was created for each application). It also made available versions of some libraries from NT (for example, RICHED32.DLL), as well as 32-bit DLLs that accepted calls from low-level Win32API calls (the “GDI” and “USER” subsystems) were automatically translated into calls to the Win16 subsystem (“thunking ").
In Windows 95, this functionality was available by default. She also used isolated address spaces for 32-bit applications, and supported a larger subset of the Win32 API. Some basic applications have become 32-bit (for example, Windows Explorer), but much of the basic system was still 16-bit. With the advent of Windows 95, most developers switched to creating 32-bit applications, which also made it possible to compile them on Windows NT systems.
| From Windows 3.1 (Win16) to Windows 95 (Win32) | |
| Change | New processor mode and addressing |
| Execution of new applications | Compatibility Layers |
| Execution of old applications | Direct |
| Execution of old drivers | Direct |
| Porting methods | High source compatibility |
From Windows 9x to Windows NT
The second step of migration from 16-bit Windows to Windows NT occurred when switching from Windows ME to Windows XP (NT-family) in 2001. Windows NT (2000, XP ...) was a completely 32-bit OS with Win32 API, but also allowed to execute 16-bit Windows applications through the translation of Win16 calls to Win32 (thunking).
The Windows NT 3.1 / 3.5 / 4.0 driver model (Windows NT Driver Model) differed from the classical Windows (VxD) approach, therefore Windows 98 (heir to Windows 95) and Windows 2000 (heir to Windows 4.0) both supported the new “Windows Driver Model”. The same driver could now work in both OSs, but each continued to maintain its own original mode for old drivers.
When Microsoft transferred home users to NT systems, most applications, games, and drivers were made on Windows XP. The only thing that had to be rewritten is system utilities.
| From Windows 9x to Windows NT | |
| Change | New OS |
| Execution of new applications | Direct |
| Execution of old applications | Win16: Broadcast API Win32: direct |
| Execution of old drivers | API support from old OS |
| Application Porting Methods | High source code compatibility API support from old OS |
From Windows i386 (Win32) to Windows x64 / x86_64 (Win64)
The transition from 32-bit Windows to 64-bit is currently taking place. Windows XP was the first Microsoft OS available for Intel 64 / AMD64 architecture, all subsequent systems (Vista, 7, 8, 10) are available in 32 and 64 bit versions. In 64-bit editions, the kernel works in 64-bit mode, like all system libraries and most applications. The 32-bit API is supported through the WOW64 subsystem (Windows-on-Windows 64). A 32-bit application is linked to 32-bit libraries, however low-level API calls are translated via WOW64 into analogues for 64-bit DLLs.
Since the drivers are executed in the same address space as the kernel, it was difficult to ensure the support of 32-bit drivers, so it is not there. Support for DOS and Win16 applications in 64-bit versions of the OS has also been discontinued.
| From Windows i386 (Win32) to Windows x64 / x86_64 (Win64) | |
| Change | New processor mode and addressing |
| Execution of new applications | Straight |
| Execution of old applications | Call Broadcast |
| Execution of old drivers | Not supported |
| Application Porting Methods | High source compatibility |
Apple Macintosh for 68K to Macintosh on PowerPC
Apple transferred its computers from Motorola 68k processors to Motorola / IBM PowerPC from 1994 to 1996. Since the Macintosh operating system, System 7, was mainly written in 68k assembler, it could not be easily converted to PowerPC OS. Instead, most of it was played through a simulation. The new “nano-core” received and processed interrupts and performed minimal memory management, abstracting the PowerPC features, and the integrated 68k emulator executed the old OS, which was modified to bind to the nano-core. Thus, System 7.1.2 for PowerPC was actually paravirtualized and executed on top of a very thin layer of the hypervisor.
The first Mac OS for PowerPC version executed most of the system code inside the emulator for 68k, including drivers. Part of the code that is critical for performance, was rewritten for direct execution. The executable loader could detect programs with PowerPC and could execute them directly. Most of the communications with the OS still went through the emulator. In later versions of Mac OS, an increasing proportion of the code corresponded from 68k to PowerPC.
| Macintosh for 68k on Macintosh for PowerPC | |
| Change | New processor |
| Execution of new applications | Call Broadcast |
| Execution of old applications | Paravirtualized old OS inside the emulator |
| Execution of old drivers | Paravirtualized old OS inside the emulator |
| Application Porting Method | High source compatibility |
From Classic Mac OS to Mac OS X
Just as Microsoft moved users from Windows to Windows NT, Apple switched from classic Mac OS to Mac OS X. While classic Mac OS was a hackish operating system with cooperative multitasking, without memory protection, it still played the part system code inside the 68k emulator, Mac OS X was based on NEXTSTEP, a modern UNIX-like OS with a completely different API.
When Apple decided to migrate, they ported the classic Mac OS system libraries (“Toolbox”), while removing calls that could not be supported on a modern OS, replacing them with alternatives, and called the new API “Carbon”. The same API became available in classic Mac OS 8.1 in 1998, so that developers were able to update their applications to work in Mac OS X, while maintaining compatibility with classic Mac OS. When Mac OS X became publicly available in 2004, applications that were “carbonized” by that time could work on both operating systems without recompiling. This is reminiscent of Microsoft’s approach to bringing Windows NT closer to Windows NT.
But since it was not expected that all applications would be converted into a “carbonized” form by the time Mac OS X appeared, the new OS also contained a virtual machine “Classic” or “Blue Box”, in which unmodified Mac OS 9 was executed and could support arbitrary number of old applications. Requests to the network and file system inside the VM were intercepted and redirected to the host OS, and integration with the window manager allowed the two desktop environments to look almost seamless.
| From Classic Mac OS to Mac OS X | |
| Change | New OS |
| Execution of new applications | Direct |
| Executions of old applications | Classic: VM from old OS Carbon: Common API for both systems |
| Execution of old drivers | The virtual machine for the old OS |
| Application Porting Method | Common API for both systems |
Mac OS X transition from PowerPC to Intel IA-32
In 2005, Apple announced a second transition from one CPU architecture to another, this time from PowerPC to Intel IA-32. Since this OS was mostly written in C and Objective C, it could easily be ported to IA-32. Apple claims that they have always supported the OS version for IA-32, starting from the very first release.
In order to run old applications that have not yet been ported to IA-32, Apple has included the Rosetta emulator in the new OS. This time it was not tightly integrated with the core, as was the case with the transition from 68k to PowerPC. The kernel provided only the ability to run an external recompilator for the application whenever it turned out to be intended for PowerPC. Rosetta translated all application code and library code and provided an interface for interacting with the OS kernel.
| Mac OS X with PowerPC on Intel IA-32 | |
| Change | New CPU Architecture |
| Execution of new applications | Direct |
| Execution of old applications | User Mode Emulator |
| Execution of old drivers | Not supported |
| Application Porting Method | High Source Portability |
Mac OS X 32 bits on Mac OS X 64 bits
The next transition for Apple was to migrate from the 32-bit Intel architecture to its 64-bit version. It occurred on Mac OS X 10.4 in 2006. Although the entire OS could have been ported to 64 bits, as was done in the case of Windows, Apple decided to use an approach more similar to switching to Windows 95: the kernel remained 32-bit, but received support for 64-bit applications. All system applications and drivers remained 32-bit, but some libraries also received 64-bit variants. Thus, the 64-bit application linked to 64-bit libraries and made 64-bit system calls, which were converted from 32-bit inside the kernel.
Mac OS X 10.5 began shipping all the libraries in the 64-bit version, the kernel remained 32-bit. In version 10.6, the kernel was translated to 64 bits, and the use of 64-bit drivers became mandatory.
| Mac OS X 32 bits on Mac OS X 64 bits | |
| Change | New CPU and Addressing Mode |
| Execution of new programs | Call Broadcast |
| Execution of old programs | Direct |
| Execution of old drivers | Direct |
| Application Porting Method | Carbon: Not supported Cocoa: Source Compatibility |
AmigaOS from 68k to PowerPC
The Amiga platform used the same OS for the 68k CPU in the Commodore era from 1985 to 1994. However, since 1997, there has been an upgrade to the board with the PowerPC architecture CPU. This operating system did not provide its source code for third-party, so it could not be ported to PowerPC. For this reason, Amiga OS itself continued to work on a dedicated 68k chip, and the extended program loader detected the PowerPC code and transferred it to the second CPU. All system calls were routed through the library layer back to 68k.
AmigaOS 4 (2006) is a true AmigaOS port on PowerPC. This transition required a lot of effort, since a lot of OS source code had to be first converted from BCPL into C language. Application support for 68k is done through binary code emulation and call API forwarding.
| AmigaOS c 68k on PowerPC (version 3.x) | |
| Change | New CPU |
| Execution of new applications | Broadcast system calls (on the new CPU) |
| Execution of old applications | Direct (on the old CPU) |
| Execution of old drivers | Direct |
| Application Porting Method | High source compatibility |
| AmigaOS from 68k to PowerPC (version 4.0) | |
| Change | New CPU |
| Execution of new applications | Direct |
| Execution of old applications | User level emulator |
| Execution of old drivers | Not supported |
| Application Porting Method | High source compatibility |
Palm OS from 68k to ARM
Palm switched from 68k to ARM in Palm OS 5 in 2002. The OS was ported to ARM, and the emulator for the 68k Palm Application Compatibility Environment (PACE) was included in its composition for running old applications. However, Palm did not recommend developers to switch to writing applications for ARM and did not even provide an environment for the execution of such applications under OS control. It was argued that most applications for Palm OS were spent most of the time in the (already optimized “native”) code of the operating system, and therefore they would not have received a tangible gain in speed when porting.
However, for applications intensively using the CPU or containing code to compress or encrypt data, Palm has provided a method for executing small pieces of ARM code inside the application for 68k. Similar ARMlets (later called PNOlets, "Palm Native Object") could be called from code for 68k. They provided a minimal data interface (one integer per input and one output). Developers themselves had to take care of transferring large structures in memory and solve problems of the order of bytes (endianness) and their alignment. Procedures from the ARM code could neither call code for 68k, nor make API calls to the operating system.
Binding to the 68k architecture (which is not native to the hardware used) as the basis for most applications actually meant using a virtual architecture of user applications, in the manner of Java or .NET. The transition to ARM was almost imperceptible to third-party developers, and in theory, this approach allowed Palm to re-change the host architecture if necessary in the future at minimal cost.
| Palm OS from 68k to ARM | |
| Change | New CPU |
| Execution of new applications | Not supported (PNOlet for procedures) |
| Execution of old applications | User level emulation |
| Execution of old drivers | Not supported |
| Application Porting Method | Not supported (PNOlet for procedures) |
findings
Let's go over the considered transitions between the OS and the CPU and how different vendors approached the migration task.
Digit (width of the address space)
The transition to the new mode of the processor is the easiest for the OS, since old applications can continue to be executed directly, and API calls can be translated. At the same time, the OS itself can either continue to use the old mode, converting system calls from new applications to the old format, or switch to the new bit depth and convert calls to old applications. There are also two ways to intercept calls themselves. You can intercept high-level API calls (such as creating a GUI window), but this is difficult to accomplish, since there are many such calls and it is difficult to write the correct converter for so many functions.
Alternatively, the OS can convert low-level system calls. In this case, the intercepted interface is very narrow. But, since old applications are linked with old libraries, and new ones with new libraries, their storage and use consumes twice as much space (memory) in cases where the user executes old and new applications at the same time.
| New mode of the processor or digit capacity | ||||
| OS | Old mode | New mode | Call Forwarding Direction | Call Pickup Level |
| Windows | 16 bits | 32 bits | new to old | libraries |
| Windows NT | 32 bits | 64 bit | old to new | core |
| Mac os x | 32 bits | 64 bit | new to old | core |
In the case of transition from 16 to 32 bits in Windows, the OS remained 16-bit, and calls were converted from a 32-bit format to a 16-bit API. When Windows NT was translated from 32 to 64 bits, the entire OS became 64-bit, and the old system calls were converted to new ones. The same transition was made differently in the case of Mac OS X: OS remained 32-bit, and 64-bit system calls were broadcast at the kernel level.
The approaches used for Windows NT and Mac OS X are similar, since in both cases 32-bit applications continued to use 32-bit libraries, and 64-bit applications used new 64-bit libraries. The difference was in the mode of the new core. In the case of Windows, there was an advantage in the form of the possibility of using more than 4 GB of memory in kernel mode and some acceleration from using the new 64-bit register set. In the case of Mac OS X, it remains possible to continue to use the old 32-bit drivers without modifications. And yes, the second step Mac OS X later completely switched to the 64-bit kernel and drivers.
CPU
It is more difficult to change the used processor, since the old machine code can no longer be executed on the new architecture. Some operating systems were not so easy to migrate.
| New processor | ||||
| OS | Old CPU | New CPU | Method of launching old applications | Call Pickup Level |
| CP / M, DOS | 8080 / Z80 | 8086 | Recompilation | - |
| Macintosh | 68K | PowerPC | Execution of OS and applications inside emulation | - |
| Mac os x | PowerPC | i386 | User level emulation | core |
| Amiga | 68K | PowerPC | Dual CPU and Call Broadcast | libraries |
| Palm | 68K | ARM | User level emulation | libraries |
Mac OS X for Intel and Palm OS for ARM were written in a sufficiently platform-independent manner to be portable to the new architecture. Both included recompilators that executed the old code. This is the simplest approach. Amiga OS could not be ported in this way, as its source code was not available. Therefore, the system included both processors - the original OS and its code was executed on the old processor, and the new processor worked on the new code, sending system calls to the old CPU.
For classic Macintosh (from 68k to PowerPC), the source code was available, but still could not be easily changed. Therefore, the transition was made similar to the situation with the Amiga, but on the same CPU. Most of the OS was executed inside the emulator, new applications were launched directly, causing system calls back from the emulator.
MS-DOS was the implementation of the old OS, made by another company. She did not support the execution of the old binary code. Instead, developers were encouraged to recompile their applications.
Operating Systems
The transition to the new OS with the preservation of all users and programs - the most difficult of those considered.
| New OS | ||
| Old OS | New OS | Method of executing old applications |
| CP / M | Dos | Compatible API |
| Dos | Windows | Virtual machine with the old OS |
| Dos | Windows NT | Emulation API |
| Windows 9x | Windows NT | Compatible API |
| Mac OS | Mac os x | Classic: a virtual machine with the old OS Carbon: API compatible |
The approach chosen depends on the plans to support the old API from the old OS. If the API was good enough to be worth supporting in the new OS, the new OS should simply support the same API. This was done for transitions from CP / M to DOS and from Windows 9x to Windows NT. In a sense, this was done to switch from classic Mac OS to Mac OS X, but in this case Carbon was not the main API of the OS, but only one of the three supported (Carbon, Cocoa, Java). Virtually everything except Cocoa is currently deprecated.
If the old API is not good enough for a new OS, but it is necessary to support the work of old applications, it makes sense to keep the old OS inside the VM along with the applications for it. This was used in the case of Windows to run DOS applications, and Mac OS X for older Mac OS applications.
If the interface of the old OS is relatively small or detailed repetition of the semantics of calls is optional, emulation of the API may be the best solution: intercept system calls of the old application with mapping them to new ones. This was used to enable DOS applications to work in Windows NT; while the level of compatibility was average.
Conclusion
Unexpectedly, to see how different ways the migration was performed for all the OSs considered, no approach completely repeated the other. The reason for this, perhaps, is that the conditions for the transition in each case were slightly different, and a lot of time was spent on developing the best solution for each specific case.
But there are general patterns. Over time, OSs became more modern and less lurid; in addition, migrations went through many small steps, avoiding big jumps. Modern operating systems, Windows NT and Mac OS X, can be ported to new architectures relatively easily, emulators help execute old applications, and system call translations can be used to display them in their native OS calls. Thanks to the abstractions used by the systems you use, the OS can be ported to a new architecture or bit depth in steps, and a part of it can remain in the old environment, and a part can be transferred to a new one. These same abstractions allow developers to replace entire subsystems and rework parts of the OS without a strong effect on users. These processes are becoming more streamlined and well-developed - and, alas, less exciting.
Literature
Unfortunately, some links of the original post lead nowhere, and I could not find the documents.
1. Pat Villani. FreeDOS Kernel: An MS-DOS Emulator for Platform Independence & Embedded System Development
2. Broken reference to Apple’s PowerPC documentation
3. Brian Maas. Palm OS 5 ARM Programming
4. PNOLet Samples
Source: https://habr.com/ru/post/269935/
All Articles