Building and Maintenance Tools


1. The Startup Process

An aspect usually given little attention in system descriptions is the process of how a system is started. Its choice, however, is itself an interesting and far from trivial design consideration and will be described here in some detail. Moreover, it directly determines the steps in which a system is developed from scratch, mirroring the steps in which it builds itself up from a bare store to an operating body.

The startup process typically proceeds in several stages, each of them bringing further facilities into play, raising the system to a higher level towards completion. The term for this strategy is bootstrapping or, in modern computer jargon, booting.

Stage 1 is initiated when power is switched on or when the reset button is pressed and released. To be precise, power-on issues a reset signal to all parts of the computer and holds it for a certain time. Pushing the reset button therefore appears like a power-on without power having been switched off. Release of the reset signal causes the processor to start the so-called boot loader. This program resides in a read-only store (ROM) and hence is always present, even on a "bare" machine. It loads data into memory, which are interpreted as code in Stage 2. In order to keep the boot loader as simple as possible - remember that it is burnt into ROM on every workstation and therefore cannot be changed without considerable effort - the format of its data must be simple. We have chosen the following structure, which had never to be changed during the entire development effort of the Oberon System because of both its simplicity and generality:
BootFile = {block}. block = size:4 address:4 {byte}.

The address of the last block, distinguished by size = 0, is interpreted as the address of the starting point of Stage 2. The size of the boot loader on Ceres is about 250 bytes.

The source of the boot data is typically a fixed location on the disk. In our case, the data occupy less than a single track, which is therefore called the boot track and remains permanently reserved.

The data loaded in Stage 1 are interpreted as code in Stage 2, representing modules Kernel, FileDir, Files, and Modules, which are said to constitute the inner core of the system. Control is transferred to module Kernel's initialization part. First, the processor's base address registers are initialized, then the chain of free module descriptors is formed. This is followed by the initialization of the required virtual address pages on machines which feature a memory management unit. Interrupt table and interrupt control unit (ICU) are initialized, and finally the initialization parts of the remaining three modules are executed, the last one being Modules. Note that only a single device driver is contained in the inner core: the disk driver in module Kernel.

The presence of Modules implies that the regular loader is available for the further build-up of the system. Module Files is present in the inner core, because it is imported by Modules, and FileDir because it is imported by Files. The initialization of module FileDir constructs the sector reservation table by recording all files registered in the file directory. This process requires the traversal of the entire directory and the reading of all file headers. It can be regarded as the garbage collection process of disk sectors. The initialization part of Modules contains the statement
M := ThisMod("Oberon")
which causes loading of module Oberon and with it automatically of all modules imported by Oberon. This constitutes Stage 3 of the boot process. In particular, Stage 3 loads and initializes the display, keyboard and mouse drivers, as well as the display, text, and font machinery. It terminates with the initialization of module Oberon itself, which contains the statement
M := Modules.ThisMod("System")

It initiates Stage 4 of the boot process and completes the system by loading the first tool module. The initialization of System opens a viewer for the log and one for a tool text. In addition to System, modules MenuViewers and TextFrames are loaded, because they are imported by System. The modules loaded so far form the outer core. In passing we note that also the default font and the text System.Tool are needed in Stage 4. After completion of Stage 4, control returns to the statements
P := ThisCommand(M, "Loop"); P
in the initialization part of Modules. Thereby control enters the central loop of the Oberon System for polling input events. Normal operation has begun.
Let us summarize the prerequisites for the four stages:
0. The bootloader must reside in the ROM 1. The boot file must reside on the boot track of the disk. 2. The modules of the outer core must reside in the file system.
These conditions are usually met. But they are not satisfied, if either a new, bare machine is present, or if the disk store is defective. In these cases, the prerequisites must be established with, of course, the aid of suitable tools. The tools needed for the case of the bare machine or the incomplete file store are called building tools, those required in the case of defects are called maintenance tools.

2. Building Tools

Building Tools allow to establish the three preconditions for the boot process on a bare machine. Condition 0 requires an assembler for programming the boot loader, and a so-called ROM-programmer, typically an external device connected by an RS-232 (V24) link. We will not discuss these tools any further. Condition 1 requires a tool to compose the boot file, and one to load it onto the boot track. Condition 2 requires a tool which establishes a file directory and is able to load files. The tool which creates the boot file is called a boot linker; this module was given the name Boot. The tool which has the capability to load the boot track and to load files is module Oberon0.

There remains the important question of how Oberon0 is loaded onto a bare machine. A partial answer is: by a boot process consisting of Stages 1 and 2 only, using a boot file in which module Modules is replaced by Oberon0. But this does not suffice. The key facility is a boot loader that admits an external source as alternative to the disk's boot track. As alternative source we use (the boot track of) the diskette. The selection of the source is determined by a switch. It makes the use of I/O devices apart from the diskette driver during the boot phase avoidable. The initiation rite for a bare machine then consists of the following steps:

1. Select the alternative boot source by the appropriate switch setting.
2. Reset. The boot file is read and Oberon0 is started.
3. Invoke the command which reads all files from the diskette, (which supposedly holds all files needed for the outer core).
4. Reset the switch and boot again. This initiates the regular boot process.

A more modern solution would be to select the network as alternate boot file source. We rejected this option in order to keep net access routines outside the ROM, in order to keep the startup of a computer independent of the presence of a server, and also in consideration of the fact that there exist machines which operate in a stand-alone mode. As it turns out, the need for the alternative boot file source arises very rarely.

The boot linker is almost identical to the module loader, with the exception that object code is not deposited in newly allocated blocks, but in a fixed buffer which is finally output to form the boot file. Its name is supplied as second parameter to the command Boot.Link. The first parameter specifies the module which is the top of the hierarchy forming the inner core
Boot.Link Modules Ceres2.Boot Boot.Link Oberon0 Ceres2.Boot0

The boot file consists of four blocks. The first block contains the module descriptors for the modules of the inner core. The second block consists of their code and global data sections. The third block contains the lengths of the first two blocks. The fourth block specifies the start address of Kernel. The load addresses of the first three blocks are fixed constants in the Boot program (4000H, 8000H, 0).
From the description of the start-up process for a bare machine given above, module Oberon0 must first initialize the file directory and then load all files contained on a boot diskette. We have chosen to extend Oberon0 into a much more versatile tool. This was not merely clever foresight, but is due to Oberon0's emergence from the development process of the Oberon System which, naturally, included a considerable amount of error detection and correction [1]. Oberon0 therefore contains a genuine command interpreter. There exist commands for inspecting memory areas, disk sectors, for inspecting the file directory, and even for loading modules. The presence of a command interpreter requires input (keyboard) and output (display) facilities. They were kept to a bare minimum and encapsulated in module IO.

DEFINITION IO;
PROCEDURE Read (VAR ch: CHAR);
PROCEDURE WriteLn;
PROCEDURE Write (ch: CHAR);
PROCEDURE WriteString (s: ARRAY OF CHAR);
PROCEDURE WriteHex (x: LONGINT);
PROCEDURE WriteInt (x, n: LONGINT);
END IO.

The command interpreter is a simple loop, accepting commands specified by a single letter followed by parameters which are either hexadecimal numbers or names. User-friedliness was not attributed any importance at this point, and it would indeed be unjustified. We refrain from elaborating on further details and concentrate on providing a list of available commands. This should give the reader an impression of the capabilities of this tool module for system initiation and for error searching.

r name read file from diskette
w name write file to diskette
d name delete file from diskette
z read all files contained on diskette
e ennumerate diskette files
!i initialize diskette directory
!f format diskette
E enumerate file directory
D name delete file
N name0 name1 rename file
M name load and initialize module
C name call command
O load Oberon and call Loop
a address display memory block (256 bytes) in hex
A address display memory block (256 bytes) as characters
k number display disk sector in hex
K number display disk sector as characters
l clear display
? list available commands
t get time and date from real time clock
T time date set real time clock
!B name load file onto boot track
!Y initialize bad sector file
!1 initialize file directory

All these additional commands give Oberon0 the character of a maintenance tool. In particular, the possibility to read individual files from the diskette allows recovery when a file required in stages 3 or 4 of the boot process has been corrupted. The O-command allows to detect at which point the process fails. Nevertheless, the original purpose of Oberon0 is to initiate a bare machine through commands !1, !B, and z.

3. Maintenance Tools

An important prerequisite for Stage 2 (and the following stages) in the boot process has not been mentioned above. Recall that the initialization of module FileDir constructs the disk sector reservation table in the Kernel from information contained on the disk. Obviously, its prerequisite is an intact, consistent file directory. A single unreadable, corrupted file directory or file header sector lets this process fail, and booting becomes impossible. To cope with this (fortunately rare) situation, a maintenance tool has been designed: module DiskCheck. It is included in a special boot file generated by the command
Boot.Link DiskCheck Ceres2.CheckBoot

DiskCheck is organized similarly to Oberon0 as a simple command interpreter, but it imports only Kernel and IO. Hence, booting involves only Stages 1 and 2 without any access to the disk. Operating DiskCheck requires care and knowledge of the structure of the file system. The available commands are the following; those that write on the disk are guarded by an exclamation mark.

r n read sector n into sector buffer, display hex
R n read sector n into sector buffer, display as characters
e n read extended index sector n into sector buffer
!W write sector buffer d n read directory sector n into dir buffer
!D write dir buffer
h n read header sector into Header buffer
!H write header buffer
x n read track containing sector n into track buffer
!Y format track
!Z write track buffer
S m, n insert sector n as header address of entry m in dir buffer
s m, n insert sector n in entry m of sector table in header buffer
L n set file length in header buffer to n
i adr val insert val at adr in sector buffer
f name read header sector of named file into header buffer
q n find all files of which sector n is part
c check consistency of files and directory
b n add faulty sector to bad sector table
l clear display
Q reset disk drive
? list available commands_
!0 n clear sector n (write zeroes)
!1 initialize file directory
!2 initialize bad sector table
!? read all disk sectors and record faulty sectors
!* format disk

Sectors are always read into a buffer, namely the sector buffer, the directory buffer, the header buffer, or the track buffer, and the number of the sector in the respective buffer is retained. Changes are made on the data stored in the respective buffer, which is displayed after each reading or change in an appropriately decoded format. Every change must be confirmed, because only a writing command transfers the buffered data to the disk.

The typical handling of the occurence of a corrupted sector in a file consists of the following steps:

1. The track containing the unreadable sector is read (x).
2. The track is reformatted (!Y).
3. The track is restored (!Z). (the faulty sector data are lost)
4. The track is reread (x) and the steps are repeated, if the condition persists.
5. If, after several tries, the sector cannot be corrected, the file must be removed.

In order to make a directory traversal possible without changing the directory data, the corresponding entry is changed (S). The simplest way is to set it equal to its neighbouring entry, thereby introducing an inconsistency (double reference) which must be corrected by deleting the file as soon as the Oberon System is operational.

6. An unrecoverably faulty sector must be made unreferenceable. This is done by appending it to a file called BadSectors, which is inherently unreadable, but which lets its sectors be marked as used in the initialization process of the sector reservation table in Boot Stage 2. A sector is appended to this file by command b.

7. When the Oberon System is available again, the recovered files must be either deleted or inspected and, if possible, corrected.
Program DiskCheck must be extremely robust. No data read can be assumed to be correct, no index can be assumed to lie within its declared bounds, no sector number can be assumed to be valid, and no directory or header page may be assumed to have the expected format. Guards and error diagnostics take a prominent place. Because any program failure must be prevented, no use is made of disk procedures provided by the Kernel. They are reprogrammed with additional guards and status reporting.

Whereas a faulty, unreadable sector in a file in the worst case leads to the loss of that file, a fault in a sector carrying a directory page is quite disastrous. Not only because the files referenced from that page, but also those referenced from descendant pages become inaccessible. A fault in the root page even causes the loss of all files. The catastrophe is of such proportions, that measures should be taken even if the case is very unlikely. After all, it may happen, and it indeed has occurred.

The only way to recover files that are no longer accessible from the directory is by scanning the entire disk. In order to make a search at all possible, every file header carries a mark field that is given a fixed, constant value. It is very unlikely, but not entirely impossible, that data sectors which happen to have the same value at the location corresponding to that of the mark, may be mistaken to be headers.

The tool performing such a scan is called Scavenger. It is, like DiskCheck a simple command interpreter, and a boot file is created by
Boot.Link Scavenger Ceres2.ScavBoot

The available commands are

s n Scan the first n sectors and collect headers
d Display names of collected files
W Build new directory
l Clear display
? Show available commands

During the scan, a new directory is gradually built up in primary store. Sectors marked as headers are recorded by their name and creation date. The scavenger is the reason for recording the file name in the header, although it remains unused there by the Oberon System. Recovery of the date is essential, because several files with the same name may be found. If one is found with a newer creation date, the older entry is overwritten.

Command W transfers the new directory to the disk. For this purpose, it is necessary to have free sectors available. These have been collected during the scan: both old directory sectors (identified by a directory mark similar to the header mark) and overwritten headers are used as free locations.

The scavenger has proven its worth on more than one occasion. Its main drawback is that it may rediscover files that had been deleted. The deletion operation by definition affects only the directory, but not the file. Therefore, the header carrying the name remains unchanged and is discovered by the scan. All in all, however, it is a small deficiency.