1 Domain documents

1.1 Problem domains

In the real world, software engineering is not only focused on software, but also the problem domain it is trying to solve.

A problem domain is the part of the world where the computer is to produce effects, together with the means available to produce them, directly or indirectly. (Kovitz 1999)

A problem domain is anything outside of programming that a software engineer needs to understand to produce correct code that can achieve the desired effects. “Directly” includes anything that the software can control to produce the desired effects, e.g. keyboards, printers, monitors, other software, etc. “Indirectly” means anything not part of the software but relevant to the problem domain e.g. appropriate people to be informed by the software when some event happens, students that move to correct classrooms according to the schedule generated by the software. To write a finance application, a software engineer needs to learn sufficient finance concepts to understand the requirements of a customer and implement such requirements, correctly.

Requirements are the effects that the machine is to exert in the problem domain by virtue of its programming.

Programming alone is not too complicated; programming to solve a problem domain is1. Not only does a software engineer need to understand how to implement the software, but also the problem domain that it tries to solve, which might require in-depth expert knowledge. The software engineer must also select the right programming techniques that apply to the problem domain he is trying to solve because many techniques that are effective in one domain might not be in another. For example, many types of applications do not require performant written code, but a short time to market. In this case, interpreted languages are widely popular because they can satisfy such a need. However, for writing huge 3D games or an operating system, compiled languages are dominant because they can generate the most efficient code required for such applications.

Often, it is too much for a software engineer to learn non-trivial domains (that might require a bachelor’s degree or above to understand the domains). Also, it is easier for a domain expert to learn enough programming to break down the problem domain into parts small enough for the software engineers to implement. Sometimes, domain experts implement the software themselves.

Problem domains: Software and Non-software.

One example of such scenario is the domain that is presented in this book: operating system. A certain amount of electrical engineering (EE) knowledge is required to implement an operating system. If a computer science (CS) curriculum does not include minimum EE courses, students in the curriculum have little chance to implement a working operating system. Even if they can implement one, either they need to invest a significant amount of time to study on their own, or they fill code in a predefined framework just to understand high-level algorithms. For that reason, EE students have an easier time implementing an OS, as they only need to study a few core CS courses. In fact, only “C programming” and “Algorithms and Data Structures” classes are usually enough to get them started writing code for device drivers, and later generalize it into an operating system.

Operating System domain.

One thing to note is that software is its own problem domain. A problem domain does not necessarily divide between software and itself. Compilers, 3D graphics, games, cryptography, artificial intelligence, etc., are parts of software engineering domains (actually it is more of a computer science domain than a software engineering domain). In general, a software-exclusive domain creates software to be used by other software. Operating System is also a domain, but is overlapped with other domains such as electrical engineering. To effectively implement an operating system, it is required to learn enough of the external domain. How much learning is enough for a software engineer? At the minimum, a software engineer should be knowledgeable enough to understand the documents prepared by hardware engineers for using (i.e. programming) their devices.

Learning a programming language, even C or Assembly, does not mean a software engineer can automatically be good at hardware programming or any related low-level programming domains. One can spend 10 years, 20 years or his entire life writing C/C++ code, and he still cannot write an operating system, simply because of the ignorance of relevant domain knowledge. Just like learning English does not mean a person automatically becomes good at reading Math books written in English. Much more than that is needed. Knowing one or two programming languages is not enough. If a programmer writes software for a living, he had better be specialized in one or two problem domains outside of software if he does not want his job taken by domain experts who learn programming in their spare time.

1.2 Documents for implementing a problem domain

Documents are essential for learning a problem domain (and actually, anything) since information can be passed down in a reliable way. It is evident that written text has been used for thousands of years to pass knowledge from generation to generation. Documents are integral parts of non-trivial projects. Without the documents:

There are many types of documents. However, to facilitate the understanding of a problem domain, these two documents need to be written: software requirement document and software specification.

1.2.1 Software Requirement Document

Software requirement document includes both a list of requirements and a description of the problem domain (Kovitz 1999).

Software solves a business problem. But which problems to solve is requested by a customer. Many of these requests make a list of requirements that our software needs to fulfill. However, an enumerated list of features is seldom useful in delivering software. As stated in the previous section, the tricky part is not programming alone but programming according to a problem domain. The bulk of software design and implementation depends upon the knowledge of the problem domain. The better understood the domain, the higher quality software can be. For example, building a house has been practiced for thousands of years and is well understood, and it is easy to build a high-quality house; software is no different. Code that is difficult to understand is usually due to the author’s ignorance of a problem domain. In the context of this book, we seek to understand the low-level working of various hardware devices.

Because software quality depends upon the understanding of the problem domain, the bulk of a software requirement document should consist of problem domain description.

Be aware that software requirements are not:

What vs How

“what” and “how” are vague terms. What is the “what”? Is it nouns only? If so, what if a customer requires his software to perform specific steps of operations, such as purchasing procedure for a customer on a website. Does it include “verbs” now? However, isn’t the “how” supposed to be step by step operations? Anything can be the “what” and anything can be the “how”.

Sketches

A software requirement document is all about the problem domain. It should not be a high-level description of an implementation. Some problems might seem straightforward to map directly from its domain description to the structure of an implementation. For example:

In the future, instead of a drop-down menu, all books are listed directly on a page in thumbnails. Books might be reimplemented as a graph, and each node is a book for finding related books, as a recommender is going to be added in the next version. The requirement document needs updating again to remove all the outdated implementation details, thus requiring additional effort to maintain the requirement document, and when the effort for syncing with the implementation is too much, the developers give up documentation, and everyone starts ranting how useless documentation is.

More often than not there is no straightforward one-to-one mapping. For example, a regular computer user expects an OS to be something that runs some program with GUI, or their favorite computer games. But for such requirements, an operating system is implemented as multiple layers, each hiding the details from the upper layers. To implement an operating system, a large body of knowledge from multiple fields is required, especially if the operating system runs on non-PC devices.

It’s best to include information related to the problem domain in the requirement document. A good way to test the quality of a requirement document is to provide it to a domain expert for proofreading, to ensure he can understand the material thoroughly. A requirement document is also useful as a help document later, or makes writing one much easier.

1.2.2 Software Specification

Software specification document states rules relating desired behavior of the output devices to all possible behavior of the input devices, as well as any rules that other parts of the problem domain must obey (Kovitz 1999).

Simply put, software specification is interface design, with constraints for the problem domain to follow e.g. the software accepts certain types of input only, such as English but no other language. For a hardware device, a specification is always needed, as software depends on its hardwired behaviors. And in fact, it is mostly the case that hardware specifications are well-defined, with the tiniest details in it. It needs to be that way because once hardware is physically manufactured, there’s no going back, and if defects exist, it is devastating damage to the company, both financially and in reputation.

Note that, similar to a requirement document, a specification only concerns interface design. If implementation details leak in, it is a burden to sync between the actual implementation and the specification, and it is soon abandoned.

Another important remark is that, though a specification document is important, it does not have to be produced before the implementation. It can be prepared in any order: before or after a complete implementation; or at the same time with the implementation, when some part is done, and the interface is ready to be recorded in the specification. Regardless of the method, what matters is a complete specification at the end.

1.3 Documents for writing an x86 Operating System

When the problem domain is different from the software domain, the requirement document and the specification are usually separate. However, if the problem domain is inside software, a specification most often includes both, and the contents of both can be mixed with each other. As the previous sections demonstrated the importance of documents, to implement an OS we will need to collect relevant documents to gain sufficient domain knowledge. These documents are as follows:

All of them are available free of charge: the Intel documents from Intel’s website, the AMD manual from AMD’s, and the System V ABI from its maintainers. Search for them by title rather than relying on a link2. The AMD manual describes the same architecture from the side of the other vendor; chapter 3, Computer Architecture, explains why this book cites Intel and when to open AMD’s instead.

Intel documents divide the requirement and specification sections clearly, but call the sections with different names. The section corresponding to the requirement document is called “Functional Description”, which consists mostly of domain description; for specification, “Register Description” section describes all programming interfaces. Both documents carry no unnecessary implementation details3. Intel documents are also great examples of how to write requirements and specifications well, as explained in this chapter.

Other than the Intel documents, other documents will be introduced in the relevant chapters.

1.4 Exercises

Exercise 1.1. Download Intel SDM Volume 3A and open its table of contents. Find the chapter on paging and the chapter on interrupt and exception handling, and write down their numbers. Compare them with the numbers this book uses (chapter 3 gives the edition the book follows). If they differ, your copy is another edition: this is why the book tells you to search by section title and not by number, and you have just done it for the first time.

Exercise 1.2. Open the Intel 3 Series Express Chipset Family Datasheet and find the sections named Functional Description and Register Description. Pick one device, for example the DRAM controller, and read one page of each section about it. Mark in the margin which sentences describe the problem domain (what the device does and why) and which describe the interface (which bit of which register does what). You have just told a requirement document from a specification; check your marks against the next section of this chapter.

Exercise 1.3. Find the chapter of the System V ABI that describes the ELF file format. It is a specification: it states rules about the bytes of the file, and says nothing about how a linker or a loader must be written. Find one sentence that is nevertheless a description of the problem domain (why the format exists, what a loader needs) and one sentence that is a pure interface rule. Chapter 5 uses this document on every page.

Exercise 1.4. Write, in half a page, the requirement document for the smallest program of Part III: a kernel that prints “Hello World” on the serial port of a QEMU machine. List what belongs to the problem domain (the serial port, its speed, QEMU, what the reader sees) and leave out everything about the implementation. Then write the specification: what the program must send, where, in which order. Keep both pages. In chapter 10, when you program the serial port, compare them with what you actually needed to know.

Exercise 1.5. Linux keeps the documentation of its configuration files in section 5 of the manual. Read man 5 fstab and decide whether it is a requirement document, a specification, or both. Quote the sentences that made you decide. Then do the same with the datasheet page of exercise 1.2, and explain which of the two documents mixes the two kinds more, and why that is a problem for the reader.

1.5 Check your understanding

  1. Why does the book claim that knowing C perfectly is not enough to write an operating system? What exactly is missing?

  2. What is the difference between a software requirement document and a software specification? Give a way to tell, from a single paragraph, which one you are reading.

  3. Why should a requirement document not mention a drop-down menu or a linked list? What goes wrong over time if it does?

  4. Why are hardware specifications usually more complete and precise than software ones?

  5. In an Intel datasheet, which section plays the role of the requirement document and which that of the specification? Why does the distinction matter when you want to program a device?

  6. Why does the book say that a specification need not be written before the implementation? What must be true at the end?

  7. The book says that software is its own problem domain. What does that mean for an operating system, which overlaps software and electrical engineering?

  1. We refer to the concept of “programming” here as someone able to write code in a language, but not necessary know any or all software engineering knowledge.↩︎

  2. Intel may change the links to the documents as they update their website, so this book doesn’t contain any link to the documents to avoid confusion for readers.↩︎

  3. As it should be, those details are trade secret.↩︎