Here’s a summary of the basics, for anyone who doesn’t need all the grueling details:

This all matters for CBL because the build process for some of the necessary packages will result in programs that can’t find their shared libraries unless we use an RPATH or RUNPATH; and the build process for other packages sets an incorrect RPATH or RUNPATH that can cause problems unless we remove it.

Usually, ld isn’t executed directly by build processes, but is instead invoked by the gcc driver program. To get gcc to pass an option along to ld, you can give it the option -Wl, which should be followed by additional words separated by commas; gcc will replace the commas with spaces and pass the resulting arguments on to ld. So to set an RPATH of /some/dir and /another/dir, you can give gcc the argument -Wl,-rpath,/some/dir:/another/dir.

An alternative that might work is to set the environment variable LD_RUN_PATH to the desired RPATH before linking — the ld documentation suggests this will work, but I haven’t tried it.

This might make your eyes glaze over a bit.

A lot of this discussion pertains only to programs in the Executable and Linkable Format, commonly abbreviated as ELF. This is the only program format used in modern GNU/Linux systems, but you should be aware that there are other executable program formats, like a.out and COFF, and a lot of the details here don’t apply to those formats.

ELF programs consist of an ELF header followed by some number of segments of various types. If you’d like to examine the full structure of an ELF program, you can use the program readelf, which is part of the GNU binutils package.

When a program is executed, the Linux kernel looks in its .interp section to find the path for its interpreter — in our case, this will be the full path of ld.so. That interpreter then looks in dynamic segments for the names of shared library files that are needed by the program and tries to find them. If any shared library can’t be found, the dynamic linker will terminate with an error.

There’s a bit of additional complexity: shared library files are also in ELF format and can also declare that they are dependent on other shared libraries. This can result in a chain of library dependencies, potentially a lengthy one! When the dynamic linker is trying to find a library, its behavior is partly determined by which object is having its references resolved: the original program, or one of the libraries it depends on, or one of their libraries, and so on. Whichever ELF file is having its references resolved is called the "loading object" here.

The rest of this section describes the procedure used by the dynamic linker to locate shared library files.

Shared library names can contain slash characters, although they usually do not. (I don’t even know how to get ld to create a program with library dependencies that specify a full path.) If a shared library name does contain any slash characters, it is treated as a relative or absolute path and the dynamic linker will only look for the library at that path.

In the common case, when a shared library is specified without any slash characters, the dynamic linker looks for it in a variety of locations. The algorithm it uses is poorly documented and there’s a lot of contradictory information about it on the web. After looking around for quite a while, I found a helpful blog post on qt.io that asserted that the relevant code is _dl_map_object in the glibc file elf/dl-load.c, and a review of that code revealed the following:

As soon as the shared library is found in any of those locations, that’s the version that will be used. If it’s not found in any of those locations, the dynamic linker will give up and crash with an error message. And, in case this was not obvious, in all the things that look like PATH — LD_LIBRARY_PATH, RPATH, RUNPATH — you can specify any number of directories separated by colons, just like the PATH environment variable.

A historical note: RPATH has been around longer than RUNPATH. RUNPATH was implemented, along with all the logic about skipping RPATH when a RUNPATH is present, because people realized that when someone specifies an LD_LIBRARY_PATH it’s usually because they really do know what they want the dynamic linker to do, and it’s rude to override that desire at program compilation time.

When you give ld the -rpath option by itself, it just creates an RPATH in the resulting program or library. When you add the option --enable-new-dtags, it still creates an RPATH (in case you run the program with an old dynamic linker that doesn’t understand RUNPATH), but it also creates a RUNPATH so that modern dynamic linkers will ignore the RPATH.

As a reward for reading this far, here’s one last option you can use if none of the above suits your purpose: before it does anything else, the dynamic linker loads any .o object files, or .so or .a library files, that are named in the environment variable LD_PRELOAD or in the /etc/ld.so.preload file. Any functions defined in those files are used in preference to any other function definitions found later in the process. That lets you override individual function definitions, if you want to override some part of the program without replacing an entire library with a different version.

You can read more about all this stuff in the ld and ld.so info and man pages, and in the "Program Library HOWTO" in the Linux Documentation Project.

The GNU binary utilities package contains a plethora of programs and libraries that can be used to produce, manipulate, and otherwise operate on (compiled and assembled) object files. I’ll briefly describe them all here, but don’t worry! Not only will there not be a test on any of this, you won’t usually need to invoke any of these programs manually. The gcc driver program will invoke as and ld as necessary to do its work, and several of the other utility programs are similary used internally during the build process of other system components but you won’t need to use them yourself.

The most important binutils programs are as, the assembler, which transforms assembly source code (.s files) into binary or "object" code (.o files); and ld, the loader or link editor (ld is usually just called the linker, but it’s hard to see why it’s called ld without knowing the other terms), which combines multiple object files into an executable program.

There are actually two different ld programs in current GNU binutils: the original version uses the binary file descriptor (BFD) library and can always be found at ld.bfd. There’s also a newer program, "gold," that doesn’t use BFD and only works for binaries that are in the "Executable and Linkable Format," aka "ELF"; it’s always available at ld.gold. A hard link to one or the other of those programs is created by the installation process so it can be executed simply as ld.

The other programs in this package are:

There are a couple of shared libraries used by those programs and available to others, as well: libbfd and libopcodes. All of the utility programs are documented much more thoroughly in man pages and the binutils info file.

As I mentioned earlier, you don’t need to run any of those programs directly because the gcc driver program streamlines the simple case of building executables — to compile a "Hello, World" program, you just need to run gcc hello.c, and let the gcc driver program run as to assemble compiled source into object files, ld to link multiple object files together to produce an executable, and maybe other programs if it needs to for some reason.

The downside of using a driver program is that it can make complex builds (when, for example, specific options need to be passed to the assembler, compiler, and linker) a lot more complicated and fussy — as you’ll see, from time to time, during the CBL process.

The combined-source-repository situation described a moment ago also complicates the construction of branch-update patches a little bit: we always produce branch-update patches by running git diff commands against the source repository, and sometimes the resulting patch for binutils includes changes to files that are not part of the binutils distribution — which means they don’t apply cleanly. We adjust for this by applying the patch, fixing any issues we find as a result, and then using the corrected patch here.

Binutils, like some other parts of the toolchain, should be built in a separate directory from the source code. As with other components that we build several times, CBL puts each distinct build in a separate location so that each build is entirely independent of, and unaffected by, the other builds.

We build a cross-binutils using the "sysroot" framework (which you’ll read more about shortly). That framework isn’t particularly well-documented, but the important thing at this point is that we need to specify the configure options --with-sysroot and --with-build-sysroot to inform the build machinery of the desired sysroot directory.

All of the toolchain components should be installed in the same filesystem location. The CROSSTOOLS parameter lets you specify that location — in this case, /tmp/cblwork/cross-tools.

Some of the warning messages present in GCC 8 and later present problems when compiling the binutils. We can tweak the generated Makefiles so those warnings won’t be converted to errors.

GMP, the GNU Multi-Precision arithmetic library, is a component in — or perhaps it is more properly called a dependency of — the GNU toolchain. It allows arithmetic operations to be performed with levels of precision other than the standard integer and floating-point types. Applications can use GMP to provide arithmetic with thousands or millions of digits of precision if that’s what they need. GMP also provides support for rational-number arithmetic, as well as integer and floating-point.

If you’re really interested in high-precision floating-point arithmetic, you might want to look into MPFR rather than GMP! The GMP people say it’s much more complete.

GMP has been needed by the Fortran GCC front-end for some time, but starting with release 4.3.0 of GCC it is needed for C (and C++) as well.

MPC, another dependency of GCC, requires a GMP built with C++ support, so we need to specify that at configure time.

This package is often built in-tree as part of GCC, rather than separately — that’s especially true when the only reason you’re buiilding GMP is because GCC requires it. When using an in-tree build, this blueprint is pretty much irrelevant. However, as of 2015-09-27, there’s an issue with in-tree builds of GMP in some cross-toolchain builds, so for CBL we build it separately.

This is not necessary, because the system or host-prerequisites version of GMP can be used just fine by the cross-compiler. If things break down here and you haven’t built the Trustworthy Host-System Programs, you should probably do that.

MPFR is a library for arbitrary-precision floating-point arithmetic. It stands for "Multiple-Precision Floating-point Rounding," I think, but that’s not really clear from their site so I might be wrong. It uses GMP internally, but provides any level of precision (including very small precision) and provides the four rounding modes from the IEEE 754-1985 standard.

Like GMP, MPFR is a component in, or dependency of, the GNU toolchain. It has been needed by the Fortran GCC front-end for some time, but starting with release 4.3.0 of GCC, MPFR is needed for C and C++ as well. GCC uses MPFR to pre-calculate the result of some mathematic functions when those functions have constant arguments, and produces the same results regardless of the math library or floating point engine used on the runtime system. This occurs in what is called the GCC "middle-end," which is kind of a silly name since it’s not an end.

This package is often built in-tree as part of GCC, rather than separately. However, as of September 2015, in-tree builds of the dependencies have some issues in certain circumstances, so in CBL we step away from the in-tree build facility altogether.

As with GMP, this is not necessary because the system or host-prerequisites version of MPFR can be used.

MPC is a C library for arbitrary-precision arithmetic on complex numbers providing correct rounding. It can be thought of an extension to the MPFR library.

Like GMP and MPFR, MPC is a component in, or dependency of, the GNU toolchain. I haven’t been able to find any description of what features of MPC are actually needed by the GNU toolchain, but MPC is a hard build-time dependency of GCC.

If your system already has MPC installed, this step can be skipped.

Like the other GCC library dependencies, this package is often built in-tree as part of GCC. As mentioned earlier, though, this sometimes introduces problems, so CBL doesn’t take advantage of the in-tree build machinery provided by GCC.

As with GMP, this is not necessary if there is a system or host-prerequisites version of MPC available.

The project homepage describes ISL as a library for manipulating sets and relations of integer points bounded by linear constraints. I’m not sure what that means. It sounds like math.

Unless you have some specific reason to want ISL for one of your own projects, the main benefit it provides is as an optional dependency of GCC: the "graphite loop optimizations" in GCC (whatever those are) require ISL to be available.

As with GMP, this is not necessary if there is a system or host-prerequisites version of ISL available.

The kernel is Linux per se — the foundation of the operating system. Often, when people say "Linux," they mean the entire operating system that lets them use a computer; properly, though, Linux is just the kernel. Many — perhaps most — of the other components that make up the full operating system are pieces of the Free Software Foundation’s GNU project, which stands for "GNU’s Not UNIX"; almost all of the program-construction tools that form the foundation of the system are GNU components.

The job of the kernel — this is a critically important point and I repeat it a lot — is to initialize and manage all of the hardware on the computer (including CPUs, memory, and I/O devices) and provide services to userspace processes. That’s a big job, but it’s also a limited one! Everything you do with your computer is the responsibility of userspace processes.

The way that Linux performs its job — more generally, the way that UNIX kernels work — is conceptually very simple: when it is executed on a computer, Linux does some hardware initialization, mounts the root filesystem, and then starts a single userspace process. That process has process ID (PID) 1, and is conventionally called init.

The init process is responsible for starting all other userspace programs and getting the machine into a usable state; after the kernel starts init, it gets out of the way and waits for that process, or for other userspace programs, to request its services.

The complexity in the Linux kernel emerges mostly from the huge variety of hardware that it supports and the many layers of functionality that can be built into it. If you unpack the Linux source code, you’ll find that the whole thing adds up to about (as of the 6.6 kernel) 1477 megabytes in total. Of that, almost ten percent (144 megabytes) is specific to the twenty-six different CPU architectures that Linux supports, and about two-thirds (971 megabytes) is device drivers. That means that for any given system, a large majority of the code that makes up the Linux kernel won’t ever be used.

In CBL, the kernel is kept as pristine as possible. In most cases, the only patches we apply to it are intended to add new default configuration settings — which is more convenient than setting dozens of configuration options manually.

The way that programs make requests of the kernel is by invoking kernel functions known as "system calls." The Linux kernel sources include header files that define all of the system calls it makes available to userspace programs.

Userspace programs don’t usually invoke system calls themselves (although they can, and some do). Instead, they invoke library functions — particularly the ones defined in the C standard library, which on GNU/Linux systems is usually the GNU libc, glibc, but might be musl or uClibc or something else altogether. Those library functions invoke system calls as needed to accomplish their work.

In this step, we’re not actually building the Linux kernel; all we’re doing is installing the header files that specify those system calls. These header files are then used by the C library and other userspace libraries and programs to invoke system calls as needed.

The Linux source tree also includes a number of other, private, header files — these define data structures and functions that should be used only within the kernel itself, and are not intended to be visible to userspace programs. The Makefile target we use here, headers_install, installs only the public header files, not these private internal headers.

The makefile target mrproper puts the source tree into a completely pristine state. The name is a reference to the Proctor & Gamble cleaning product that people in the United States know as "Mr. Clean"; in many parts of Europe, it is marketed under the brand name "Mr. Proper."

The install_headers target deletes the entire target directory before it does the installation. This is inconvenient, because we might have header files there that we want to retain (such as from the binutils build). To avoid any issues, we’re going to install the headers to a temporary location and copy them to the real target location from there.

Notice that we’re actually installing the headers into the scaffolding directory underneath the sysroot directory. That’s common to the entire host-side portion of CBL: Everything that will be conveyed to the target system is under the scaffolding location. That way, when the final system is completed, we can get rid of /scaffolding and be fairly confident that everything remaining has been built entirely from source and (if it continues to work) has no dependencies on the scaffolding programs and libraries.

GCC is the GNU Compiler Collection. This is the single most important package in CBL, even including the Linux kernel itself: without a compiler, you can’t build any software. We use GCC to bootstrap everything, including itself.

Unfortunately, GCC is also probably the most complex package we need to build, and it has a very complex configuration and build process because it is tied so intricately to other packages. This is particularly true of glibc, the C standard library we use in CBL, which also has a complex build process of its own!

On the plus side, if you can get GCC built and working properly, you’re past the biggest hurdle of building a complete GNU/Linux system entirely from source.

The GCC installation process is documented in the "Installation" manual, found in the source distribution in the INSTALL directory. If you have problems getting GCC built, that’s an excellent resource for figuring out what’s going wrong. If you want to get a better understanding of the GCC configuration and build process — how it actually works — read the "Source Tree Structure and Build System" section of the GCC Internals document, found in the source distribution in gcc/doc/gccint.info.

The most important compilers in the collection, for purposes of bootstrapping a system, are the C and C++ compilers; but GCC also includes compilers for D, Fortran, Go, Ada, Objective-C, and probably other languages as well. And it supports a huge number of machine architectures! Since CBL relies absolutely on having support for at least two types of computer for every build, that support for many architectures is probably the most important aspect of GCC for our purposes.

The program you usually invoke to build C programs, gcc, is not actually a compiler. It’s just a driver that knows how to invoke other programs, using rules called "spec strings" that tell it exactly what other programs it needs to invoke, and what command-line arguments it should provide, to turn source code into an executable program. (We’ll talk more specifically about spec strings in Adjusting the GCC specs.)

I find it really important to keep that in mind throughout toolchain construction, so I’ll expand on that point: gcc just invokes other programs. Some of those programs — the C preprocessor cpp, cc1 (the C compiler itself), and internal utility programs like collect2 (which is sort of a first-stage linker, used to set up calls to constructors and other initialization routines as a program starts to run) — are part of the GCC package. Others, like the as and ld programs, are distributed separately (in the case of as and ld, that package is GNU binutils).

To get from source code to an executable C program, gcc actually performs a series of steps:

You can find out exactly what commands gcc is running by giving it the -v (for "verbose") command line argument. That’s a handy trick when things are going wrong and you’re not sure why!

The term "compilation" — which is the job of the compiler — actually refers only to the second of those steps: source code is compiled into assembly code. It’s not precisely correct to say that you’re "compiling" source code into an executable program! That’s a common conversational shorthand, but it masks the complete story about what’s going on. It would be more correct to say that you’re "pre-processing, compiling, assembling, and linking" source code to produce an executable program.

On the other hand, that’s a lot of words, and the complete story is seldom really something you need to keep in mind. This is probably why the shorthand term is so popular.

Here, I’m focusing mainly on the C programming language. The two languages are very closely related, to the point that the earliest C++ compiler didn’t produce binary object code at all; it compiled C++ programs into C source code, which was then compiled into binary programs by a C compiler. A lot of the details about how C works apply exactly the same in C++; the differences are mostly things like the naming convention used for source files (C language source code files have the extension .c. There is no single convention that is used consistently for C++ source files! According to the gcc manual page, extensions for C++ source files include .cc, .cp, .cpp, .cxx, .CPP, .C, and .c++) and the program used to invoke the compiler (g++ rather than gcc).

I’m not discussing other languages, like Objective-C or Fortran, at all, because I don’t know anything about how they work!

C source code files generally have the extension .c. Along with these, you’ll often find header files, typically with the extension .h; these contain function declarations, macro definitions, constant definitions, and other things like that. These files can contain preprocessor directives, which are evaluated and executed by the C preprocessor before the code is compiled. Commonly, C source files contain #include directives that cause the preprocessor to copy the content of header files into the C file as part of this evaluation.

In addition to the header files that are included as part of the source code for C packages, UNIX systems have system header files that can be included by any program that needs them. These files can typically be found in the /usr/include directory and its subdirectories.

In addition to executable programs, packages can install libraries. These are basically collections of related functions that can then be used in other programs. These libraries are commonly installed in the /lib and /usr/lib directories (and their subdirectories). Sometimes, libraries are installed into variants of those directories, which can be set up on systems that support more than one kind of CPU instructions. For example, the AMD64 CPUs that are used in modern Intel and AMD systems support both 64-bit and 32-bit instruction sets, and GCC can produce both 64-bit and 32-bit object code. If you have one of these computer systems, you may want to have both 64-bit and 32-bit programs. In that cases, 64-bit programs will need to be linked against 64-bit verisons of libraries, and 32-bit programs will have to be linked against 32-bit versions, but both versions of the libraries will have the same filename. To allow these "multilib" systems, libraries will be installed in directories like /usr/lib32 and /usr/lib64 rather than simply /usr/lib.

In most cases, when a package installs a library in any of those directories, it also installs header files (into /usr/include) that specify how to call the functions in that library.

There are two different kinds of library files. Both are used when a program (or sometimes another library) are linked. Static libraries have the extension .a; when a program is linked against a static library, the binary code for methods invoked by that program is actually copied from the library into the program. Shared libraries, with the file extension .so, are the other kind. When a program is linked against a shared library, a reference to the library is written into the program, but the code for library functions is not. The benefit of this is that, no matter how many programs you have that use a common function like memcpy or printf, only one copy of the function code needs to be present on the system, in the shared library that defines them. Programs with such library references are called "dynamically linked," and when they are run, the program interpreter is responsible for tracking down the shared libraries it uses and finding the code for functions within them. We’ll talk more about how it does that later on!

One library present in the Little Blue Linux system is simply called "the C standard library." The functions in that library are defined as a part of the C language, and almost every C program uses some of those functions. (For example, the "Hello, World" program that is traditionally the first C program written by those learning the language uses the printf function defined in the standard library.)

This package does not include an implementation of the standard library.^[6] In the GNU system (and in Little Blue Linux), the standard library is provided by the glibc package — which also provides the program interpreter that resolves shared library references in dynamically-linked programs. Although glibc includes both static and shared versions of libraries, it does not really work well with statically-linked programs. So the vast majority of executable programs found on GNU/Linux systems are dynamically linked.

After being compiled and linked, executable C programs are typically installed into directories like /bin, /sbin, /usr/bin, and /usr/sbin. The same directories also contain all of the other programs that are part of the system, regardless of whether their source code is written in C.

Many of these topics are discussed in more detail elsewhere in CBL. The others are discussed in more detail somewhere, but you might have to do some searching! In any case, that’s a basic summary of how C programs work on LBL — and other GNU/Linux — systems.

A few external dependencies were added to gcc in release 4.3: the GMP, MPFR, and MPC libraries are required for all compiler builds, and a few optimizations — the graphite loop optimizations — are only available if the Integer Set Library (ISL) is available. The graphite loop optimizations are not critically important, but there’s no reason not to include them if it’s not difficult to do.

Sources for the required — and optional — libraries can be included in the GCC sources in directories named gmp, isl, mpc, and mpfr; if they are, then they’ll be built automatically along with gcc. There’s actually a fairly large number of packages that the build machinery for GCC detects and incorporates into builds automatically. This is convenient, but restrictive: in-tree builds are only reliable when specific versions of the dependency libraries are present, and for CBL I prefer to use the latest stable release of everything.

There are also some cross-compilation scenarios in which the in-tree library builds actually do the wrong thing and produce a compiler that will not work right; that’s another reason that we avoid them here.

There’s a problem introduced in GCC 6.1 involving hard-coded paths to C++ header files. This prevents them from being found by the scaffolding compiler (which lives at a different filesystem location when the target system is booted). We can apply a patch to work around that.

When specifying locations for the dependency libraries (GMP, MPFR, etc), the specified library location should be used both at build time (with an -L linker directive) and at run time (with an -rpath link directive). This is not always done by the normal GCC build process, but we can patch it easily to add that behavior.

Even though the math libraries needed by GCC (GMP, ISL, MPC, and MPFR) have just been built here and GCC is configured to find them in their correct locations, some of the programs that make up GCC are built without an RPATH or RUNPATH, so the dynamic loader will look for those libraries at runtime in the normal host system library directories. This was filed as GCC bug 84153, but as of the last time I looked at it, the GCC maintainers don’t consider it a problem.

To be fair, this really is not usually much of a problem! The issue appears when the version of the library used in CBL is a major version later than the version installed on the host system. In that case, the GCC we’re building in this section won’t work because it won’t be able to find the version of the libraries it depends on.

This is the sort of situation that LD_LIBRARY_PATH is intended to resolve, but there’s a gotcha: in some of the host-scaffolding builds, the native toolchain is used to build some programs that are run as part of the build process itself, and the LD_LIBRARY_PATH set in the environment for the cross-build is unset when building those native programs.

That means that to get the this native gcc to be reliable, we need to set a RUNPATH or RPATH to tell the dynamic loader where to look for shared libraries. The way you normally do this is to set an LDFLAGS environment variable with a value like -Wl,-rpath,/home/random/cbltools/lib when running the configure script, so that ld would be told to build programs with that RPATH, but it turns out the GCC build doesn’t use the LDFLAGS linker arguments when constructing some of the programs that need the dependency libraries, like cc1, cc1plus, and lto1.

We can work around that by patching the configure script so that any time it’s given arguments like --with-gmp or --with-gmp-lib, the flags it collects will include a -Wl,-rpath option along with the -L option. That’s what this patch does.

GCC 13.2.0 sometimes does not build properly when using glibc 2.38 because libsanitizer expects to do things related to the libcrypt library provided by glibc, but starting in glibc 2.38, that library is no longer built by default. I found a patch in GCC bug 111057 that fixes this.

Bootstrapping GCC as part of a cross-toolchain is tricky. You can build the compiler per se without a working standard library (libc), but that compiler won’t be able to produce executable programs: GCC can only create programs if it has access to a set of C runtime object files that it expects to be provided by the C library — and, realistically, it also needs a C library to compile programs because essentially all C programs invoke standard library functions.

Obviously, we can’t compile the C library without a compiler! So there’s a chicken-and-egg bootstrapping problem.

To work around that cycle of dependencies, we’re going to start by building just the parts of the compiler we really need at this point: a plain cross-compiler, and a minimal version of the libgcc support library that the compiler needs in order to function. (GCC needs libgcc because sometimes, while processing C code, the compiler generates references to functions defined in libgcc rather than generating assembler code.)

GCC can most easily be built as part of a cross-toolchain by using the "sysroot" framework — and, in fact, the GCC developers don’t support any other method for creating cross-toolchains. To perform a sysroot build, you specify the configure options --with-sysroot and --with-build-sysroot; and when building GCC, set the environment variables LDFLAGS_FOR_TARGET and CPPFLAGS_FOR_TARGET to specify --sysroot as well.

…​At least, that’s what Carlos O’Donell said in a comment on GCC bug 35532. The documentation on sysroot builds is not particularly easy to find — or at least, it wasn’t when this was written. (If you know where sysroot builds are documented, please tell me!)

The sysroot concept is pretty clever, in fact. The basic idea is, when you build a cross-toolchain, you give it a local directory path — a directory that exists on the build system — and tell the cross-toolchain that that local directory will eventually become the root filesystem directory on the target system. The cross-toolchain then knows to look for system header files and shared library files and so on in the sysroot location.

CBL uses the standard sysroot approach, in a slightly-nonstandard way: everything is installed not into the sysroot directory per se, but into a subdirectory of the sysroot called /scaffolding. That way, when we finally get booted into the target system, the root filesystem will be empty, except for the /scaffolding directory: everything we create after that, and install outside of /scaffolding, will become part of the final system.

Depending on the target, GCC has a variety of options that control how it operates. Generally, these can all be specified with command-line arguments beginning with -m. For many of these options, default values can also be specified when GCC is being configured.

For example, for many targets, GCC can build programs with a variety of application binary interfaces (ABIs) — this is the machine-code-level interface between programs, libraries, and the operating system; it defines things like the way that registers are used when invoking functions.

An example of a CPU that supports multiple ABIs is the 64-bit x86 architecture, which is called x86_64 or amd64. As mentioned earlier, these processors can run programs in 64 bit mode (with 64-bit pointers and all AMD64 processor features enabled), 32 bit mode (32-bit pointers and only i686 processor features enabled — in this mode, fewer CPU registers are availble to programs, for example), or a hybrid "x32" mode (32-bit pointers but all AMD64 processor features enabled). You can specify the ABI to which GCC will compile by using an -m32, -m64, or -mx32 command-line argument.

You can also override the default ABI when configuring GCC by specifying a --with-abi configure directive; or, for some target architectures, other options, like --with-multilib-list or --enable-targets or probably a combination of those things. Confusingly, even though the way you set an x86 GCC to generate code for the 64-bit ABI is to use the configuration directive --with-abi. Although there is a runtime command line option -mabi, it doesn’t override that selection; it only selects between the sysv calling convention used by UNIX-ish systems and the ms convention used by Microsoft Windows.

This confusion is characteristic of the GCC configuration and usage options: a lot of the options that are available, and their meaning, depends on what architecture is targeted by the compiler, and the same options or terms can mean very different things for different targets.

There are a few ways to figure out what options are available for a specific target architecture: the installation manual lists some of them, mostly in the "configuration" section. The GCC manual also has information in section 3.18 ("Machine-Dependent Options" — if you’ve built a set of trusted host tools specifically for the CBL build, the correct info file can be found at /home/random/cbltools/share/info/gcc.info). You can also look at the configuration script, gcc/config.gcc, to see what options are accepted for each type of target. And, finally, after building GCC for the target architecture, you can run gcc --help=target to see what options are available and what values they can have; and you can also find the actual compiler, cc1 (for C) or cc1plus (for C++) under the libexec/gcc/aarch64-cbl-linux-gnu/$VERSION directory, and run it with the --help command-line argument to see what options are enabled and disabled.

In addition to the selection of ABI, GCC can be instructed to optimize for specific CPUs or CPU families for many target types. The command-line arguments that control this behavior, like the ABI arguments, are target-specific — look for -mtune, -march, -mcpu, and things like that.

In CBL, any set of these configure directives can be specified for the cross-toolchain in the TARGET_GCC_CONFIG parameter. It’s generally a good idea to specify default values for all of the options supported by the target platform. (If you don’t want to set any options at all, you can set the parameter to an empty string.)

GCC is generally built with a large number of libraries included. Some of those fail in some circumstances — for example, x86 CPUs can’t build the libquadmath library when the C library being used is uClibc (or, at least, that was the case some time ago, the last time I tried); and the libsanitizer library fails to build when compiling for 64-bit Sparc machines. The easiest way to work around those problems at this stage is simply to disable those libraries, and, considering that the cross-toolchain we’re building here is just going to be used to build the ephemeral scaffolding programs, that seems like a reasonable approach here. So if any libraries cause the build to fail, try adding an appropriate --disable directive to TARGET_GCC_CONFIG.

You might notice that, in this build, we’re using the same host-system builds of the various arithmetic dependency libraries as we used for the host-prerequisite GCC (or the same ones that are used for the native system GCC, if you’ve skipped the host-prerequisites). It’s totally unnecessary to build them again for this GCC — the cross-compiler we’re building here will only run on the host system, so the target architecture is irrelevant to it.

To build the minimal libgcc, we specify the configuration options --without-headers and --with-newlib. This is a bit sloppy — the first of those options is the only one that should be necessary — but the last time I tried building a minimal compiler without the newlib directive, it didn’t work.

Once this minimal compiler is finished, we’ll use it to build the C library; and then we can use the files provided by the C library to go back and build a full, functional GCC compiler.

The only targets we build at this point are all-gcc, which produces the plain compiler, and all-target-libgcc, which produces the minimal libgcc needed by that compiler.

glibc is the C standard library produced as part of the GNU project. It contains the implementation for all of the functions that are assumed to be available in C programs — like printf and so on. It also provides a dynamic loader, which almost all programs use to find shared libraries at runtime, and a few miscellaneous utility programs.

Since all C and C++ userspace programs (except the Linux kernel itself) link against the C library, glibc is by far the most deeply-embedded component of a GNU/Linux installation. Upgrading the Linux kernel is a relatively trivial operation compared to upgrading the C library installed on a computer.

There are alternative C libraries that can be used instead of glibc. However, glibc is the C standard library used by the vast majority of GNU/Linux systems; using an alternative library like musl or uClibc-ng as the primary C library may cause problems somewhere down the line.

This builds a sysroot libc (for the target architecture), configured to install into the /scaffolding subdirectory of the sysroot.

That might be a little bit opaque, so let’s break it down a little bit. As mentioned earlier, the sysroot framework is all about setting up a path on the host system that will eventually become the root filesystem on the target system. And, also as mentioned earlier, the goal in the first stage of the CBL build process is to set up a kernel and minimal userspace for the target system, with the entirety of that userspace contained in a /scaffolding subdirectory rather than using the conventional filesystem paths (like /bin and /lib and the /usr directory structure and all the rest of that stuff you’re used to seeing).

What we’re building here is the libc that’s going to be used to build all the scaffolding programs and libraries — the stuff that we’re cross-compiling — so we want it to be contained in the scaffolding directory, and found by the scaffolding programs at runtime in that location. Hence, it’s a sysroot libc, with an extra prefix to move it from its usual normal /lib and /usr/lib directories to the /scaffolding/lib directory.

That means we configure it with a prefix of /scaffolding, but then when we install it we tell it that the root of the installation location is the sysroot directory /tmp/cblwork/sysroot.

Simple, right?

If you ever want to set up a completely standard sysroot toolchain, by the way, it works pretty much the same way as this but you specify --prefix=/usr and --with-headers=/tmp/cblwork/sysroot/usr/include. There is some magic in the glibc configuration or build machinery related to the --prefix directive: if you specify a prefix of /usr, the bits of glibc that are conventionally installed in /lib will be put there, rather than in /usr/lib.

Since this glibc is built for the target machine architecture, a number of tests run by the configure script won’t work right. The way we work around that, for glibc as with everything else that uses the GNU build system, is by setting the correct values in a config.cache ahead of time.

An option that we’re not using here is --enable-kernel, which can limit the amount of compatibility code built into glibc to support old Linux kernel versions. Since the glibc being built here is temporary and will be discarded in its entirety, saving a little bit of space here is kind of pointless. That option also makes the build more fragile, since the kernel version that will be checked by the code is the host system kernel; we don’t want to make any more presumptions about the host system than we must.

The glibc build process uses the makeinfo program to create the documentation, and the texinfo source file specifies a document encoding of UTF-8. When using some versions of perl, this leads to problems — it’s unclear why; perhaps this is because makeinfo only works right with UTF-8 documents when being run with UTF-8 localization settings, or maybe something else is going wrong.

Regardless of exactly what triggers the issue, there’s an easy way to work around it: just remove the @documentencoding directive from the libc manual source file.

As mentioned earlier, in gcc: The driver program, the gcc program we think of as a compiler really just runs other programs, and it uses a bunch of directives called "spec strings" to determine what programs to run and what options to give them. The format of specs strings is documented in the GCC documentation in section 3.15, "Specifying Subprocesses and the Switches to Pass to Them." Spec strings don’t have the most readable structure — I find it helpful to think of them as being written in a domain-specific language, because to me they look as much like line noise as complicated regular expressions do — but sometimes there’s no better way to figure out what is going on than to read the specs, and there is often no better way to adjust the behavior of gcc than to modify the specs it is using.

We have to do this — modify the specs — a few times throughout the CBL process, primarily because we need to control how gcc runs ld to link programs.

You can see the spec strings that gcc will use by running gcc with the -dumpspecs option. The default specs are built in to gcc, but you can provide your own specs to override the default behavior by using the -specs= command-line argument or by creating a specs file and putting it in a specific location in the filesystem.

We’re going to do the latter. The basic process is to first dump the specs file to the location where gcc will look for it:

Then we can modify it however we need to, and gcc will use the modified version.

Not to belabor the point, but remember that the whole purpose of this cross-toolchain is to let us build the scaffolding that will then allow us to build the final target system entirely from source code.

While we’re using the scaffolding tools to build the final system, we want to do a native build of everything, including glibc, so we want the scaffolding to be independent of any filesystem locations that will still be present in the final system. In particular, this includes /lib and /usr/lib. That way, once we’re done doing the target system build, we can delete the scaffolding directory altogether and be confident that there are no lingering host-system artifacts or ephemera on it.

When the standard GNU toolchain builds an executable, it almost always links it against the dynamic link library (sometimes called the "dynamic loader" or "program interpreter"; this is something like ld-linux.so.2 or ld.so.1, and is conventionally found in a top-level directory called /lib or a multilib variant like /lib32 or /lib64).^[7] That’s normally fine, but we want the scaffolding to be entirely independent of /lib. So we need to adjust our cross-toolchain so that the programs it builds look in the /lib directory under the scaffolding location for their libraries, including the dynamic link library. This is done by modifying the GCC specs file.

There’s another problem we need to work around, as well: gcc doesn’t provide any good way to tell it where to find some object files that need to be linked into every program: crti.o, crti1.o, and crtn.o. These are provided as a part of glibc, so like the rest of glibc they were installed into /tmp/cblwork/sysroot/scaffolding, specifically into its /lib subdirectory. But, of course, that’s not a location where gcc normally expects to find them. So at this point if you were to try compiling a "Hello World" program with your cross-toolchain, it would complain that the cross-ld can’t find crt1.o or crti.o.

You can tell exactly what is going wrong by repeating the compile with -v, to get gcc to print out all the commands it’s running: cc1 to compile the code, then as to assemble it, then collect2 to link it. And apparently collect2 is running ld, which is producing the error message. (That’s weird, but you get used to weird stuff when you’re trying to figure out toolchain problems. There’s also no explicit execution of cpp; that’s because the C pre-processor is actually implemented in the libcpp library and is invoked as function calls to that library by the cc1 compiler program.)

Once you have the actual command line that’s failing, you can try adjusting it to see if there’s an easy way to get it to work. For example, the command line just names the startup files without any path components at all. After fussing around for a while looking for pleasant alternatives, the best solution I found was to specify the filenames with absolute paths. So that’s what we’re going to do in the specs file: replace every occurrence of the bare filename with absolute paths, for all the object files that appear in the scaffolding/lib directory.

We can now verify that the cross-toolchain is able to build programs successfully, and is set up to link against the sysroot glibc in the /scaffolding directory, by compiling any simple program (like "Hello World") and then running readelf on it — there will be a line in the program headers section that says "Requesting program interpreter:" and should contain the path to the dynamic link library in the scaffolding location.

After building a significant toolchain component, it’s a good idea to make sure that it works as intended. This is a simple smoke-test: it just compiles a "Hello, World" program and then inspects it to make sure it was built as expected and runs properly.

This verifies that the cross-toolchain and emulator work properly: look at the machine type and dynamic linker location for a compiled program, and then make sure that it runs in the userspace emulator. This proves that the cross-toolchain and QEMU were properly built with a compatible target architecture and so on.

This is compiled with:

To verify that it’s linked properly, use readelf.

The Machine line should indicate the target architecture, rather than the host architecture, and the program interpreter requested by the program should be under the /scaffolding/lib directory (which is where the dynamic loader will be found once we’re booted into the target system). If either of those is not the case, the grep commands will fail, causing the CBL build process to abort.

One last thing we can usefully do at this point is verify that the user-mode QEMU emulator can actually run programs for the target architecture.

This is a bit tricky when running the dynamically-linked program we just built, because it is expecting to find the program interpreter at /scaffolding/lib but in fact it’s actually at that location under the sysroot directory. Luckily, the user-mode QEMU emulator can be told where to find library files, using the -L command line argument or the QEMU_LD_PREFIX environment variable.

This will produce a friendly greeting, or — if something goes wrong — will, again, cause the build process to abort.

(If you’d like, you can try running the hello program outside of QEMU as well; that should produce an error message like "cannot execute binary file." And, again, if it doesn’t, that means something is horribly wrong!)

pkg-config is a program that makes it easy to find installed libraries and header files and things. Many programs use pkg-config in their build processes to find out if the libraries they depend on are present on the build system, and to find out what linker and include directives should be used to compile and link against them.

The original pkg-config program can be found at pkg-config.freedesktop.org. At some point in its development, its developers decided to use functions from the glib library; unfortunately, glib uses pkg-config to find its own dependencies (really, just zlib — glib doesn’t have a lot of dependencies), which introduced a cyclic dependency. That doesn’t cause any really intractable problems — a couple of environment variables can be defined when building glib so that it doesn’t have to use pkg-config to find zlib — but cyclic dependencies are always kind of horrifying: not to sound like a broken record, but the idea with CBL is to start with a minimal set of binaries and a pile of source code and turn that into a whole system. (Even worse than the glib dependency, pkg-config requires itself, so that it can find and link against the glib library, unless additional environment variables are provided.)

The situation has since been resolved, but it was resolved by bundling glib along with pkg-config. This isn’t a particularly elegant solution: the pkg-config distribution is about 11.5 megabytes of code, unpacked, and about 9.5 megabytes of that is glib.

A different approach was taken in a fork of pkg-config, "pkg-config-lite," which includes just the snippet of glib that is needed by pkg-config: that’s better, but still not ideal.

This brings us to pkgconf, a completely separate implementation of the pkg-config program. It has no external dependencies, doesn’t bundle any third-party code within its own distribution, and also has a design that I find preferable to the original pkg-config (it internally builds a directed acyclic graph of dependencies, rather than building an in-memory database of all known pkg-config files at runtime and then resolving dependencies from that database).

As distributed, the pkgconf program is missing a lot of files that are normally produced by the GNU autotools; the conventional way to build it is to start by running the autogen.sh script provided with the package distribution. However, one of the places the pkgconf program is built is as part of the host prerequisites; if that’s being done, it’s probably not a great idea to rely on the autotools being present at prerequisite build time. We can instead simply patch the source distribution to create the files that would normally be created via autogen.sh.

This is built here because pkg-config is already present on the host system (as a prerequisite, if nothing else), and it probably knows about a lot of host system libraries. While building the scaffolding, we need to ensure that their build processes don’t find (and try to link against) any of the host system libraries. Since the host system libraries are for a different machine architecture, this would cause build failures. We can do that by putting a new version of pkg-config, configured only to look in the scaffolding directory, at the head of the PATH while building them.

We could probably get by with a simple shell script that always just says "I couldn’t find anything!" when asked for dependencies. On the other hand, it takes around ten seconds to build pkgconf and set it up, so why not just do that?

In addition to the pkg-config symlink, we create a symbolic link aarch64-cbl-linux-gnu-pkg-config so that any build process that tries to find dependencies using a target-system pkg-config program will find ours.

Most Linux filesystems support "extended attributes" — arbitrary name/value pairs that can be associated with files or directories. These can be used for any purpose; for example, you might attach a "user.file_encoding" extended attribute to a text file, if it’s encoded unusually.

One of the primary uses of extended attributes is to implement access control lists or capabilities.

The attr package provides programs that allow extended attributes to be viewed and modified.

Since the configuration repository preserves extended attribute metadata as well as basic owner and mode, and since the package-users build script sets a user.package_owner attribute on all files that are installed by packages, we need the getfattr and setfattr commands early in the target system build.

Bash is a shell — it provides a command line prompt, parses commands and pipelines entered on the command line, executes programs based on those commands and pipelines, and then does it all again. If you’re interacting with a Linux system, and you’re not using a graphical environment like Xorg, you’re using a shell like bash. And, probably, you’re using bash — there are other shells, but they are not nearly as common.

The bash maintainers don’t provide all patchlevels for convenient download; you may have to download the tarfiles for the main version (like 4.3) and then separately download and apply all the update patches. The files.freesa.org repository, for convenience, has a tarfile that includes all of the patches for the version specified in this blueprint.

Several of the tests run by the configure script don’t work right when bash is being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

Bash includes its own memory allocation routine, but historically it has not always been reliable. We use --without-bash-malloc to disable it so that the C standard library’s malloc is used instead. For the scaffolding version of bash, we also configure it to be linked statically — that reduces the number of pieces that are needed to get the minimal scaffolding-based system to boot.

M4 is a macro processor: it copies its standard input to standard output, expanding macros as it goes. It has a set of built-in macros that it understands, and it’s possible to add user-defined ones as well. M4 is used extensively in the GNU build system, primarily by Autoconf.

Bison is a parser generator. That means it takes a description of a language (in a specialized format called a "context-free grammar") and converts it into a parser for that language. This is mostly useful when writing compilers: the parser is the part of a compiler that takes a stream of tokens and figures out what syntax elements they represent. Parsers are fiddly and difficult to get correct, so it’s common to generate the code for parsers using tools like Bison.

The name "Bison" is kind of a joke. The original parser generator that was commonly available on UNIX systems is called "Yacc," which is an acronym for "Yet Another Compiler-Compiler." When the GNU project implemented their own parser generator, they called it "Bison" because "Yacc" sounds like "Yak."

Bison’s installation process also creates a program called yacc that simply runs bison in yacc emulation mode.

The absolute path to the m4 program is written into bison, so we need to make sure that the scaffolding bison will expect to find m4 in the scaffolding directory. This adds a little complexity, though: the bison configuration script checks to see whether lex is available and, if it is, whether it is flex. flex uses m4, so if we tell the bison configuration that m4 can be found in the scaffolding, flex will try to use that version of m4, which was built for the target system rather than the host system. As with other packages that use the GNU Build System, we can use config.cache to tell bison that flex is simply not available.

Bzip2 is a compression program like gzip, but uses the Burroughs-Wheeler block sorting algorithm and Huffman coding. It is a lot slower than gzip for both compressing and decompressing, but compresses data much better in most cases.

Bzip2 doesn’t use the GNU build system, so there isn’t a configure script.

There isn’t really a configuration step for bzip2, but we’ll use the configuration stage to modify the build process so it won’t try to run the tests — they won’t work when building with a cross-toolchain.

The regular Makefile only creates a static version of the library, which is fine; however, it doesn’t compile that library with -fPIC, which causes problems later on when other programs try to use it. We can add that directive directly in the Makefile.

Because bzip2 doesn’t use the GNU build system, we need to specify the cross-tools as arguments to make.

The Core Utilities are basic file, shell, and text manipulation utility programs: things like cat, chmod, chown, cp, …​ stuff like that. You use these all the time.

A couple of the tests run by the configure script don’t work right when the coreutils are being cross-compiled. As with other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

Diffutils is a bundle of programs that find differences between files and help you to merge them together: cmp, diff, diff3, sdiff.

The most important program here is diff, which finds differences between files. It can find differences between two files, in the simplest case; but, more commonly, diff is used to find all the differences between all the files in two entire directory trees.

You can save the output from diff when used this way as a "patch file", and then use the patch program to take one of the directory trees and make it look just like the other one.

This is handy when distributing modified versions of software packages: if you’ve made a minor change to GCC, for example, and you want to submit that change to the GCC mailing list for consideration by the GCC maintainers, you can use diff between the original GCC source tree and your modified GCC source tree, and then include the output from diff in your email.

When cross-compiling, the configure script guesses that the getopt function provided by GNU glibc is not available and tries to use an internal one instead. This doesn’t work, for some reason (possibly because of behavior changes in GCC 7). Since the glibc getopt is available, we can just tell the configure script not to guess about it.

util-linux is a grab-bag of miscellaneous Linux utility programs, for all kinds of things: disk partitioning, filesystem creation and validation and mounting…​ all kinds of stuff like that. (I’d expect a lot of this stuff, like more, mount, and umount — to be a part of the GNU system, perhaps in the coreutils package, but that’s not the only thing I find surprising about the GNU/Linux world.)

We configure with an option that tells the build machinery not to chgrp the wall program to the tty group. This will probably make wall non-functional, but that’s perfectly okay — wall can be used to send messages simultaenously to all users logged on to the system, as for example if the system is going to be shut down or something of the sort, which is not very useful on single-user computers and certainly not necessary in the minimal scaffolding userspace.

You’ll notice that we’re configuring this package with a prefix of /tmp/cblwork/sysroot/scaffolding instead of just /scaffolding (and installing without a DESTDIR). That’s because, when we configure and install util-linux the way we’re doing most of the scaffolding components, we’ve sometimes encountered problems where some components can’t find the libuuid library that is provided by this package. There are probably other ways to address that, and it’s not really clear whether those components are strictly necessary, but this non-standard configuration scheme doesn’t cause any problems and works fine.

The installation routine for util-linux changes the ownership and mode of some programs (like mount and umount) to be setuid to root. This doesn’t work when running the installation as a normal user, as is being done in this section, so we disable that.

The ext2fsprogs are utilities for managing the ext2 (and later) filesystems. ext2 was the first popular filesystem for Linux systems and its latest incarnation, ext4, is the most common. This package contains programs for creating filesystems, reconfiguring them, checking them for problems, that sort of thing.

There’s some overlap between e2fsprogs and util-linux: both provide libuuid and libblkid libraries, a uuidd daemon, and a fsck script. We skip those from e2fsprogs.

There is a glitch in the configuration logic that creates a file outside of the DESTDIR-specified path when there is no /etc/cron.d directory. This can be avoided by explicitly specifying that there is no crond directory at all.

Expat is a library that provides a stream-oriented XML parser. It’s used by all sorts of other programs.

file guesses file types by looking for particular characteristic byte values within them. If you’ve got a file and you’re not sure what it actually is, you can run file on it and be told things like "that’s a text file," or "that’s a JPEG image."

On Microsoft Windows, file types are inferred based on the extension of the file — exe files are executable programs, jpg programs are JPEG images, and so on. file does things differently: it looks at the content of the file and tries to determine the file type by looking for characteristic patterns within it, using a compiled database of these patterns called the "magic file," which you can read more about with man 5 magic. The magic file is produced during the build process by running file -C (with the freshly-built file program) to compile a bunch of "magic fragment" files into the database that it normally uses.

The Find Utilities are basic directory searching programs: mostly find and locate. This package also includes updatedb, which creates the file name database that locate uses; and xargs, which takes lists of file names produced by find and generates command lines that operate on all the files in those lists.

The GNU version of find has no support for extended attributes, but since Little Blue Linux uses extended attributes to preserve the name of the package owning files even when they are `chown`ed or `chgrp`ed, we need that feature for use in some of the package user scripts. I found a patch from Nicolas Iooss that adds this functionality.

A couple of the tests run by the configure script don’t work right when the findutils are being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

AWK is a special-purpose programming language that makes it easy to do simple text manipulations. (The name is an acronym of the three people who designed the language: Alfred Aho, Peter Weinberger, and Brian Kernighan.) Gawk is the GNU implementation of the AWK language.

If you’re comfortable in perl or python or ruby, you probably won’t have much use for awk; on the other hand, if you’re working in a bash script, awk might turn out to be very handy.

Zlib is a compression library. It implements the same Lempel-Ziv compression algorithm as gzip and info-zip, which means it doesn’t compress data as effectively as most other algorithms (like the Burrows-Wheeler algorithm used by bzip2 or the LZMA algorithm used by xz-utils), but on the other hand it doesn’t use very much memory to compress, and it’s pretty fast.

There are a lot of programs that link against zlib to get basic compression capabilities. We’re not totally sure which components will fail to build if it’s not available, but zlib itself has no dependencies and is really fast and small to build, so making it available is no big deal.

Gettext is a set of tools that can be used by other programs to provide internationalization and localization capabilities. This lets a single program provide user interfaces and user-facing messages in a variety of languages without requiring it to be rebuilt.

One of the tests run by the configure script doesn’t work right when the coreutils are being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

If the emacs editor is available on the system, the gettext build will do something with it. I don’t really understand what it is, but it generates a couple of gigabytes worth of log output and makes the build process take orders of magnitude longer than it otherwise would. Luckily, it is easy to disable that with a configuration flag.

Grep searches input files for lines that match patterns called "regular expressions." It often prints those lines out as well.

We have heard that the program name "grep" was originally derived from the ed command "g/re/p", which does basically the same thing as the command-line grep program, and wikipedia makes this claim as well. Maybe it’s true!

The GNU libc library provides regular expression functions, but grep ignores them by default and uses its own bundled regex library. We use a configuration flag to override that behavior.

Gzip is a compression program that basically does the same thing as the zlib library, but with a command-line interface.

The default -Werror setting for gzip causes a format-truncation message produced by GCC 8 to abort the build. I am not all that concerned about format-truncation problems when printing error messages, so I just disable that setting.

The generated Makefiles for gzip include a -Wabi directive that GCC 8 complains about. We can just remove that directive.

The Kbd package contains key-table files and keyboard utilities.

The vlock program requires Pluggable Authentication Modules (Linux-PAM), which is not part of the basic CBL system, so we always disable it. Conversely, other programs that aren’t built by default might be helpful, so we enable those.

We need some of the programs provided by kbd (notably, openvt, which will let us run interactive shells on virtual terminals in the minimal target system) in the minimal userspace.

Libffi is a portable library that helps programs written in one language to invoke functions that were written in a different language.

LibreSSL is a fork of OpenSSL with the goals of modernizing the codebase, improving the reliability and security of the code, and applying better development practices.

As an added bonus, the LibreSSL package includes a program called nc (short for "netcat") that can be used as a simple TCP and UDP client or server program — this can be enormously handy if you just want to shove data over a network connection, or test network client or server programs, or something like that. You can man 1 nc if you’d like to see more about it.

LibreSSL should not be needed in the scaffolding, but the current version of rubygems has issues when used in a ruby installation without openssl support.

Generally, when you log in to a UNIX system, you provide a username and a password. Obviously, the password needs to be kept secret; I don’t think there’s any need to belabor that point. When you log in, the system — in this case, the login program — has to validate that the password you entered is correct. It does this by consulting an authentication database file.

If some nefarious person were to want to compromise the security of a system (I know it’s shocking, but there are some nefarious people out there), they might try to do this by obtaining a copy of the database file. System designers and administrators try to keep that file locked down so that it’s not feasible for nefarious people to make a copy of it, but some bad actors are very clever and find ways to do so. To make that situation less problematic, the database file does not actually contain the passwords that users enter. Instead, when users enter a password, the passwd program mangles the password using a hashing function that cannot be reversed, and stores the mangled, or "hashed," string rather than the actual password. When a user logs in, the login program uses the same hashing function on the password that they provide, and compares the hashed value to the hashed value in the password database. If they are the same, then login can conclude that the user entered the correct password.

libxcrypt is an implementation of these hashing functions.

Historically, these hashing functions — primarily crypt and crypt_r — have been provided by glibc. Rather than being part of the main shared library libc.so, these functions were put in a separate library libcrypt.so. However, in glibc release 2.38, this library is no longer built by default; it can be constructed by setting the --enable-crypt configure flag, but the recommendation in the glibc release notes is to switch to an alternative implementation like this one. Conveniently, libxcrypt builds a libcrypt.so library that is compatible with the one that has historically been provided by glibc.

YAML — the name stands for "YAML Ain’t Markup Language" — is a human-readable data serialization format. You can read all about it on https://yaml.org/. (Perhaps worthy of note, directives in litbuild blueprints are written in a very YAML-ish syntax.)

LibYAML is a library, originally written as part of the PyYAML project, for parsing YAML into data structures and emitting YAML from data structures. It’s a convenient serialization format for cases where someone might need to look at the serialized data and understand what it is without any arcane hackery or tools.

The Ruby standard library includes a YAML implementation that depends on LibYAML.

GNU make is a build automation program. To use make, you set up one or more configuration files — the primary one being called, by convention, Makefile — that declare recipes for producing intermediate and final program artifacts, and dependencies between artifacts, and the various targets that can be produced. At runtime, make looks to see what artifacts exist already and which source files have a timestamp indicating that they’ve changed after dependent artifacts were produced, figures out based on that analysis exactly which artifacts need to be rebuilt, and then executes the recipes that will produce those artifacts.

That’s all pretty awesome!

The downside is that Makefiles are not particularly clear or readable, and for large projects they get pretty big and complicated. That’s why there is a thing called "litbuild"!

But the vast majority of system components use make to automate their build process, so you really have to have it available regardless.

Ncurses ("new curses") is a library that provides terminal control features so programs can provide advanced text-based user interfaces. It’s a free-software version of a similar library (which was simply called "curses") that was developed at Berkeley.

There are lots of programs, even just in the scaffolding we’re setting up, that use ncurses. bash and vim are among them.

Patches are available in a version subdirectory of the overall package download location — for example, patches for ncurses 6.2 are in the "6.2" subdirectory of the ncurses package directory. Patches need to be applied in order; there are instructions in the README file there, but the basic idea is that you find the latest patch bundle, called something like ncurses-6.1-20181020-patch.sh, and run that as a shell script; then you apply all the patch files later than that in sequence. When I update ncurses for CBL, I fetch all the patches available at that point, verify their GPG signatures, and compile them into a single branch-update patch, available in the file repository at files.freesa.org.

When configuring the scaffolding ncurses, we specify --without-ada, since Ada isn’t built in the CBL process. We also specify --enable-overwrite so that the header files are installed into /tmp/cblwork/sysroot/scaffolding/include rather than an ncurses subdirectory of it, because that makes it easier for other programs to find the headers, and --with-build-cc so that the native tools built as part of the ncurses build process use the native compiler instead of the cross-compiler. We also specify --disable-stripping because otherwise the install program might try to strip binaries using the host system strip rather than the cross-toolchain strip.

Remember diffutils? The most important program included in diffutils is diff, which finds differences between two files or directory trees. Patch is the conceptual inverse of diff — it takes a file with all the differences found by diff as input (which is by convention called a "patch file"), and applies all of those differences as changes to files in a directory tree.

This is really handy. For example, the CBL repository contains a bunch of source tarfiles that were obtained directly from the project web sites; and it also contains patch files that can be applied to those source tarfiles to apply changes that we have found to be necessary or important when building a CBL system.

Pth is a portable threads library; it provides cooperative priority-based scheduling for multiple threads within a program.

The build system for pth doesn’t work well when parallelized, so we explicitly disable parallel make jobs by specifying -j1.

On some servers, the version of config.guess distributed with pth is too old to recognize the target triplet. It’s easy enough to update it to the version distributed with GCC.

In some cross-compilation scenarios, pth fails to compile because it considers versions of the Linux kernel later than 2.9 to be "braindead." I got a patch from https://bugzilla.redhat.com/attachment.cgi?id=591825 that corrects the problem.

Ruby is a programming language. There are several implementations of that language — rubinius and jruby are others — but the canonical one is the C implementation that was created by Yukihiro "Matz" Matsumoto, and that’s also the best one for our purposes in CBL because it has the fewest upstream dependencies.

CBL includes Ruby as a core component because CBL itself is designed to be used with the litbuild program — CBL consists entirely of litbuild blueprints — and litbuild is written in Ruby.

On 64-bit ARM systems, the Ruby build process exposes an issue with GCC (documented at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84521) when compiled with -fomit-frame-pointer and an optimization setting of -O1 or -O2. Since the default behavior for modern versions of GCC is to omit frame pointers, it’s important on such systems to specify -fno-omit-frame-pointer when building Ruby. It’s possible that other architectures have a simliar issue; if your ruby build crashes with a "stack smashing detected" error message, try adding that option to your builds as well. (The default configuration for CBL specifies -fno-omit-frame-pointer in the target-system CFLAGS.)

Once we boot into the target system, we’ll use litbuild to generate all the scripts that will build the final CBL components. That means we’ll need ruby available!

As with the file package, ruby can have issues in cross-compilation scenarios unless there is a native ruby installation of the same version. This doesn’t always happen, but the installation of a scaffolding ruby 2.7.0 failed when the system ruby was version 2.6.5; if something similar goes wrong for you, a version mismatch is a good thing to check.

The version of rubygems (about which see more below) distributed with ruby 2.7 and later really wants to load the ruby openssl library, which requires OpenSSL (or a fork of it, like LibreSSL).

Ruby 3.2 requires the native libyaml library to build psych, which is needed for the litbuild gem and possibly others.

Sed is kind of like a text editor, but instead of allowing you to modify text files interactively (like ed and vim do), sed acts as a filter: it takes a text file as an input stream, makes changes to that input stream, and produces the resulting modified text as an output stream. (The name, sed, means "Stream EDitor").

This is a fairly powerful and flexible thing to be able to do, but it’s not something people often need to do — like gawk, sed is most often handy in the context of a bash script or something like that, where a more powerful general-purpose language like ruby or python or perl isn’t convenient for whatever reason.

On the other hand, the automated build processes for some of the CBL components use sed, so you have to have it around anyway.

Tar is a program for manipulating archives of files. The name derives from its original purpose, back in the dawn of time, which was to manipulate archives on magnetic tape — ergo, "tar," for Tape ARchive. Actually storing archives on magnetic tape is pretty rare these days, but people still use tar-format files for all kinds of things.

These files are known as "tarfiles," for obvious reasons, or as "tarballs" for less obvious reasons. A speculation I found on Wikipedia is that the term references the tendency of actual tarballs (blobs of petroleum floating in the ocean) to get all sorts of things stuck to them; apparently, someone thought that was reminiscent of how the tar program collects a bunch of files together.

All of the source code distribution packages for CBL packages are compressed tarfiles, so the tar program is really important for CBL!.

The tar package also provides a program called "rmt" that lets you manipulate magnetic tape drives attached to other computers. This is an even more fringe thing to do in this day and age, but It only weighs in at about 50kb, so it’s not hurting anything.

Several of the tests run by the configure script don’t work right when tar is being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

Texinfo is the official documentation system for the GNU project. It uses input files in a standard format to produce a variety of outputs, both printed and online.

Texinfo is used in the automated build process of many of the CBL component packages.

In release 7.1 of texinfo, the install-info command fails in some cases. I think this is because install-info now has code to ensure that the /usr/share/info/dir file is updated atomically: it writes the new version to a temporary location, then moves it over the original file, overwriting it. That doesn’t work on LB Linux, because it uses the package user approach to package management, and /usr/share/info, like other install directories, has the sticky bit set. (If you don’t know what any of that means, you can read the package-user manual — on Little Blue Linux systems, you can find it in /usr/share/doc/package-users.) For some reason I do not understand, many packages are not affected by this, but a few are. As I write this, so far, I’ve come across this with nettle and glibc, but perhaps there are others. Whenever this happens, I use a simple workaround: simply chown the dir file before installing the package, and chown it back afterward.

When cross-compiling, the configure script makes some incorrect guesses about the availability of functions provided by glibc. We can override those guesses in a config.cache file.

Even after doing that, the presence of the gnulib version of getopt.h causes the build to break. That’s easy enough to work around simply by removing it.

The build process creates two scripts that expect perl to be in the location where it’s found on the host system, rather than the place it will be available in the initial target system, so we fix them up after installation.

Back in the early days of Unix, someone wrote a text editor called "ed" (which stood for, get this, "EDitor"). Ed is a line editor, which means that generally it lets you modify a line at a time. Ed is still available, and in fact still actively developed — GNU ed 1.14.1 was released in January of 2017 — but most people want a more interactive editor, and so one of the old proprietary Unixes eventually included an editor called "vi" ("VIsual editor") to fill this need.

Since vi is proprietary, various people implemented similar programs, which they released under open source licenses of various sorts. One of the people who decided to do this is Bram Moolenaar, who created vim; its name stands for "Vi IMproved," because it’s massively superior to vi. Bram Moolenaar died in August of 2023, but there were other contributors to the vim project who continue to develop the program.

Vim is distributed under a different software license than most of the packages that make up CBL: the Vim License. This license is GPL-compatible, but is described as "charityware" — users of vim are urged to make a donation for needy children in Uganda.

As far as that goes, you probably don’t need to install vim at all. There are plenty of other text editors that some people prefer. The default editor in some distributions is nano, and some people like their text editor to be their primary tool for interacting with the computer and use emacs. Whatever your preferences, though, you absolutely do need a text editor of some sort, and vim is my favorite, so that’s the one that winds up in the basic CBL blueprints.

As with many scaffolding pieces, the configure script for vim doesn’t play nicely with cross-compilation, so we pre-fill a config.cache with a bunch of values that it won’t be able to discover on its own.

An interesting difference between vim and most other packages we’re building is that it automatically picks up entries from a config.cache file located in the src/auto directory, and apparently doesn’t support providing a reference to config.cache in the top-level configure run.

There are a lot of programs that expect a text editor called "vi" to exist, so we’ll create a symlink after installing vim.

The XZ Utils package provides a compression/decompression program with an interface similar to gzip, but using the LZMA algorithm. This is very slow when compressing, very fast when decompressing, and compresses more effectively than most of the other common compression programs, so it’s used fairly often when providing archive files for download.

The compression algorithm used by the XZ Utils is the same as that used by lzip, but the file format is more complex and the compressed output it produces is usually slightly larger than that produced by lzip.

Having constructed all the scaffolding we can build on the host system, it’s time to set up for the second (target) stage of the build. The target-side build will be run automatically when the target system is booted.

As mentioned elsewhere, you can add data to the entropy pool without any difficulty from any userspace process: all you have to do is write data to /dev/urandom. However, that doesn’t help to initialize the entropy pool, because it doesn’t increase the kernel’s estimate of how much entropy is actually available. To do that, we have to execute a system call that is only available to processes running as the superuser.

The system call in question is ioctl, which is kind of a catch-all that exposes arbitrary functionality in device drivers. ioctl itself is short for "Input-Output Control." The idea is: since there is a huge variety of input/output devices that might be available to your system, and there’s no way for Linux to provide specific system calls to exercise every distinct function of every one of those devices, any device driver can define a set of ioctl operations. Userspace applications can then use the ioctl system call to invoke any of those operations, using a file descriptor that references one of the device’s special files under /dev.

The device driver for the /dev/random and /dev/urandom special files happens to provide an ioctl that lets you add data to the entropy pool. We’re going to write a tiny C program that feeds some data to the kernel using that ioctl, so that the entropy pool will be fully initialized immediately.

Note that it is an extremely bad idea to use this program unless you are confident that the input data really is unpredictable, or unless (as in the case of the target-side CBL build) you are equally confident that there is no reason to be concerned about the quality of random data available to userspace processes.

In many cases, this program isn’t needed in a full working system — the rngd daemon from the rng-tools package feeds the kernel entropy from hardware random number generators, like the one built into modern x86-architecture CPUs. But CBL supports multiple computer architectures, and some of those don’t have anything suitable for rngd, so addentropy can be helpful in those cases.

The addentropy program is derived from ec2seed, at https://github.com/akkornel/ec2seed, which is available under the terms of the GNU GPL Version 3. Accordingly, there’s no issue with distributing the addentropy source as part of CBL; all the code within CBL is similarly licensed under the GNU GPL Version 3.

As with almost all C programs, we start by including header files.

We define a constant for the number of bytes we’re going to feed to the entropy pool. That’s really not all that important, but it makes some later parts of the program slightly clearer.

Before and after adding data to the entropy pool, we can obtain the kernel’s approximation of how much entropy it already contains, using another ioctl provided by the /dev/random device driver. Let’s define a function for that.

Now we can write the main routine for addentropy. As usual in C programs, we start by defining all of the variables we’ll use.

We need to open a file descriptor on the /dev/urandom special file so that we can invoke ioctl operations on it. And we need to allocate a buffer that we can read the data in to.

According to the kernel source file random.c, which is the source code for the random-number device driver, the entropy_count field of rand_pool_info structures is denominated in units that are an eighth of a bit each. I have no idea why. So maybe we should multiply RANDOM_BYTE_COUNT by 64 rather than by 8 here? But this appears to work perfectly well, so I’m not messing with it.

Now we need to obtain some data (which we will presume to be random). The way we’re going to do that is simply read it from the standard input stream. Since we’re using some low-level file manipulation functions, we have to deal with the possibility that any given call to read actually produces less data than we request; therefore, we set up a loop and read repeatedly until we have all the data we want.

Now we can add the data to the entropy pool, using the RNDADDENTROPY ioctl. We’ll also print out a description of what happened.

There’s nothing left to do other than clean up the resources we’ve allocated and terminate the program. Technically we don’t have to clean up the resources — Linux will do that automatically as the process is reaped — but it’s good practice to do it.

That’s it!

The command we use to compile the program, and the location where it will wind up, depends on whether we’re building this in the scaffolding or for the final target system.

For the scaffolding, we need to use the cross-compiler to compile addentropy, and it needs to wind up in the scaffolding bin directory.

When the kernel runs init, it does so with a minimal (maybe empty?) environment. Bash does not work well unless some basic environment variables are set — like PATH and LD_LIBRARY_PATH for program and library lookups, for example — so we’ll start by getting those set.

When problems occur during the target-side build, the init script bails out to an interactive shell that also has an empty environment. So it’s convenient to have all the commands that set up the environment in a separate script that can be easily sourced in from that rescue shell. The same script can be sourced from init to avoid any duplication between the two.

GNU/Linux systems have extensive support for internationalization and localization. Before we have the full system set up, it’s a good idea to specify a simple default setting for those features.

As the final system programs and libraries are built, they should be used in preference to the ones in /scaffolding, so we’ll put the directories where they’ll live in front of the /scaffolding directories.

The TARGET_SYSTEM_MAKEFLAGS parameter specifies the MAKEFLAGS that should be set for all target-side processes. (If a specific package has issues with parallel builds, it can be overridden to -j1 or unset entirely for those packages.)

There are some programs that expect HOME to be set to a reasonable value. We can go ahead and set it here in case any of the programs we’re using is among those.

We also need to set environment variables for all the litbuild configuration parameters that might not be using default parameter values. It’s not hard to define these properly, because from this point onward we don’t need to worry about HOST and TARGET and so on: everything is just going to be a native build. We do need to pass a number of parameters from the host-side build along through to the target-side build, though!

Many of the parameters used in the litbuild blueprints will have static values defined by convention within CBL. For those, we don’t care about the host-side parameters that were used, we can just set paths as we wish.

Before we can execute the build process, we need to get the target system to a baseline functional state — kernel filesystems are mounted, the root filesystem is writable, and litbuild is available to generate scripts from blueprints. This could be done inline as part of the init script, but I’m writing it to a separate script, startup-system.sh instead. When the CBL process is modified, problems frequently arise, leading to build failures in one place or another, and a technique I often use to resolve those issues is to reboot the target system with the init command-line parameter changed from /scaffolding/sbin/init to /scaffolding/bin/bash — so I wind up at an interactive bash prompt. At that point it’s very convenient to be able to source setenv.sh and then run startup-system.sh to get the target system to a baseline usable state without kicking off the full target build again.

The actual init script will also start by running startup-system.sh, of course.

The commands in startup-system.sh are designed to be idempotent: that is to say, you can run them any number of times and they’ll ensure that the system is in the expected state when they have completed, but won’t make unnecessary changes. This is helpful in cases where the build process does not complete successfully and the target system needs to be restarted.

The first thing to do is remount the root filesystem so we can write to it.

Linux systems have several magical filesystems. Two of these, /sys and /proc, don’t actually contain real files and directories, they just contain things that look like files and directories, but actually provide insight into the state of the kernel and system — and in many cases also provide the ability to manipulate that state — using a file interface. For example, /proc/cmdline shows the command line that was passed to the kernel by the boot loader, and /proc/filesystems shows the set of filesystems that are supported by the kernel.

/dev is different — it’s the canonical location on the system for nodes, aka "special" files. These represent I/O devices and allow those devices to be accessed as though they were files. Linux has a feature called "devtmpfs" that automatically populates a RAM-based filesystem with nodes for all of the I/O devices it knows about.

Additional directories that should have RAM-based filesystems mounted on them are /run and /tmp. /run is a canonical location for files containing volatile system state information — things like the runtime s6 scan directory — and /tmp is the conventional location for temporary files that should not persist across reboots. It’s convenient to use a tmpfs for that purpose. This does limit the amount of data that can be written to /tmp: by default, a tmpfs filesystem is limited to half the physical RAM on the system. For small target systems, this can be a problematic constraint; if you wind up having any issues as a result, you can replace the mount command here with something like rm -rf /tmp; mkdir -m 1777 /tmp to ensure that everything from the previous boot (if any) has been cleared up but without using a (size-limited) tmpfs for it.

Some types of computers — basically all common x86 computers, for example — have batteries that maintain the system clock when the power is turned off. Others, like the Raspberry Pi, do not! Build tools rely on file timestamps to determine what build steps need to be performed and which steps can be skipped, so we need to make sure that the system clock is set to something reasonable before we start building things.

We’ll make sure the init script has a current timestamp now:

And when we run the startup-system.sh script, we’ll set the clock to the time of the init script. This may not be good enough if the build is restarted, and some files have a timestamp later than the init script, but it’s easy. (A more rigorous solution would be to inspect all the files on the root filesystem, and set the clock to the same timestamp as the latest timestamp found there. But that would take substantially longer, so I’m starting with the easy approach and will modify it further as necessary.)

There’s a little kludge we’re going to use to avoid long delays before the target-side build starts, but to explain what we’re doing I’m going to have to digress a few steps — you can skip this whole section if you’re not interested in some abstruse technical details.

Still with me? Brave soul. Okay, the Linux kernel has a built-in feature that lets you obtain cryptographically-strong random numbers. This is not something that computers are naturally good at, because computers are highly deterministic — there are plenty of pseudo-random number generation (PRNG) algorithms, but they all have to be seeded with some initial value and if you don’t have a source for that initial seed value that’s really random, you wind up with very predictable pseudo-random numbers. If you’re doing something involving cryptography, that’s no good! When a cryptographic algorithm calls for random bytes — in this context, sometimes this is called "entropy," although I remember hearing at some point that the terms aren’t strictly synonymous — the values of those bytes really need to be unpredictable, or else the strength of the algorithm collapses entirely.

Linux makes up for this by collecting some bits of entropy from unpredictable events. The most easily-observed of these events are human inputs, like keystrokes and mouse movements; by measuring the time between keystrokes and taking the least significant few bits from each interval, or measuring the interval between receiving network packets or some other kinds of hardware interrupts or other things like that, Linux can gather a fair amount of really random data. It mixes these random bits into an "entropy pool" it maintains. (You can read all about this in drivers/char/random.c in the Linux source tree.)

There are two easy ways that userspace processes can request random data from the entropy pool: they can read it out of the character device /dev/random, which only provides as much random data as Linux estimates to be available and blocks after that’s exhausted, and /dev/urandom, which is basically a PRNG that is periodically re-initialized with a really-random seed value — good enough for most purposes, and it never blocks. That latter source, /dev/urandom, is what the SecureRandom class in the Ruby standard library uses to initialize its PRNG; it’s also used by lots of other programs that need random data.

Before Linux 4.16, that was the end of the story: there was /dev/random, which you would use if you wanted completely random bytes for things like cryptographic key generation and one-time pads; and /dev/urandom, which was perfect for getting more-or-less random data without any delays. In the 4.16 release series, though, this behavior changed slightly in order to address a security concern: now, if any process tries to read from /dev/urandom before the kernel’s PRNG has been initialized with a really-random seed value, the read blocks until that initialization is complete.

This is still fine in most circumstances! But the CBL target-side build is unusual: it’s intended to be completely automated, and it’s not on a network because that would make things more complex than they need to be. So there aren’t any keystrokes or network packets to measure timings from! That means that initializing the kernel’s PRNG can take fifteen minutes or longer after booting the target system. It also turns out that some of the programs used in the target build — for example, the litbuild program used to generate the scripts for that build — wind up reading from one of the random devices, which causes the target-side build to hang — sometimes for quite a while — shortly after it starts.

You can feed entropy to the pool just by writing data to /dev/urandom (or, I think, /dev/random), but the kernel doesn’t trust that any data you write that way is really random, so this doesn’t help unblock the build. There’s also a system call available to privileged processes (that is, processes that are running as root) that adds random data to the pool and assures the kernel that it really is random, so we can avoid unnecessary delays in the build by using a tiny C program to invoke that system call.

Of course, at this point we don’t have any reliable source of random data to use with that program — that’s the whole problem! But, luckily, we don’t actually need one. We’re not doing anything in the CBL build where the lack of cryptographically-strong random data is a problem. So we’re just going to lie to the kernel: we’ll write some non-random data to the entropy pool and assert that it is random.

Specifically, we’re going to pretend that the program code for the addentropy program is random, even though it’s really not at all!

We need to create the rest of the basic sytem directory structure. This doesn’t have to happen right now, but this is as good a time as any.

If the build process was interrupted and restarted, the directory structure and symbolic links will already exist, so we can skip this stuff. The same kind of logic applies in many later parts of this script, so there will be additional guard if statements around those blocks.

The directory structure for GNU/Linux systems is pretty well standardized — the Linux Foundation maintains the "Filesystem Hierarchy Standard" document,^[11] and I generally like the conventions laid out by that document, so the directory structure created here for Little Blue Linux is mostly consistent with those guidelines.

There’s one significant exception, though: the FHS mandates having all of the programs and libraries needed to boot the system in top-level directories /bin, /lib, and /sbin. Other programs and libraries that make up the operating system are in similar directories, but under /usr — that is, /usr/bin, /usr/lib, and /usr/sbin. A lot of GNU/Linux distributions have stopped doing this, because there are benefits to having all of the read-only files that make up the operating system in one place, and the /usr directory structure is a good place for that. That’s how we set up Little Blue Linux, as well: the top-level directories exist, but they are symbolic links to the actual directories under /usr.

As promised earlier, we need to create symbolic links so that packages that install files in /bin, /lib, and /sbin are able to do that — the files will wind up in the corresponding locations in the /usr hierarchy.

Also, all the multilib directories that the GNU toolchain components sometimes insist on using should be symbolic links to /usr/lib. This is still a kludge, but it’s the only way to avoid needing to specify some arbitrary set of lib directories in ld.so.conf.

If you’re building for an architecture that uses different multilib directory names, you might need to create additional symbolic links here. If you do that, you’ll probably also need to make a corresponding tweak in Create Symbolic Links For Scaffolding Lib Directories; also, look at the specs file when it’s being adjusted to see whether the multilib paths are present in the linking specs. If they are, you may need to make changes there as well.

Conventionally, information about filesystems that are currently mounted is available in the file /etc/mtab, and some packages therefore expect /etc/mtab to contain this information. The historical convention is that the mount and umount programs, which attach and detach filesystems from the filesystem hierarchy, would also add and remove corresponding lines from the mtab file.

The mtab file isn’t actually needed any more, though, because the /proc filesystem contains a file that always contains the kernel’s view of what filesystems are mounted. We can create a symbolic link to the conventional location, for the use of any packages that look for it there.

The mount and umount programs do still rely on a configuration file at a standard location for filesystem information: /etc/fstab. This file contains information on what filesystems are part of the system, and where they should be mounted. As is generally the case, if you want to know all about fstab, you can read the manual page with man 5 fstab. On standard Little Blue Linux systems, we the root filesystem is an ext4 filesystem with the label lblroot, so we should write an entry for it. Also, the /tmp directory should be cleaned out automatically at boot time, and an easy way to do that is simply to mount a tmpfs file system there.

Users are defined in the file /etc/passwd. This file can also contain hashed versions of users' passwords, as well as other user account metadata — hence its name — but usually it does not! This is because passwd has to be readable by everyone (it’s not really important why this is the case, so I won’t get into that here) and shortly after UNIX systems became popular it became clear that it was a bad idea to allow all users to see even the hashed version of passwords.^[12]

So in most UNIX operating systems, the /etc/passwd file (confusingly) does not contain even the hashed version of passwords; those are found in a file called /etc/shadow, which can only be accessed by root. The extraction of passwords into /etc/shadow, and maintenance of them there, is done by the shadow package, which we’ll set up early in the target-side build.

These files have colon-delimited fields and newline-delimited records, so they’re pretty easy to read; man 5 passwd describes the file format. The second field is for the hashed password itself. If the field is blank, that account requires no password to log in; if it contains a single "x," that indicates that the password is stored in /etc/shadow as described a moment ago.

Users can belong to groups. Group definitions are similar to user definitions, but are in the file /etc/group — and, similarly to the passwd and shadow files, the password for groups that require passwords are usually found in /etc/gshadow.^[13]

Initially we’re just going to create the root user and group, and a few other system users and groups that are expected to exist by various programs.

There are some programs that write log data — like process resource usage data — to specific files, but only if those files already exist. Let’s create them.

The root user should have shell startup files just as other users do, and there’s no more-convenient place to set those up. Generally for the root user I like avoiding any significant shell startup activities, I only set environment variables, and I like having the same set of environment variables regardless of whether I’m using a login shell (for which the .bash_profile script is sourced) or a non-login shell (for which .bashrc is sourced), so I link those together.

More typically, .bash_profile can be used for any commands that should only be run at login time — like starting an ssh-agent process, for example — and .bashrc can be used for commands that should be run any time a new subshell is executed. (It’s common for .bash_profile to source in .bashrc as well, but this is by no means necessary.).

In addition to typical environment variables like PATH, we again want root’s environment to include all of the configuration parameters that will be used when running litbuild on the final system. Some of these, like PATCH_DIR and TARFILE_DIR, will be useful after the build is complete, so those will be written to .bash_profile. Others won’t, so we’ll write those to a .cblrc file that gets sourced in from .bashrc — once the build is complete, we can remove the command that sources that file from .bash_profile.

The code that sets these environment variables digs a little bit deeper into the bash bag of tricks than usual; the idea is, for each of the environment variables we want to set, we’ll echo both the name of the variable and the result of expanding the name of the variable as an environment variable into the .bash_profile script — an indirect reference. To do this we need to escape the first $, because otherwise bash expands $$ into its own process ID, which is not helpful.

I didn’t actually figure out how to do indirect references in bash; I did a something search and found everything I needed in chapter 28 of the "Advanced Bash-Scripting Guide," which is part of the Linux Documentation Project.

The Linux kernel can write pages of RAM to storage devices like disk partitions or files in the filesystem, which frees up those pages to be allocated to other programs; this is called "swapping", and the blocks of storage allocated to this purpose are referred to as "virtual memory" because they work like (very slow) RAM, or "swap space" because it’s space allocated to swapping. The terms are pretty much interchangeable.

It’s always a good idea to have some swap space available. That way, allocated memory that hasn’t been used in a long time can be written out to the swap area. If any process needs to access those pages, the kernel can re-read them from swap, so the only drawback is that access to those pages of memory takes longer than if they were already in RAM.

A configuration parameter, TARGET_SWAP_DEVICE, can be used to specify a block device to be used as virtual memory. If it exists and isn’t already in use, the following code will set it up. If the device exists but is something other than swap space, it will be ignored — just in case the parameter has an incorrect value and refers to an important filesystem or something.

CBL systems use s6 and s6-rc to manage the system state. We can set up the very first parts of that here — that is, the source directory that will contain service definitions, and the run-level bundle directories inside it. As other parts of the system are built, directories for other services will be created and, in many cases, added to the rl-default bundle.

We also set up an network-services bundle that will be part of the rl-default bundle if networking and network services should be part of the standard system run state.

This is all explained in much more detail in later sections: s6-rc, Configure the system initialization framework, and Construct the s6-rc service database.