Here’s a summary of the basics, for anyone who doesn’t need all the grueling details:

This all matters for CBL because the build process for some of the necessary packages will result in programs that can’t find their shared libraries unless we use an RPATH or RUNPATH; and the build process for other packages sets an incorrect RPATH or RUNPATH that can cause problems unless we remove it.

Usually, ld isn’t executed directly by build processes, but is instead invoked by the gcc driver program. To get gcc to pass an option along to ld, you can give it the option -Wl, which should be followed by additional words separated by commas; gcc will replace the commas with spaces and pass the resulting arguments on to ld. So to set an RPATH of /some/dir and /another/dir, you can give gcc the argument -Wl,-rpath,/some/dir:/another/dir.

An alternative that might work is to set the environment variable LD_RUN_PATH to the desired RPATH before linking — the ld documentation suggests this will work, but I haven’t tried it.

This might make your eyes glaze over a bit.

A lot of this discussion pertains only to programs in the Executable and Linkable Format, commonly abbreviated as ELF. This is the only program format used in modern GNU/Linux systems, but you should be aware that there are other executable program formats, like a.out and COFF, and a lot of the details here don’t apply to those formats.

ELF programs consist of an ELF header followed by some number of segments of various types. If you’d like to examine the full structure of an ELF program, you can use the program readelf, which is part of the GNU binutils package.

When a program is executed, the Linux kernel looks in its .interp section to find the path for its interpreter — in our case, this will be the full path of ld.so (the dynamic linker). That interpreter then looks in dynamic segments of the program for the names of shared library files that it needs, and tries to find them. If any shared library can’t be found, the dynamic linker will terminate with an error.

There’s a bit of additional complexity: shared library files are also in ELF format and can also declare that they need shared libraries. This can result in a chain of library dependencies, potentially a lengthy one! When the dynamic linker is trying to find a library, its behavior is partly determined by which object is having its references resolved: the original program, or one of the libraries it depends on, or one of their libraries, and so on. Whichever ELF file is having its references resolved is called the "loading object" here.

The rest of this section describes the procedure used by the dynamic linker to locate shared library files.

Shared library names can contain slash characters, although they usually do not. (I don’t even know how to get ld to create a program with library dependencies that specify a full path.) If a shared library name does contain any slash characters, it is treated as a relative or absolute path and the dynamic linker will only look for the library at that path.

In the common case, when a shared library is specified without any slash characters, the dynamic linker looks for it in a variety of locations. The algorithm it uses is poorly documented and there’s a lot of contradictory information about it on the web.^[8] After looking around for quite a while, I found a helpful blog post on qt.io that pointed me to the _dl_map_object function in the glibc file elf/dl-load.c, and a review of that code revealed the following algorithm:

As soon as the shared library is found in any of those locations, that’s the file that will be used. If it’s not found in any of those locations, the dynamic linker will give up and crash with an error message. And, in case this was not obvious, in all the things with a name that contains PATH — LD_LIBRARY_PATH, RPATH, RUNPATH — you can specify any number of directories separated by colons, just like the PATH environment variable.

A historical note: RPATH has been around longer than RUNPATH. RUNPATH was implemented, along with all the logic about skipping RPATH when a RUNPATH is present, because people realized that when someone specifies an LD_LIBRARY_PATH it’s usually because they really do know what they want the dynamic linker to do, and it’s rude to override that desire at program compilation time.

When you give ld the -rpath option by itself, it just creates an RPATH in the resulting program or library. When you add the option --enable-new-dtags, it still creates an RPATH (in case you run the program with an old dynamic linker that doesn’t understand RUNPATH), but it also creates a RUNPATH so that modern dynamic linkers will ignore the RPATH and use LD_LIBRARY_PATH by preference.

As a reward for reading this far, here’s one last option you can use if none of the above suits your needs: before it does anything else, the dynamic linker loads any .o object files, or .so or .a library files, that are named in the environment variable LD_PRELOAD or in the /etc/ld.so.preload file. Any functions defined in those files are used in preference to any other function definitions found later in the process. That lets you override individual function definitions, if you want to override some part of the program without replacing an entire library with a different version.

The GNU binary utilities package contains a plethora of programs and libraries that can be used to produce, manipulate, and otherwise operate on (compiled and assembled) object files. I’ll briefly describe them all here, but don’t worry! Not only will there not be a test on any of this, you usually won’t even need to invoke any of these programs manually. The gcc driver program will invoke as and ld as necessary to do its work, and several of the other utility programs are similarly used internally during the build process of other system components but you won’t need to use them yourself.

The most important binutils programs are as, the assembler, which transforms assembly source code (.s files) into binary or "object" code (.o files); and ld, the loader or link editor (ld is usually just called the linker, but it’s hard to see why it’s called ld without knowing the other terms), which combines multiple object files into an executable program.

Starting with release 2.44, there are two separate distribution tar archives available: binutils and binutils-with-gold. In addition to the traditional ld linker, which uses the binary file descriptor (BFD) library, the "with gold" distribution also includes an additional linker called "gold." Gold doesn’t use BFD and only works for binaries that are in the "Executable and Linkable Format," aka "ELF". Since that’s the only binary format commonly used in LB Linux, it might be preferable to use gold. I’m not actually sure what the benefits are. The BFD linker is always available as ld.bfd, and the gold linker is available as ld.gold if it is present at all; additionally, one or the other of those programs has a hard link so it can also be executed simply as ld.

The other programs in this package are:

There are a couple of shared libraries used by those programs and available to others, as well: libbfd and libopcodes. All of the utility programs are documented much more thoroughly in man pages and the binutils info file.

As I mentioned earlier, you don’t need to run any of those programs directly because the gcc driver program streamlines the simple case of building executables — to compile a "Hello, World" program, you just need to run gcc hello.c, and let the gcc driver program run as to assemble compiled source into object files, ld to link multiple object files together to produce an executable, and maybe other programs if it needs to for some reason.

The downside of using a driver program is that it can make complex builds (when, for example, specific options need to be passed to the assembler, compiler, and linker) a lot more complicated and fussy — as you’ll see, from time to time, during the CBL process.

The combined-source-repository situation described a moment ago also complicates the construction of branch-update patches a little bit: we always produce branch-update patches by running git diff commands against the source repository, and sometimes the resulting patch for binutils includes changes to files that are not part of the binutils distribution — which means they don’t apply cleanly. We adjust for this by applying the patch, fixing any issues we find as a result, and then using the corrected patch here.

Binutils, like some other parts of the toolchain, should be built in a separate directory from the source code. As with other components that we build several times, CBL puts each distinct build in a separate location, so that each build is entirely independent of, and unaffected by, the other builds.

We build a cross-binutils using the "sysroot" framework (which you’ll read more about shortly). That framework isn’t particularly well-documented, but the important thing at this point is that we need to specify the configure options --with-sysroot and --with-build-sysroot to inform the build machinery of the desired sysroot directory.

All of the toolchain components should be installed in the same filesystem location. The CROSSTOOLS parameter lets you specify that location — in this case, /home/lbl/work/crosstools.

Some of the warning messages present in GCC 8 and later present problems when compiling the binutils. We can tweak the generated Makefiles so those warnings won’t be converted to errors.

GMP, the GNU Multi-Precision arithmetic library, is a component in — or perhaps it is more properly called a dependency of — the GNU toolchain. It allows arithmetic operations to be performed with levels of precision other than the standard integer and floating-point types. Applications can use GMP to provide arithmetic with thousands or millions of digits of precision if that’s what they need. GMP also provides support for rational-number arithmetic, as well as integer and floating-point.

If you’re really interested in high-precision floating-point arithmetic, you might want to look into MPFR rather than GMP! The GMP people say it’s much more complete.

GMP has been needed by the Fortran GCC front-end for some time, but starting with release 4.3.0 of GCC it is needed for C (and C++) as well.

MPC, another dependency of GCC, requires a GMP built with C++ support, so we need to specify that at configure time.

This package is often built in-tree as part of GCC, rather than separately — that’s especially true when the only reason you’re building GMP is because GCC requires it. When using an in-tree build, this blueprint is pretty much irrelevant. However, as of 2015-09-27, there’s an issue with in-tree builds of GMP in some cross-toolchain builds, so for CBL we build it separately.

This is not necessary, because the system or host-prerequisites version of GMP can be used just fine by the cross-compiler. If things break down here and you haven’t built the Trustworthy Host-System Programs, you should probably do that.

MPFR is a library for arbitrary-precision floating-point arithmetic. It stands for "Multiple-Precision Floating-point Rounding," I think, but that’s not really clear from their site so I might be wrong. It uses GMP internally, but provides any level of precision (including very small precision) and provides the four rounding modes from the IEEE 754-1985 standard.

Like GMP, MPFR is a component in, or dependency of, the GNU toolchain. It has been needed by the Fortran GCC front-end for some time, but starting with release 4.3.0 of GCC, MPFR is needed for C and C++ as well. GCC uses MPFR to pre-calculate the result of some mathematical functions when those functions have constant arguments, and produces the same results regardless of the math library or floating point engine used on the runtime system. This occurs in what is called the GCC "middle-end," which is kind of a silly name since it’s not an end.

This package is often built in-tree as part of GCC, rather than separately. However, as of September 2015, in-tree builds of the dependencies have some issues in certain circumstances, so in CBL we step away from the in-tree build facility altogether.

As with GMP, this is not necessary because the system or host-prerequisites version of MPFR can be used.

MPC is a C library for arbitrary-precision arithmetic on complex numbers providing correct rounding. It can be thought of an extension to the MPFR library.

Like GMP and MPFR, MPC is a component in, or dependency of, the GNU toolchain. I haven’t been able to find any description of what features of MPC are actually needed by the GNU toolchain, but MPC is a hard build-time dependency of GCC.

If your system already has MPC installed, this step can be skipped.

Like the other GCC library dependencies, this package is often built in-tree as part of GCC. As mentioned earlier, though, this sometimes introduces problems, so CBL doesn’t take advantage of the in-tree build machinery provided by GCC.

As with GMP, this is not necessary if there is a system or host-prerequisites version of MPC available.

The project homepage describes ISL as a library for manipulating sets and relations of integer points bounded by linear constraints. I’m not sure what that means. It sounds like math.

Unless you have some specific reason to want ISL for one of your own projects, the main benefit it provides is as an optional dependency of GCC: the "graphite loop optimizations" in GCC (whatever those are) require ISL to be available.

As with GMP, this is not necessary if there is a system or host-prerequisites version of ISL available.

The kernel is Linux per se — the foundation of the operating system. Often, when people say "Linux," they mean the entire operating system that lets them use a computer; properly, though, Linux is just the kernel. Many — perhaps most — of the other components that make up the full operating system are pieces of the Free Software Foundation’s GNU project, which stands for "GNU’s Not UNIX"; almost all of the program-construction tools that form the foundation of the system are GNU components.

The job of the kernel — this is a critically important point and I repeat it a lot — is to initialize and manage all of the hardware on the computer (including CPUs, memory, and I/O devices) and provide services to userspace processes. That’s a big job, but it’s also a limited one! Everything you do with your computer is the responsibility of userspace processes.

The way that Linux performs its job — more generally, the way that UNIX kernels work — is conceptually very simple: when it is executed on a computer, Linux does some hardware initialization, mounts the root filesystem, and then starts a single userspace process. That process has process ID (PID) 1, and is conventionally called init.

The init process is responsible for starting all other userspace programs and getting the machine into a usable state; after the kernel starts init, it gets out of the way and waits for that process, or for other userspace programs, to request its services.

The complexity in the Linux kernel emerges mostly from the huge variety of hardware that it supports and the many layers of functionality that can be built into it. If you unpack the Linux source code, you’ll find that the whole thing adds up to about (as of the 6.6 kernel) 1477 megabytes in total. Of that, almost ten percent (144 megabytes) is specific to the twenty-six different CPU architectures that Linux supports, and about two-thirds (971 megabytes) is device drivers. That means that for any given system, a large majority of the code that makes up the Linux kernel won’t ever be used.

In CBL, the kernel is kept as pristine as possible. In most cases, the only patches we apply to it are intended to add new default configuration settings — which is more convenient than setting dozens of configuration options manually.

The way that programs make requests of the kernel is by invoking kernel functions known as "system calls." The Linux kernel sources include header files that define all of the system calls it makes available to userspace programs.

Userspace programs don’t usually invoke system calls themselves (although they can, and some do). Instead, they invoke library functions — particularly the ones defined in the C standard library, which on GNU/Linux systems is usually the GNU libc, glibc, but might be musl or uClibc or something else altogether. Those library functions invoke system calls as needed to accomplish their work.

In this step, we’re not actually building the Linux kernel; all we’re doing is installing the header files that specify those system calls. These header files are then used by the C library and other userspace libraries and programs to invoke system calls as needed.

The Linux source tree also includes a number of other, private, header files — these define data structures and functions that should be used only within the kernel itself, and are not intended to be visible to userspace programs. The Makefile target we use here, headers_install, installs only the public header files, not these private internal headers.

The makefile target mrproper puts the source tree into a completely pristine state. The name is a reference to the Proctor & Gamble cleaning product that people in the United States know as "Mr. Clean"; in many parts of Europe, it is marketed under the brand name "Mr. Proper."

The install_headers target deletes the entire target directory before it does the installation. This is inconvenient, because we might have header files there that we want to retain (such as from the binutils build). To avoid any issues, we’re going to install the headers to a temporary location and copy them to the real target location from there.

Notice that we’re actually installing the headers into the scaffolding directory underneath the sysroot directory. That’s common to the entire host-side portion of CBL: Everything that will be conveyed to the target system is under the scaffolding location. That way, when the final system is completed, we can get rid of /scaffolding and be fairly confident that everything remaining has been built entirely from source and (if it continues to work) has no dependencies on the scaffolding programs and libraries.

GCC is the GNU Compiler Collection. This is the single most important package in CBL, even including the Linux kernel itself: without a compiler, you can’t build any software. We use GCC to bootstrap everything, including itself.

Unfortunately, GCC is also probably the most complex package we need to build, and it has a very complex configuration and build process because it is tied so intricately to other packages. This is particularly true of glibc, the C standard library we use in CBL, which also has a complex build process of its own!

On the plus side, if you can get GCC built and working properly, you’re past the biggest hurdle of building a complete GNU/Linux system entirely from source.

The GCC installation process is documented in the "Installation" manual, found in the source distribution in the INSTALL directory. If you have problems getting GCC built, that’s an excellent resource for figuring out what’s going wrong. If you want to get a better understanding of the GCC configuration and build process — how it actually works — read the "Source Tree Structure and Build System" section of the GCC Internals document, found in the source distribution in gcc/doc/gccint.info.

The most important compilers in the collection, for purposes of bootstrapping a system, are the C and C++ compilers; but GCC also includes compilers for D, Fortran, Go, Ada, Objective-C, and probably other languages as well. And it supports a huge number of machine architectures! Since CBL relies absolutely on having support for at least two types of computer for every build, that support for many architectures is probably the most important aspect of GCC for our purposes.

The program you usually invoke to build C programs, gcc, is not actually a compiler. It’s just a driver that knows how to invoke other programs, using rules called "spec strings" that tell it exactly what other programs it needs to invoke, and what command-line arguments it should provide, to turn source code into an executable program. (We’ll talk more specifically about spec strings in Adjusting the GCC specs.)

I find it really helpful to keep that in mind throughout toolchain construction, so I’ll expand on that point: gcc just invokes other programs. Some of those programs — the C preprocessor cpp, cc1 (the C compiler itself), and internal utility programs like collect2 (which is sort of a first-stage linker, used to set up calls to constructors and other initialization routines as a program starts to run) — are part of the GCC package. Others, like the as and ld programs, are distributed separately (in the case of as and ld, that package is GNU binutils).

To get from source code to an executable C program, gcc actually performs a series of steps:

You can find out exactly what commands gcc is running by giving it the -v (for "verbose") command line argument. That’s a handy trick when things are going wrong and you’re not sure why!

The term "compilation" — which is the job of the compiler — actually refers only to the second of those steps: source code is compiled into assembly code. That means it’s not precisely correct to say that you’re "compiling" source code into an executable program! That’s a common conversational shorthand, but it masks the complete story about what’s going on. It would be more precise to say that you’re "pre-processing, compiling, assembling, and linking" source code to produce an executable program.

On the other hand, that’s a lot of words, and it’s seldom really important to keep the complete story in mind. This is probably why the shorthand term is so popular.

Here, I’m focusing mainly on the C programming language. C and C++ are very closely related, to the point that the earliest C++ compiler didn’t produce binary object code at all; it compiled C++ programs into C source code, which was then compiled into binary code by a C compiler. A lot of the details about how C works apply exactly the same in C++; the differences are mostly things like the naming convention used for source files^[10] and the program used to invoke the compiler (g++ rather than gcc).

I’m not discussing other languages, like Objective-C or Fortran, at all, because I don’t know anything about how they work!

C source code files generally have the extension .c. Along with these, you’ll often find header files, typically with the extension .h; these contain function declarations, macro definitions, constant definitions, and other things like that. These files can contain preprocessor directives, which are evaluated and executed by the C preprocessor before the code is compiled. Commonly, C source files contain #include directives that cause the preprocessor to copy the content of header files into the C file as part of this evaluation.

In addition to the header files that are included as part of the source code for C packages, UNIX systems have system header files that can be included by any program that needs them. These files can typically be found in the /usr/include directory and its subdirectories.

In addition to executable programs, packages can install libraries. These are basically collections of related functions that can then be used in other programs. These libraries are commonly installed in the /lib and /usr/lib directories (and their subdirectories). Sometimes, libraries are installed into variants of those directories, which can be set up on systems that support more than one kind of CPU instructions. For example, the AMD64 CPUs that are used in modern Intel and AMD systems support both 64-bit and 32-bit instruction sets, and GCC can produce both 64-bit and 32-bit object code. If you have one of these computer systems, you may want to have both 64-bit and 32-bit programs. In that cases, 64-bit programs will need to be linked against 64-bit versions of libraries, and 32-bit programs will have to be linked against 32-bit versions, but both versions of the libraries will have the same filename. To allow these "multilib" systems, libraries will be installed in directories like /usr/lib32 and /usr/lib64 rather than simply /usr/lib.

In most cases, when a package installs a library in any of those directories, it also installs header files (into /usr/include) that specify how to call the functions in that library.

There are two different kinds of library files. Both are used when a program (or sometimes another library) are linked. Static libraries have the extension .a; when a program is linked against a static library, the binary code for methods invoked by that program is actually copied from the library into the program. Shared libraries, with the file extension .so, are the other kind. When a program is linked against a shared library, a reference to the library is written into the program, but the code for library functions is not. The benefit of this is that, no matter how many programs you have that use a common function like memcpy or printf, only one copy of the function code needs to be present on the system, in the shared library that defines them. Programs with such library references are called "dynamically linked," and when they are run, the program interpreter is responsible for tracking down the shared libraries it uses and finding the code for functions within them. We’ll talk more about how it does that later on!

One library present in the Little Blue Linux system is simply called "the C standard library." The functions in that library are defined as a part of the C language, and almost every C program uses some of those functions. (For example, the "Hello, World" program that is traditionally the first C program written by those learning the language uses the printf function defined in the standard library.)

This package does not include an implementation of the standard library.^[11] In the GNU system (and in Little Blue Linux), the standard library is provided by the glibc package — which also provides the program interpreter that resolves shared library references in dynamically-linked programs. Although glibc includes both static and shared versions of libraries, it does not really work well with statically-linked programs. So the vast majority of executable programs found on GNU/Linux systems are dynamically linked.

After being compiled and linked, executable C programs are typically installed into directories like /bin, /sbin, /usr/bin, and /usr/sbin. The same directories also contain all of the other programs that are part of the system, regardless of whether their source code is written in C.

Many of these topics are discussed in more detail elsewhere in CBL. The others are discussed in more detail somewhere, but you might have to do some searching! In any case, that’s a basic summary of how C programs work on LBL — and other GNU/Linux — systems.

A few external dependencies were added to gcc in release 4.3: the GMP, MPFR, and MPC libraries are required for all compiler builds, and a few optimizations — the graphite loop optimizations — are only available if the Integer Set Library (ISL) is available. The graphite loop optimizations are not critically important, but there’s no reason not to include them if it’s not difficult to do.

Sources for the required — and optional — libraries can be included in the GCC sources in directories named gmp, isl, mpc, and mpfr; if they are, then they’ll be built automatically along with gcc. There’s actually a fairly large number of packages that the build machinery for GCC detects and incorporates into builds automatically. This is convenient, but restrictive: in-tree builds are only reliable when specific versions of the dependency libraries are present, and for CBL I prefer to use the latest stable release of everything.

There are also some cross-compilation scenarios in which the in-tree library builds actually do the wrong thing and produce a compiler that will not work right; that’s another reason that we avoid them here.

There’s a problem that was introduced in GCC 6.1 involving hard-coded paths to C++ header files. This prevents them from being found by the scaffolding compiler (which lives at a different filesystem location when the target system is booted). We can apply a patch to work around that.

When specifying locations for the dependency libraries (GMP, MPFR, etc), the specified library location should be used both at build time (with an -L linker directive) and at run time (with an -rpath directive). This is not always done by the normal GCC build process, but we can patch it easily to add that behavior.

Even though the math libraries needed by GCC (GMP, ISL, MPC, and MPFR) have just been built here and GCC is configured to find them in their correct locations, some of the programs that make up GCC are built without an RPATH or RUNPATH, so the dynamic loader will look for those libraries at runtime in the normal host system library directories. This was filed as GCC bug 84153, but as of the last time I looked at it, the GCC maintainers don’t consider it a problem.

To be fair, this really is not usually much of a problem! The issue appears when the version of the library used in CBL is a major version later than the version installed on the host system. In that case, the GCC we’re building in this section won’t work because it won’t be able to find the version of the libraries it depends on.

This is the sort of situation that LD_LIBRARY_PATH is intended to resolve, but there’s a gotcha: in some of the host-scaffolding builds, the native toolchain is used to build some programs that are run as part of the build process itself, and the LD_LIBRARY_PATH set in the environment for the cross-build is unset when building those native programs.

That means that to get the this native gcc to be reliable, we need to set a RUNPATH or RPATH to tell the dynamic loader where to look for shared libraries. The way you normally do this is to set an LDFLAGS environment variable with a value like -Wl,-rpath,/usr/lib when running the configure script, so that ld would be told to build programs with that RPATH, but it turns out the GCC build doesn’t use the LDFLAGS linker arguments when constructing some of the programs that need the dependency libraries, like cc1, cc1plus, and lto1.

We can work around that by patching the configure script so that any time it’s given arguments like --with-gmp or --with-gmp-lib, the flags it collects will include a -Wl,-rpath option along with the -L option. That’s what this patch does.

Bootstrapping GCC as part of a cross-toolchain is tricky. You can build the compiler per se without a working standard library (libc), but that compiler won’t be able to produce executable programs: GCC can only create programs if it has access to a set of C runtime object files that it expects to be provided by the C library — and, realistically, it also needs a C library to compile programs because essentially all C programs invoke standard library functions.

Obviously, we can’t compile the C library without a compiler! So there’s a chicken-and-egg bootstrapping problem.

To work around that cycle of dependencies, we’re going to start by building just the parts of the compiler we really need at this point: a plain cross-compiler, and a minimal version of the libgcc support library that the compiler needs in order to function.^[12]

GCC can most easily be built as part of a cross-toolchain by using the "sysroot" framework — and, in fact, the GCC developers don’t support any other method for creating cross-toolchains. To perform a sysroot build, you specify the configure options --with-sysroot and --with-build-sysroot; and when building GCC, set the environment variables LDFLAGS_FOR_TARGET and CPPFLAGS_FOR_TARGET to specify --sysroot as well.

…​At least, that’s what Carlos O’Donell said in a comment on GCC bug 35532, and it seems to work. The documentation on sysroot builds is not particularly easy to find — or at least, it wasn’t when this was written. (If you know where sysroot builds are documented, please tell me!)

The sysroot concept is pretty clever, in fact. The basic idea is, when you build a cross-toolchain, you give it a local directory path — a directory that exists on the build system — and tell the cross-toolchain that that local directory will eventually become the root filesystem directory on the target system. The cross-toolchain then knows to look for system header files and shared library files and so on in the sysroot location.

CBL uses the standard sysroot approach, in a slightly-nonstandard way: everything is installed not into the sysroot directory per se, but into a subdirectory of the sysroot called /scaffolding. That way, when we finally get booted into the target system, the root filesystem will be empty, except for the /scaffolding directory: everything we create after that, and install outside of /scaffolding, will become part of the final system. The /scaffolding directory itself, and everything in it, will then be deleted.

Depending on the target, GCC has a variety of options that control how it operates. Generally, these can all be specified with command-line arguments beginning with -m. For many of these options, default values can also be specified when GCC is being configured.

For example, for many targets, GCC can build programs with a variety of application binary interfaces (ABIs) — this is the machine-code-level interface between programs, libraries, and the operating system; it defines things like the way that registers are used when invoking functions.

An example of a CPU that supports multiple ABIs is the 64-bit x86 architecture, which is called x86_64 or amd64. As mentioned earlier, these processors can run programs in 64 bit mode (with 64-bit pointers and all AMD64 processor features enabled), 32 bit mode (32-bit pointers and only i686 processor features enabled — in this mode, fewer CPU registers are available to programs, for example), or a hybrid "x32" mode (32-bit pointers but all AMD64 processor features enabled). You can specify the ABI to which GCC will compile by using an -m32, -m64, or -mx32 command-line argument.

You can also override the default ABI when configuring GCC by specifying a --with-abi configure directive; or, for some target architectures, other options, like --with-multilib-list or --enable-targets or probably a combination of those things. The options available, and how they work, evolved organically over time, so the situation gets pretty confusing. For example, the way you tell an x86 GCC to generate code for the 64-bit ABI is to use the configuration directive --with-abi. There is also a runtime command line option -mabi, but it doesn’t override that selection; it only selects between the sysv calling convention used by UNIX-ish systems and the ms convention used by Microsoft Windows.

This confusion is characteristic of the GCC configuration and usage options: a lot of the options that are available, and their meaning, depends on what architecture is targeted by the compiler, and the same options or terms can mean very different things for different targets.

If you’re trying to figure out what options are available for a specific target architecture, you have a couple of options. The installation manual lists some of them, mostly in the "configuration" section. The GCC manual also has information in section 3.18 ("Machine-Dependent Options" — if you’ve built a set of trusted host tools specifically for the CBL build, the correct info file can be found at /usr/share/info/gcc.info). You can also look at the configuration script, gcc/config.gcc, to see what options are accepted for each type of target. And, finally, after building GCC for the target architecture, you can run gcc --help=target to see what options are available and what values they can have; and you can also find the actual compiler, cc1 (for C) or cc1plus (for C++) under the libexec/gcc/x86_64-cbl-linux-gnu/$VERSION directory, and run it with the --help command-line argument to see what options are enabled and disabled.

In addition to the selection of ABI, for many target types GCC can be instructed to optimize for specific CPUs or CPU families. The command-line arguments that control this behavior, like the ABI arguments, are target-specific — look for -mtune, -march, -mcpu, and things like that.

In CBL, any set of these configure directives can be specified for the cross-toolchain in the TARGET_GCC_CONFIG parameter. It’s generally a good idea to specify default values for all of the options supported by the target platform. (If you don’t want to set any options at all, you can set the parameter to an empty string.)

GCC is generally built with a large number of libraries included. Some of those fail in some circumstances — for example, x86 CPUs can’t build the libquadmath library when the C library being used is uClibc (or, at least, that was the case some time ago, the last time I tried); and the libsanitizer library fails to build when compiling for 64-bit Sparc machines. The easiest way to work around those problems at this stage is simply to disable those libraries, and, considering that the cross-toolchain we’re building here is just going to be used to build the ephemeral scaffolding programs, that seems like a reasonable approach here. So if any libraries cause the build to fail, try adding an appropriate --disable directive to TARGET_GCC_CONFIG.

You might notice that, in this build, we’re using the same host-system builds of the various arithmetic dependency libraries as we used for the host-prerequisite GCC (or the same ones that are used for the native system GCC, if you’ve skipped the host-prerequisites). It’s totally unnecessary to build them again for this GCC — the cross-compiler we’re building here will only run on the host system, so the target architecture is irrelevant to it.

To build the minimal libgcc, we specify the configuration options --without-headers and --with-newlib. This is a bit sloppy — the first of those options is the only one that should be necessary — but the last time I tried building a minimal compiler without the newlib directive, it didn’t work.

Once this minimal compiler is finished, we’ll use it to build the C library; and then we can use the files provided by the C library to go back and build a full, functional GCC compiler.

The only targets we build at this point are all-gcc, which produces the plain compiler, and all-target-libgcc, which produces the minimal libgcc needed by that compiler.

glibc is the C standard library produced as part of the GNU project. It contains an implementation for all of the functions that are assumed to be available in C programs — like printf and so on. It also provides a dynamic loader, which almost all programs use to find shared libraries at runtime, and a few miscellaneous utility programs.

Since all C and C++ userspace programs (except the Linux kernel itself) link against the C library, glibc is by far the most deeply-embedded component of a GNU/Linux system. Upgrading the Linux kernel can be a relatively trivial operation compared to upgrading the C library installed on a computer.

There are alternative C libraries that can be used instead of glibc. However, glibc is the C standard library used by the vast majority of GNU/Linux systems; using an alternative library like musl or uClibc-ng as the primary C library may cause problems somewhere down the line.

This builds a sysroot libc (for the target architecture), configured to be installed into the /scaffolding subdirectory of the sysroot.

That might be a little bit opaque, so let’s break it down a little bit. As mentioned earlier, the sysroot framework is all about setting up a path on the host system that will eventually become the root filesystem on the target system. And, also as mentioned earlier, the goal in the first stage of the CBL build process is to set up a kernel and minimal userspace for the target system, with the entirety of that userspace contained in a /scaffolding subdirectory rather than using the conventional filesystem paths (like /bin and /lib and the /usr directory structure and all the rest of that stuff you’re used to seeing).

What we’re building here is the libc that’s going to be used to build all the scaffolding programs and libraries — the stuff that we’re cross-compiling — so we want it to be contained in the scaffolding directory, and found by the scaffolding programs at runtime in that location. Hence, it’s a sysroot libc, with an extra prefix to move it from its usual normal /lib and /usr/lib directories to the /scaffolding/lib directory.

That means we configure it with a prefix of /scaffolding, but then when we install it we tell it that the root of the installation location is the sysroot directory /home/lbl/work/sysroot.

Simple, right?

If you ever want to set up a completely standard sysroot toolchain, by the way, it works pretty much the same way as this but you specify --prefix=/usr and --with-headers=$SYSROOT_PATH/usr/include. There is some magic in the glibc configuration or build machinery related to the --prefix directive: if you specify a prefix of /usr, the bits of glibc that are conventionally installed in /lib will be put there, rather than in /usr/lib.

Since this glibc is built for the target machine architecture, a number of tests run by the configure script won’t work right. The way we work around that, for glibc as with everything else that uses the GNU build system, is by setting the correct values in a config.cache ahead of time.

An option that we’re not using here is --enable-kernel, which can limit the amount of compatibility code built into glibc to support old Linux kernel versions. Since the glibc being built here is temporary and will be discarded in its entirety, saving a little bit of space here is kind of pointless. That option also makes the build more fragile, since the kernel version that will be checked by the code is the host system kernel; we don’t want to make any more presumptions about the host system than we must.

The glibc build process uses the makeinfo program to create the documentation, and the texinfo source file specifies a document encoding of UTF-8. When using some versions of perl, this leads to problems — it’s unclear why; perhaps this is because makeinfo only works right with UTF-8 documents when being run with UTF-8 localization settings, or maybe something else is going wrong.

Regardless of exactly what triggers the issue, there’s an easy way to work around it: just remove the @documentencoding directive from the libc manual source file.

As mentioned earlier, in gcc: The driver program, the gcc program we think of as a compiler really just runs other programs, and it uses a bunch of directives called "spec strings" to determine what programs to run and what options to give them. The format of specs strings is documented in the GCC documentation in section 3.15, "Specifying Subprocesses and the Switches to Pass to Them." Spec strings don’t have the most readable structure — I find it helpful to think of them as being written in a domain-specific language, because to me they look as much like line noise as complicated regular expressions do — but sometimes there’s no better way to figure out what is going on than to read the specs, and there is often no better way to adjust the behavior of gcc than to modify the specs it is using.

We have to do this — modify the specs — a few times throughout the CBL process, primarily because we need to control how gcc runs ld to link programs.

You can see the spec strings that gcc will use by running gcc with the -dumpspecs option. The default specs are built in to gcc, but you can provide your own specs to override the default behavior by using the -specs= command-line argument or by creating a specs file and putting it in a specific location in the filesystem.

We’re going to do the latter. The basic process is to first dump the specs file to the location where gcc will look for it:

Then we can modify it however we need to, and gcc will use the modified version.

Not to belabor the point, but remember that the whole purpose of this cross-toolchain is to let us build the scaffolding that will then allow us to build the final target system entirely from source code.

While we’re using the scaffolding tools to build the final system, we want to do a native build of everything, including glibc, so we want the scaffolding to be independent of any filesystem locations that will still be present in the final system. In particular, this includes /lib and /usr/lib. That way, once we’re done doing the target system build, we can delete the scaffolding directory altogether and be confident that there are no lingering host-system artifacts or ephemera on it.

When the standard GNU toolchain builds an executable, it almost always links it against the dynamic link library (sometimes called the "dynamic loader" or "program interpreter"; this is something like ld-linux.so.2 or ld.so.1, and is conventionally found in a top-level directory called /lib or a multilib variant like /lib32 or /lib64).^[13] That’s normally fine, but we want the scaffolding to be entirely independent of /lib. So we need to adjust our cross-toolchain so that the programs it builds look in the /lib directory under the scaffolding location for their libraries, including the dynamic link library. This is done by modifying the GCC specs file.

There’s another problem we need to work around, as well: gcc doesn’t provide any good way to tell it where to find some object files that need to be linked into every program: crti.o, crti1.o, and crtn.o. These are provided as a part of glibc, so like the rest of glibc they were installed into /home/lbl/work/sysroot/scaffolding, specifically into its /lib subdirectory. But, of course, that’s not a location where gcc normally expects to find them. So at this point if you were to try compiling a "Hello World" program with your cross-toolchain, it would complain that the cross-ld can’t find crt1.o or crti.o.

You can tell exactly what is going wrong by repeating the compile with -v, to get gcc to print out all the commands it’s running: cc1 to compile the code, then as to assemble it, then collect2 to link it. And apparently collect2 is running ld, which is producing the error message. (That’s weird, but you get used to weird stuff when you’re trying to figure out toolchain problems. There’s also no explicit execution of cpp; that’s because the C pre-processor is actually implemented in the libcpp library and is invoked as function calls to that library by the cc1 compiler program.)

Once you have the actual command line that’s failing, you can try adjusting it to see if there’s an easy way to get it to work. For example, the command line just names the startup files without any path components at all. After fussing around for a while looking for pleasant alternatives, the best solution I found was to specify the filenames with absolute paths. So that’s what we’re going to do in the specs file: replace every occurrence of the bare filename with absolute paths, for all the object files that appear in the scaffolding/lib directory.

We can now verify that the cross-toolchain is able to build programs successfully, and is set up to link against the sysroot glibc in the /scaffolding directory, by compiling any simple program (like "Hello World") and then running readelf on it — there will be a line in the program headers section that says "Requesting program interpreter:" and should contain the path to the dynamic link library in the scaffolding location.

After building a significant toolchain component, it’s a good idea to make sure that it works as intended. This is a simple smoke-test: it just compiles a "Hello, World" program and then inspects it to make sure it was built as expected and runs properly.

This verifies that the cross-toolchain and emulator work properly: look at the machine type and dynamic linker location for a compiled program, and then make sure that it runs in the userspace emulator. This proves that the cross-toolchain and QEMU were properly built with a compatible target architecture and so on.

This is compiled with:

To verify that it’s linked properly, use readelf.

The Machine line should indicate the target architecture, rather than the host architecture, and the program interpreter requested by the program should be under the /scaffolding/lib directory (which is where the dynamic loader will be found once we’re booted into the target system). If either of those is not the case, the grep commands will fail, causing the CBL build process to abort.

One last thing we can usefully do at this point is verify that the user-mode QEMU emulator can actually run programs for the target architecture.

This is a bit tricky when running the dynamically-linked program we just built, because it is expecting to find the program interpreter at /scaffolding/lib but in fact it’s actually at that location under the sysroot directory. Luckily, the user-mode QEMU emulator can be told where to find library files, using the -L command line argument or the QEMU_LD_PREFIX environment variable.

This will produce a friendly greeting, or — if something goes wrong — will, again, cause the build process to abort.

(If you’d like, you can try running the hello program outside of QEMU as well; that should produce an error message like "cannot execute binary file." And, again, if it doesn’t, that means something is horribly wrong!)

pkg-config is a program that makes it easy to find things like installed libraries and header files. Many programs use pkg-config in their build processes to find out if the libraries they depend on are present on the build system, and to find out what linker and include directives should be used to compile and link against them.

The original pkg-config program can be found at pkg-config.freedesktop.org. At some point in its development, its developers decided to use functions from the glib library; unfortunately, glib uses pkg-config to find its own dependencies (really, just zlib — glib doesn’t have a lot of dependencies), which introduced a cyclic dependency. That doesn’t cause any really intractable problems — a couple of environment variables can be defined when building glib so that it doesn’t have to use pkg-config to find zlib — but cyclic dependencies are always kind of horrifying: not to sound like a broken record, but the idea with CBL is to start with a minimal set of binaries and a pile of source code and turn that into a whole system. (At some point after the glib dependency was introduced, pkg-config started requiring itself, so that it can find and link against the glib library, unless additional environment variables are provided.)

The situation has since been resolved, but it was resolved by bundling glib along with pkg-config. This isn’t a particularly elegant solution: the pkg-config distribution is about 11.5 megabytes of code, unpacked, and about 9.5 megabytes of that is glib.

A different approach was taken in a fork of pkg-config, "pkg-config-lite," which includes just the snippet of glib that is needed by pkg-config: that’s better, but still not ideal.

This brings us to pkgconf, a completely separate implementation of the pkg-config program. It has no external dependencies, doesn’t bundle any third-party code within its own distribution, and also has a design that I find preferable to the original pkg-config (it internally builds a directed acyclic graph of dependencies, rather than building an in-memory database of all known pkg-config files at runtime and then resolving dependencies from that database). So that’s what we use in LB Linux.

As distributed, the pkgconf program is missing a lot of files that are normally produced by the GNU autotools; the conventional way to build it is to start by running the autogen.sh script provided with the package distribution. However, one of the places the pkgconf program is built is as part of the host prerequisites; if that’s being done, it’s probably not a great idea to rely on the autotools being present at prerequisite build time. We can instead simply patch the source distribution to create the files that would normally be created via autogen.sh.

This is built here because pkg-config is already present on the host system (as a prerequisite, if nothing else), and it probably knows about a lot of host system libraries. While building the scaffolding, we need to ensure that their build processes don’t find (and try to link against) any of those host system libraries. Since the host system libraries are for a different machine architecture, this would cause build failures. We can do that by putting a new version of pkg-config, configured only to look in the scaffolding directory, at the head of the PATH while building them.

We could probably get by with a simple shell script that always just says "I couldn’t find anything!" when asked for dependencies. On the other hand, it takes around ten seconds to build pkgconf and set it up, so why not just do that?

In addition to the pkg-config symlink, we create a symbolic link x86_64-cbl-linux-gnu-pkg-config so that any build process that tries to find dependencies using a target-system pkg-config program will find ours.

Most Linux filesystems support "extended attributes" — arbitrary name/value pairs that can be associated with files or directories. These can be used for any purpose; for example, you might attach a "user.file_encoding" extended attribute to a text file, if it’s encoded unusually.

One of the primary uses of extended attributes is to implement access control lists or capabilities.

The attr package provides programs that allow extended attributes to be viewed and modified.

Since the configuration repository preserves extended attribute metadata as well as basic owner and mode, and since the package-users build script sets a user.package_owner attribute on all files that are installed by packages, we need the getfattr and setfattr commands early in the target system build.

Ncurses ("new curses") is a library that provides terminal control features so programs can provide advanced text-based user interfaces. It’s a free-software version of a similar library (which was simply called "curses") that was developed at Berkeley.

There are lots of programs, even just in the scaffolding we’re setting up, that use ncurses. bash and vim are among them.

Patches are available in a version subdirectory of the overall package download location — for example, patches for ncurses 6.2 are in the "6.2" subdirectory of the ncurses package directory. If there’s a README file in that directory, follow the instructions in it. If not, just fetch all the patch files (and, optionally, GPG signature files) and apply those in order. When I update ncurses for CBL, I fetch all the patches available at that point, verify their GPG signatures, and compile them into a single branch-update patch, available in the file repository at files.freesa.org. If you don’t want to make sure you’re using the absolute latest version, you can just use that.

When configuring the scaffolding ncurses, we specify --without-ada, since Ada isn’t built in the CBL process. We also specify --enable-overwrite so that the header files are installed into /home/lbl/work/sysroot/scaffolding/include rather than an ncurses subdirectory of it, because that makes it easier for other programs to find the headers, and --with-build-cc so that the native tools built as part of the ncurses build process use the native compiler instead of the cross-compiler. We also specify --disable-stripping because otherwise the install program might try to strip binaries using the host system strip rather than the cross-toolchain strip.

Bash is a shell: it provides a command line prompt, parses commands and pipelines entered on the command line, executes programs based on those commands and pipelines, and then does it all again.^[14] If you’re interacting with a Linux system, and you’re not using a graphical environment like Xorg, you’re using a shell like bash. And, probably, you’re using bash — there are other shells, but they are not nearly as common.

The bash maintainers don’t provide all patchlevels for convenient download; you may have to download the tarfiles for the main version (like 4.3) and then separately download and apply all the update patches. The files.freesa.org repository, for convenience, has a tarfile that includes all of the patches for the version specified in this blueprint.

Bash includes an ancient version of the termcap library, which is adequate to get it to build. Ncurses is far superior, and we need it anyway for other packages, so it’s worthwhile to make sure it’s available when building bash.

Bash includes its own memory allocation routine, but historically it has not always been reliable. We use --without-bash-malloc to disable it so that the C standard library’s malloc is used instead. For the scaffolding version of bash, we also configure it to be linked statically; that reduces the number of pieces that are needed to get the minimal scaffolding-based system to boot.

In GCC 15, the default C language standard is C23. Bash has problems when compiled with that version, so we use CFLAGS to change it to C17. There’s a separate CFLAGS_FOR_BUILD used when compiling code that is only used at build time, which we also need to set.

In some circumstances, bash needs to redefine the getenv function. It can’t check whether this is necessary when cross-compiling, so it simply assumes it should do so. WHen building against glibc versions before 2.41, this was okay, but now it causes a compilation error. So we explicitly disable the redefinition of getenv here.

M4 is a macro processor: it copies its standard input to standard output, expanding macros as it goes. It has a set of built-in macros that it understands, and it’s possible to add user-defined ones as well. M4 is used extensively in the GNU build system, primarily by Autoconf.

M4 doesn’t build using the C23 language standard, so we need to specify an earlier version.

Bison is a parser generator. That means it takes a description of a language (in a specialized format called a "context-free grammar") and converts it into a parser for that language. This is mostly useful when writing compilers: the parser is the part of a compiler that takes a stream of tokens and figures out what syntax elements they represent. Parsers are fiddly and difficult to get correct, so it’s common to generate the code for parsers using tools like Bison.

The name "Bison" is kind of a joke. The original parser generator that was commonly available on UNIX systems is called "Yacc," which is an acronym for "Yet Another Compiler-Compiler." When the GNU project implemented their own parser generator, they called it "Bison" because "Yacc" sounds like "Yak."

Bison’s installation process also creates a program called yacc that simply runs bison in yacc emulation mode.

The absolute path to the m4 program is written into bison, so we need to make sure that the scaffolding bison will expect to find m4 in the scaffolding directory. This adds a little complexity, though: the bison configuration script checks to see whether lex is available and, if it is, whether it is flex. flex uses m4, so if we tell the bison configuration that m4 can be found in the scaffolding, flex will try to use that version of m4, which was built for the target system rather than the host system. As with other packages that use the GNU Build System, we can use config.cache to tell bison that flex is simply not available.

Bzip2 is a compression program like gzip, but uses the Burroughs-Wheeler block sorting algorithm and Huffman coding. It is a lot slower than gzip for both compressing and decompressing, but compresses data much better in most cases.

Bzip2 doesn’t use the GNU build system, so there isn’t a configure script.

There isn’t really a configuration step for bzip2, but we’ll use the configuration stage to modify the build process so it won’t try to run the tests — they won’t work when building with a cross-toolchain.

The regular Makefile only creates a static version of the library, which is fine; however, it doesn’t compile that library with -fPIC, which causes problems later on when other programs try to use it. We can add that directive directly in the Makefile.

Because bzip2 doesn’t use the GNU build system, we need to specify the cross-tools as arguments to make.

The Core Utilities are basic file, shell, and text manipulation utility programs: things like cat, chmod, chown, cp, …​ stuff like that. You use these all the time.

A couple of the tests run by the configure script don’t work right when the coreutils are being cross-compiled. As with other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

Diffutils is a bundle of programs that find differences between files and help you to merge them together: cmp, diff, diff3, sdiff.

The most important program here is diff, which finds differences between files. It can find differences between two files, in the simplest case; but, more commonly, diff is used to find all the differences between all the files in two entire directory trees.

You can save the output from diff when used this way as a "patch file", and then use the patch program to take one of the directory trees and make it look just like the other one.

This is handy when distributing modified versions of software packages: if you’ve made a minor change to GCC, for example, and you want to submit that change to the GCC mailing list for consideration by the GCC maintainers, you can use diff between the original GCC source tree and your modified GCC source tree, and then include the output from diff in your email.

When cross-compiling, the configure script guesses that the getopt function provided by GNU glibc is not available and tries to use an internal one instead. This doesn’t work, for some reason (possibly because of behavior changes in GCC 7). Since the glibc getopt is available, we can just tell the configure script not to guess about it.

util-linux is a grab-bag of miscellaneous Linux utility programs, for all kinds of things: disk partitioning, filesystem creation and validation and mounting…​ all kinds of stuff like that. (I’d expect a lot of this stuff — like more, mount, and umount — to be a part of the GNU system, perhaps in the coreutils package, but that’s not the only thing I find surprising about the GNU/Linux world.)

We configure with an option that tells the build machinery not to chgrp the wall program to the tty group. This will probably make wall non-functional, but that’s perfectly okay — wall can be used to send messages simultaneously to all users logged on to the system, as for example if the system is going to be shut down or something of the sort. This is not very useful on single-user computers, and is certainly not necessary in the minimal scaffolding userspace.

You’ll notice that we’re configuring this package with a prefix of /home/lbl/work/sysroot/scaffolding instead of just /scaffolding (and installing without a DESTDIR). That’s because, when I configure and install util-linux the same way as other scaffolding components, I’ve sometimes encountered problems where some components can’t find the libuuid library provided by this package. There are probably other ways to address that, and it’s not really clear whether those components are strictly necessary, but this non-standard configuration scheme doesn’t cause any problems and works fine.

The installation routine for util-linux changes the ownership and mode of some programs (like mount and umount) to be setuid to root. This doesn’t work when running the installation as a normal user, as is being done in this section, so we disable that.

One of the installed utilities is lastlog2, which requires sqlite. We’ll disable it for this phase.

The ext2fsprogs are utilities for managing the ext2 (and later) filesystems. ext2 was the first popular filesystem for Linux systems and its latest incarnation, ext4, is the most common. This package contains programs for creating filesystems, reconfiguring them, checking them for problems, that sort of thing.

There’s some overlap between e2fsprogs and util-linux: both provide libuuid and libblkid libraries, a uuidd daemon, and a fsck script. We skip those from e2fsprogs.

There is a glitch in the configuration logic that creates a file outside of the DESTDIR-specified path when there is no /etc/cron.d directory. This can be avoided by explicitly specifying that there is no crond directory at all.

e2fsprogs needs to be built with an earlier standard than c23.

Expat is a library that provides a stream-oriented XML parser. It’s used by all sorts of other programs.

file guesses file types by looking for particular characteristic byte values within them. If you’ve got a file and you’re not sure what it actually is, you can run file on it and be told things like "that’s a text file," or "that’s a JPEG image."

On Microsoft Windows, file types are inferred based on the extension of the file — exe files are executable programs, jpg programs are JPEG images, and so on. file does things differently: it looks at the content of the file and tries to determine the file type by looking for characteristic patterns within it, using a compiled database of these patterns called the "magic file," which you can read more about with man 5 magic. The magic file is produced during the build process by running file -C (with the freshly-built file program) to compile a bunch of "magic fragment" files into the database that it normally uses.

The Find Utilities are basic directory searching programs: mostly find and locate. This package also includes updatedb, which creates the file name database that locate uses; and xargs, which takes lists of file names produced by find and generates command lines that operate on all the files in those lists.

The GNU version of find has no support for extended attributes, but since Little Blue Linux uses extended attributes to preserve the name of the package that owns files even when they are `chown`ed and `chgrp`ed, we need that feature for use in some of the package user scripts. I found a patch from Nicolas Iooss that adds this functionality.

After applying that patch, it’s also necessary to update the autotools-generated files used by the build system. That’s in a separate patch to make it simpler to respin the xattr patch for new versions of this package.

A couple of the tests run by the configure script don’t work right when the findutils are being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

AWK is a special-purpose programming language that makes it easy to do simple text manipulations. (The name is an acronym of the three people who designed the language: Alfred Aho, Peter Weinberger, and Brian Kernighan.) Gawk is the GNU implementation of the AWK language.

If you’re comfortable in perl or python or ruby, you probably won’t have much use for awk; on the other hand, if you’re working in a bash script, awk might turn out to be very handy.

Zlib is a compression library. It implements the same Lempel-Ziv compression algorithm as gzip and info-zip, which means it doesn’t compress data as effectively as most other algorithms (like the Burrows-Wheeler algorithm used by bzip2 or the LZMA algorithm used by xz-utils), but on the other hand it doesn’t use very much memory to compress, and it’s pretty fast.

There are a lot of programs that link against zlib to get basic compression capabilities. We’re not totally sure which components will fail to build if it’s not available, but zlib itself has no dependencies and is really fast and small to build, so making it available is no big deal.

Gettext is a set of tools that can be used by other programs to provide internationalization and localization capabilities. This lets a single program provide user interfaces and user-facing messages in a variety of languages without requiring it to be rebuilt.

One of the tests run by the configure script doesn’t work right when the coreutils are being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

If the emacs editor is available on the system, the gettext build will do something with it. I don’t really understand what it is, but it generates a couple of gigabytes worth of log output and makes the build process take orders of magnitude longer than it otherwise would. Luckily, it is easy to disable that with a configuration flag.

Grep searches input files for lines that match patterns called "regular expressions." It often prints those lines out as well.

We have heard that the program name "grep" was originally derived from the ed command "g/re/p", which does basically the same thing as the command-line grep program, and wikipedia makes this claim as well. Maybe it’s true!

The GNU libc library provides regular expression functions, but grep ignores them by default and uses its own bundled regex library. We use a configuration flag to override that behavior.

Gzip is a compression program that basically does the same thing as the zlib library, but with a command-line interface.

The default -Werror setting for gzip causes a format-truncation message produced by GCC 8 to abort the build. I am not all that concerned about format-truncation problems when printing error messages, so I just disable that setting.

The generated Makefiles for gzip include a -Wabi directive that GCC 8 complains about. We can just remove that directive.

The Kbd package contains key-table files and keyboard utilities.

The vlock program requires Pluggable Authentication Modules (Linux-PAM). This is not part of the basic LB Linux system, so we always disable it. Conversely, other programs that aren’t built by default might be helpful, so we enable those.

We need some of the programs provided by kbd (notably, openvt, which will let us run interactive shells on virtual terminals in the minimal target system) in the minimal userspace.

Libffi is a portable library that helps programs written in one language to invoke functions that were written in a different language.

LibreSSL is a fork of OpenSSL with the goals of modernizing the codebase, improving the reliability and security of the code, and applying better development practices.

As an added bonus, the LibreSSL package includes a program called nc (short for "netcat") that can be used as a simple TCP and UDP client or server program — this can be enormously handy if you just want to shove data over a network connection, or test network client or server programs, or something like that. You can man 1 nc if you’d like to see more about it.

LibreSSL should not be needed in the scaffolding, but the current version of rubygems has issues when used in a ruby installation without openssl support.

Generally, when you log in to a UNIX system, you provide a username and a password. Obviously, the password needs to be kept secret; I don’t think there’s any need to belabor that point. When you log in, the system — in this case, the login program — has to validate that the password you entered is correct. It does this by consulting an authentication database file.

If some nefarious person were to want to compromise the security of a system (I know it’s shocking, but there are some nefarious people out there), they might try to do this by obtaining a copy of the database file. System designers and administrators try to keep that file locked down so that it’s not feasible for nefarious people to make a copy of it, but some bad actors are very clever and find ways to do so. To make that situation less problematic, the database file does not actually contain the passwords that users enter. Instead, when users enter a password, the passwd program mangles the password using a hashing function that cannot be reversed, and stores the mangled, or "hashed," string rather than the actual password. When a user logs in, the login program uses the same hashing function on the password that they provide, and compares the hashed value to the hashed value in the password database. If they are the same, then login can conclude that the user entered the correct password.

libxcrypt is an implementation of these hashing functions.

Historically, these hashing functions — primarily crypt and crypt_r — have been provided by glibc. Rather than being part of the main shared library libc.so, these functions were put in a separate library libcrypt.so. However, in glibc release 2.38, this library is no longer built by default; it can be constructed by setting the --enable-crypt configure flag, but the recommendation in the glibc release notes is to switch to an alternative implementation like this one. Conveniently, libxcrypt builds a libcrypt.so library that is compatible with the one that has historically been provided by glibc.

YAML — the name stands for "YAML Ain’t Markup Language" — is a human-readable data serialization format. You can read all about it on https://yaml.org/. (Perhaps worthy of note, directives in litbuild blueprints are written in a very YAML-ish syntax.)

LibYAML is a library, originally written as part of the PyYAML project, for parsing YAML into data structures and emitting YAML from data structures. It’s a convenient serialization format for cases where someone might need to look at the serialized data and understand what it is without any arcane hackery or tools.

The ruby standard library includes a YAML implementation that depends on LibYAML.

GNU make is a build automation program. To use make, you set up one or more configuration files — the primary one being called, by convention, Makefile — that declare recipes for producing intermediate and final program artifacts, and dependencies between artifacts, and the various targets that can be produced. At runtime, make looks to see what artifacts exist already and which source files have a timestamp indicating that they’ve changed after dependent artifacts were produced, figures out based on that analysis exactly which artifacts need to be rebuilt, and then executes the recipes that will produce those artifacts.

That’s all pretty awesome!

The downside is that Makefiles are not particularly clear or readable, and for large projects they get pretty big and complicated. That’s why there is a thing called "litbuild"!

But the vast majority of system components use make to automate their build process, so you really have to have it available regardless.

Remember diffutils? The most important program included in diffutils is diff, which finds differences between two files or directory trees. Patch is the conceptual inverse of diff — it takes a file with all the differences found by diff as input (these files are, by convention, called "patch files"), and applies all of those differences as changes to the files in a directory tree.

This is really handy. For example, the CBL repository contains a bunch of source tarfiles that were obtained directly from the project web sites; and it also contains patch files that can be applied to those source tarfiles to apply changes that we have found to be necessary or important when building a CBL system.

One downside to the patch program is that it can’t apply some of the patches that are created with git format-patch: git is able to create patches that include differences for binary files, but patch cannot handle those. If the output from patch includes any lines saying git binary diffs are not supported, you will need to use git apply rather than patch.

Pth is a portable threads library; it provides cooperative priority-based scheduling for multiple threads within a program.

The build system for pth doesn’t work well when parallelized, so we explicitly disable parallel make jobs by specifying -j1.

On some servers, the version of config.guess distributed with pth is too old to recognize the target triplet. It’s easy enough to update it to the version distributed with GCC.

In some cross-compilation scenarios, pth fails to compile because it considers versions of the Linux kernel later than 2.9 to be "braindead." I got a patch from https://bugzilla.redhat.com/attachment.cgi?id=591825 that corrects the problem.

Ruby is a programming language. There are several interpreters for that language — rubinius and jruby are others — but the canonical one is this package, the C implementation that was created by Yukihiro "Matz" Matsumoto. This is also the best one for our purposes in CBL, because it has the fewest upstream dependencies.

CBL includes ruby as a core component because CBL itself is designed to be used with the litbuild program — CBL consists entirely of litbuild blueprints — and litbuild is written in ruby.

On 64-bit ARM systems, the ruby build process exposes an issue with GCC (documented at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=84521) when compiled with -fomit-frame-pointer and an optimization setting of -O1 or -O2. Since the default behavior for modern versions of GCC is to omit frame pointers, it’s important on such systems to specify -fno-omit-frame-pointer when building ruby. It’s possible that other architectures have a similar issue; if your ruby build crashes with a "stack smashing detected" error message, try adding that option to your builds as well. (The default configuration for CBL specifies -fno-omit-frame-pointer in the target-system CFLAGS.)

Ruby is an interpreted language, like perl and python, rather than a compiled language like C. As with perl and python, ruby programs and libraries are packaged in a language-specific way.

For ruby, that packaging format is called "ruby gems," which can be created and managed using the gem program distributed as a part of standard ruby installations.

Gem files are not especially complicated — they are tarfiles containing a compressed data tarfile that contains the files that make up the package, a compressed metadata.yaml file that contains a bunch of metadata about the gem, and a compressed checksums file with SHA256 and SHA512 checksums of the data and metadata.yaml files.

The gem program allows you to download and install a gem, along with any other gems it depends on, from an external repository. By default, it uses the repository https://rubgems.org/, but you can change that if you wish: the gem sources subcommand allows you to add and remove URLs that will be used to find ruby gems when you run gem install.

Gems are built from "gemspec" files, which contain most of the metadata that winds up in the metadata.yaml file in the gem. You can build a gem using gem build and providing the path to a gemspec.

For my purposes, the type of metadata I find most useful and interesting is information about dependencies, which are specified in gemspec files using the add_dependency attribute. Dependencies in gemspecs can include version constraints, with several operators that control how they work. For example, '>= 2.1.0' indicates that version 2.1.0 or any later version may be used, while ['>= 2.1.0', '< 3.0'] specifies that version 2.1.0 or any later version up to version 3.0 is okay. Rubygems also supports a shorthand for the latter constraint: ~> 2.1 also means "any version 2.1 or later, but less than 3.0."

When you use the gem install command, the ruby gems you specify will be downloaded from the sources gem knows about, along with any dependencies they specify with add_dependency, and installed for use with your ruby installation. If you add the --development option, gem will also install development dependencies that were specified with add_development_dependency in the gemspec.

Once we boot into the target system, we’ll use litbuild to generate all the scripts that will build the final CBL components. That means we’ll need ruby available!

As with the file package, ruby can have issues in cross-compilation scenarios unless there is a native ruby installation of the same version. This doesn’t always happen, but the installation of a scaffolding ruby 2.7.0 failed when the system ruby was version 2.6.5; if something similar goes wrong for you, a version mismatch is a good thing to check.

The version of rubygems distributed with ruby 2.7 and later really wants to load the ruby openssl library, which requires OpenSSL (or a fork of it, like LibreSSL).

Ruby 3.2 requires the native libyaml library to build psych, which is needed for the litbuild gem and possibly others.

One of the configure tests gets confused when cross-compiling (or perhaps whenever Ruby is being compiled for 64-bit ARM targets), so we’ll use a config.cache to skip it.

Sed is kind of like a text editor, but instead of allowing you to modify text files interactively (like ed and vim do), sed acts as a filter: it takes a text file as an input stream, makes changes to that input stream, and produces the resulting modified text as an output stream. (The name, sed, means "Stream EDitor").

This is a fairly powerful and flexible thing to be able to do, but it’s not something people often need to do — like gawk, sed is most often handy in the context of a bash script or something like that, where a more powerful general-purpose language like ruby or python or perl isn’t convenient for whatever reason.

On the other hand, the automated build processes for some of the CBL components use sed, so you have to have it around anyway.

Tar is a program for manipulating archives of files. The name derives from its original purpose, back in the dawn of time, which was to manipulate archives on magnetic tape — ergo, "tar," for Tape ARchive. Actually storing archives on magnetic tape is pretty rare these days, but people still use tar-format files for all kinds of things.

These files are known as "tarfiles," for obvious reasons, or as "tarballs" for less obvious reasons. A speculation I found on Wikipedia is that the term references the tendency of actual tarballs (blobs of petroleum floating in the ocean) to get all sorts of things stuck to them; apparently, someone thought that was reminiscent of how the tar program collects a bunch of files together.

All of the source code distribution packages for CBL packages are compressed tarfiles, so the tar program is really important for CBL!.

The tar package also provides a program called "rmt" that lets you manipulate magnetic tape drives attached to other computers. This is an even more fringe thing to do in this day and age, but It only weighs in at about 50kb, so it’s not hurting anything.

Several of the tests run by the configure script don’t work right when tar is being cross-compiled. As with some other programs that use the GNU build system, we can short-circuit those tests by pretending that the configure script was run previously and knows the results from those tests.

Texinfo is the official documentation system for the GNU project. It uses input files in a standard format to produce a variety of outputs, both printed and online.

Texinfo is used in the automated build process of many of the CBL component packages.

In release 7.1 of texinfo, the install-info command fails in some cases. I think this is because install-info now has code to ensure that the /usr/share/info/dir file is updated atomically: it writes the new version to a temporary location, then moves it over the original file, overwriting it. That doesn’t work on LB Linux, because it uses the package user approach to package management, and /usr/share/info, like other install directories, has the sticky bit set. (If you don’t know what any of that means, you can read the package-user manual — on Little Blue Linux systems, you can find it in /usr/share/doc/package-users.) For some reason I do not understand, many packages are not affected by this, but a few are. As I write this, so far, I’ve come across this with nettle and glibc, but perhaps there are others. Whenever this happens, I use a simple workaround: simply chown the dir file before installing the package, and chown it back afterward.

When cross-compiling, the configure script makes some incorrect guesses about the availability of functions provided by glibc. We can override those guesses in a config.cache file.

Even after doing that, the presence of the gnulib version of getopt.h causes the build to break. That’s easy enough to work around simply by removing it.

The build process creates two scripts that expect perl to be in the location where it’s found on the host system, rather than the place it will be available in the initial target system, so we fix them up after installation.

Back in the early days of Unix, someone wrote a text editor called "ed" (which stood for, get this, "EDitor"). Ed is a line editor, which means that generally it lets you modify a line at a time. Ed is still available, and in fact still actively developed — GNU ed 1.14.1 was released in January of 2017 — but most people want a more interactive editor, and so one of the old proprietary Unixes eventually included an editor called "vi" ("VIsual editor") to fill this need.

Since vi is proprietary, various people implemented similar programs, which they released under open source licenses of various sorts. One of the people who decided to do this is Bram Moolenaar, who created vim; its name stands for "Vi IMproved," because it’s massively superior to vi. Bram Moolenaar died in August of 2023, but there were other contributors to the vim project who continue to develop the program.

Vim is distributed under a different software license than most of the packages that make up CBL: the Vim License. This license is GPL-compatible, but is described as "charityware" — users of vim are urged to make a donation for needy children in Uganda.

As far as that goes, you probably don’t need to install vim at all. There are plenty of other text editors that some people prefer. The default editor in some distributions is nano, and some people like their text editor to be their primary tool for interacting with the computer and use emacs. Whatever your preferences, though, you absolutely do need a text editor of some sort, and vim is my favorite, so that’s the one that winds up in the basic CBL blueprints.

As with many scaffolding pieces, the configure script for vim doesn’t play nicely with cross-compilation, so we pre-fill a config.cache with a bunch of values that it won’t be able to discover on its own.

An interesting difference between vim and most other packages we’re building is that it automatically picks up entries from a config.cache file located in the src/auto directory, and apparently doesn’t support providing a reference to config.cache in the top-level configure run.

There are a lot of programs that expect a text editor called "vi" to exist, so we’ll create a symlink after installing vim.

The XZ Utils package provides a compression/decompression program with an interface similar to gzip, but using the LZMA algorithm. This is very slow when compressing, very fast when decompressing, and compresses more effectively than most of the other common compression programs, so it’s used fairly often when providing archive files for download.

The compression algorithm used by the XZ Utils is the same as that used by lzip, but the file format is more complex and the compressed output it produces is usually slightly larger than that produced by lzip.

Having constructed all the scaffolding we can build on the host system, it’s time to set up for the second (target) stage of the build. The target-side build will be run automatically when the target system is booted.

As mentioned elsewhere, you can add data to the entropy pool without any difficulty from any userspace process: all you have to do is write data to /dev/urandom. However, that doesn’t help to initialize the entropy pool, because it doesn’t increase the kernel’s estimate of how much entropy is actually available. To do that, we have to execute a system call that is only available to processes running as the superuser.

The system call in question is ioctl, which is kind of a catch-all that exposes arbitrary functionality in device drivers. ioctl itself is short for "Input-Output Control." The idea is: since there is a huge variety of input/output devices that might be available to your system, and there’s no way for Linux to provide specific system calls to exercise every distinct function of every one of those devices, any device driver can define a set of ioctl operations. Userspace applications can then use the ioctl system call to invoke any of those operations, using a file descriptor that references one of the device’s special files under /dev.

The device driver for the /dev/random and /dev/urandom special files happens to provide an ioctl that lets you add data to the entropy pool. We’re going to write a tiny C program that feeds some data to the kernel using that ioctl, so that the entropy pool will be fully initialized immediately.

Note that it is an extremely bad idea to use this program unless you are confident that the input data really is unpredictable, or unless (as in the case of the target-side CBL build) you are equally confident that there is no reason to be concerned about the quality of random data available to userspace processes.

In many cases, this program isn’t needed in a full working system — the rngd daemon from the rng-tools package feeds entropy to the kernel entropy from hardware random number generators, like the one built into modern x86-architecture CPUs. But CBL supports multiple computer architectures, and some of those don’t have anything suitable for rngd, so addentropy can be helpful in those cases.

The addentropy program is derived from ec2seed, at https://github.com/akkornel/ec2seed, which is available under the terms of the GNU GPL Version 3. Accordingly, there’s no issue with distributing the addentropy source as part of CBL; all the code within CBL is already licensed under the GNU GPL Version 3.

As with almost all C programs, we start by including header files.

We define a constant for the number of bytes we’re going to feed to the entropy pool. That’s really not all that important, but it makes some later parts of the program slightly clearer.

Before and after adding data to the entropy pool, we can obtain the kernel’s approximation of how much entropy it already contains, using another ioctl provided by the /dev/random device driver. Let’s define a function for that.

Now we can write the main routine. As usual in C programs, we start by defining all of the variables we’ll use.

We need to open a file descriptor on the /dev/urandom special file so that we can invoke ioctl operations on it. And we need to allocate a buffer that we can read the data in to.

According to the kernel source file random.c, which is the source code for the random-number device driver, the entropy_count field of rand_pool_info structures is denominated in units that are an eighth of a bit each. I have no idea why. So maybe we should multiply RANDOM_BYTE_COUNT by 64 rather than by 8 here? But this appears to work perfectly well, so I’m not messing with it.

Now we need to obtain some data (which we will presume to be random). We’re simply going to read the data from the standard input stream. Since we’re using some low-level file manipulation functions, we have to deal with the possibility that any given call to read might actually produce less data than we request; therefore, we set up a loop and read repeatedly until we have all the data we want.

Now we can add the data to the entropy pool, using the RNDADDENTROPY ioctl. We’ll also print out a description of what happened.

There’s nothing left to do other than clean up the resources we’ve allocated and terminate the program. Technically we don’t have to clean up the resources — Linux will do that automatically as the process is reaped — but it’s good practice to do it.

That’s it!

The command we use to compile the program, and the location where it will wind up, depends on whether we’re building this in the scaffolding or for the final target system.

For the scaffolding, we need to use the cross-compiler to compile addentropy, and it needs to wind up in the scaffolding bin directory.

When the kernel runs init, it does so with a minimal (maybe empty?) environment. Bash does not work well unless some basic environment variables are set — like PATH and LD_LIBRARY_PATH for program and library lookups, for example — so we’ll start by getting those set.

When problems occur during the target-side build, the init script bails out to an interactive shell that also has an empty environment. So it’s convenient to have all the commands that set up the environment in a separate script that can be easily sourced in from that rescue shell. The same script can be sourced from init to avoid any duplication between the two.

GNU/Linux systems have extensive support for internationalization and localization. Before we have the full system set up, it’s a good idea to specify a simple default setting for those features.

As the final system programs and libraries are built, they should be used in preference to the ones in /scaffolding, so we’ll put the directories where they’ll live in front of the /scaffolding directories.

The TARGET_SYSTEM_MAKEFLAGS parameter specifies the MAKEFLAGS that should be set for all target-side processes. (If a specific package has issues with parallel builds, it can be overridden to -j1 or unset entirely for those packages.)

There are some programs that expect HOME to be set to a reasonable value. We can go ahead and set it here in case any of the programs we’re using is among those.

We also need to set environment variables for all the litbuild configuration parameters that might not be using default parameter values. It’s not hard to define these properly, because from this point onward we don’t need to worry about HOST and TARGET and so on: everything is just going to be a native build. We do need to pass a number of parameters from the host-side build along through to the target-side build, though!

Many of the parameters used in the litbuild blueprints will have static values defined by convention within CBL. For those, we don’t care about the host-side parameters that were used, we can just set paths as we wish.