A QEMU-to-QEMU Build Process

My goal here is to make it really easy and straightforward to automate the whole process of building many different types of Little Blue Linux systems. The eventual plan is for this to run as part of a continuous integration (CI) pipeline, every time the CBL repository changes.^[1]

Wiring up the CI pipeline is outside the scope of what I’m trying to accomplish here! But this is the script that I run to do the full build, after I set things up as described in [Preparing For The Automated Build]:

As with everything else in CBL, of course, you should modify that script to suit your own personal preferences. The BUILD-INFO file produced by that script also has timestamps added to it throughout this process, so the final result lets you calculate how long every part of the build (and post-build process) takes.

The timestamps recorded by this script use the %s date format, which converts to the number of seconds since the UNIX epoch (midnight UTC on January 1, 1970). That’s not very readable, but it’s very convenient for doing time arithmetic — you can get the number of seconds taken for the full build just by subtracting the end timestamp from the beginning timestamp. If you want to convert these timestamps to a more readable date and time string, you can use the date program: to convert, for example, 1630600291 to a date and time, you can use date -d @1630600291. (You can also use a more verbose but also more straightforward syntax: date -d "1970-01-01 UTC 1630600291 seconds" means just what it looks like: the UNIX epoch plus some number of seconds.)

This process assumes that you have a computer with:

You don’t have to use LB Linux for the base system image, but this process makes some assumptions about the host system and will need to be adjusted if those assumptions are not met. For example, it expects to do the build in /home/lbl, so if there is no such directory you’ll have to change the definition of HDIR here. Also, there must be a getty process running on the serial device, configured to set TERM to a simple terminal type (such as tty); and the root account must not have a password set.

This blueprint needs additional configuration parameters beyond what are needed for the CBL process; they are used primarily to construct the command lines to run the host and target QEMU processes. It’s most convenient to define these in the same cbl-configuration.sh file that will be used for the automated CBL process, since some of the configuration parameters defined and documented in CBL per se are also needed for this automated build process.

The default parameters are suitable for an x86_64 (that is, 64-bit Intel or AMD) host system and an AArch64 (64-bit ARM) target system — the same as the default CBL configuration parameters.

Most of these parameters are similar to parameters defined in CBL, and are used to construct an appropriate QEMU command line for the host system.

The QEMU for the host build should be given as much memory and CPU as possible, considering the hardware it’s running on and any other processing that will be going on during the build. My main CBL development machine has 128 GiB of RAM and 20 CPU cores; I give the host QEMU 24 GiB of RAM and 12 cores, as with the default values here.

The HOST_QEMU parameters are similar to the CBL parameters that define how the target QEMU process will be constructed. The HOST_QEMU_MACHINE should specify the full -machine argument; if that argument needs to include directives to use KVM acceleration or something similar (as with the default value), include those.

By default, QEMU will be told to use the host CPU type. At least for Intel-architecture, this tells QEMU to inform the virtual machine that its CPU is of the same type as the actual physical CPU in the host computer; that avoids any kind of instruction emulation or transformation, so it’s the most efficient way to run QEMU.

And, similarly to the other parameters needed here, the virtual hardware to be simulated by the QEMU process and the device naming conventions for the host system must be specified.

Since we will eventually want to run the resulting Little Blue Linux system in QEMU and with networking enabled, we will need to know what network driver to use for it.

This process will be done in a number of phases — the first few will set up and execute the CBL process itself, and then a few additional things will be done to get the resulting artifacts all tidied up and organized.

Throughout this process, we’re going to make extensive use of a trick for automating interactions with virtual machines: we will run QEMU processes from ruby scripts. All of these will be run using the -nographic directive, which (as mentioned in the launch-qemu blueprint) will cause the standard input and output streams from the QEMU process to be mapped to a serial device in the virtual machine. Then we’ll use the ruby expect library to interact with the QEMU virtual machine through that serial console — everything sent to the standard input of the child process will effectively be typed into the serial console of the virtual machine, and all of the output produced by those commands will show up on the standard output of the child process. Ruby expect provides a facility similar to the expect TCL program that is a part of the base LB Linux system and used by the DejaGnu test framework; I’m using the ruby library, rather than expect per se, because I’m way more comfortable with ruby than TCL.

Using expect like this is a bit verbose, but not complicated. Basically, expect lets you tell ruby to examine all of the output arriving from some stream and wait until a specific pattern shows up. Once it does, you can examine what was written and respond accordingly. Mostly, we’re just detecting shell prompts and then running the next command in sequence.

The first time we do this will be to run the host side of the CBL process: we’ll run the host-system QEMU, copy a bunch of stuff to the virtual machine, and then use the serial console to run litbuild and run the host-side script.

If that process completes successfully, the script will copy the kernel and the build log files back to the computer where the script is running, and shut down the VM.

Then we’ll run a second QEMU process that is essentially the same as the one found in the launch-qemu blueprint; that will run the target side of the CBL process without any manual (or automated) interaction.

That’s all there is to CBL itself, but as mentioned earlier there are a few other things I like to do after the build is complete — and, as always, I like to avoid doing manual work whenever possible!

Since I want to be able to boot the resulting LBL system, and some builds don’t set up a boot loader, we’ll want to copy the kernel file out of the target system so we can use QEMU as a boot loader. And as with the default build process, we might as well copy the log files from the target side of the build out as well, so we have all the build logs consolidated in a single location in case we want to review them later on. This will be done using another expect-driven host-architecture QEMU.

A number of packages don’t have their test suites run during the CBL process, because of idiosyncratic issues described in their respective blueprints. So we’ll then use a third expect-driven QEMU to launch the target system and run the blueprint that will repeat those builds with the test suites.

And, last, since we use a copy-on-write image format for all of these QEMU processes, the final disk image that contains the full Little Blue Linux system will be a lot larger than it really needs to be, so we will remaster it using a fourth expect-driven QEMU process.

As with everything else in CBL — if any of that does not sound good to you, change it! This is your system. You can do whatever you want with it.

The first thing we need to do is set up disk image files that will be used by QEMU. For the host system, we’ll create an image that will be used for swap storage, and a working image based on the provided lbl.qcow2 image — that working image will be used as the root filesystem during the host-side build, so that the original lbl image will not be affected at all by the build (and can be a read-only file if desired).

We’ll also need images for what will become the root filesystem for the target system, and a swap volume for the target.

Some of the normal CBL configuration parameters — especially those dealing with directories to be used during the build — should always be set to specific values for this automated build, ignoring the normal default values and also ignoring whatever other values are specified earlier in the confiugration file. You may want to make some adjustments here — for example, you may want to modify the hostname or domain name used in the build. Rather than modifying the cbl-configuration.sh script in-place, we’ll concatenate a second file with overriding values to it and put the result in a new full-config.sh script; full-config.sh will be the configuration file actually used during the build.

For the host-system QEMU, we’ll use user-mode networking. This is a pretty limited facility — much more limited than virtual networking with a bridge and TAP interfaces as described in the setup-virtual-network blueprint — but does not require any elevated privileges and provides a handy TFTP server so files can be copied to and from the virtual machine without any extra work. Here, we’re using tftp=transfer, which will permit TFTP to copy files to or from the transfer directory.

Notice the use of the -nographic directive — this is key to interacting efficiently with the virtual machine. The other arguments are pretty typical.

Now let’s write the ruby script that will launch QEMU and drive it with expect. I like to see what’s going on in the host QEMU system, so I set $expect_verbose in the script. That causes all the console output to be written to the standard output of the script, rather than swallowed silently; we’ll direct it all to a log file.

Once QEMU is running, we need to wait for the network in the virtual machine to be available. We can tell that that’s the case by looking at the kernel messages written out through the serial console and noticing when the default route has been added.

Now we’re going to log in as root. Depending on how the boot process goes, we may have gotten a login prompt from the getty process running on the console already, or we might not; if we just hit return, getty will repeat its prompt, so we’ll do that.

Everything in the run-host-build script is designed to be idempotent. That is, if any command has already run and set things up as desired, it won’t be repeated in a way that will cause problems. So if the host build crashes partway through, you might be able to just run this script again and have it pick up where it left off. It’s hard to make any guarantees about that, but it’s at least worth a try.

Interacting with the host QEMU console through the expect facility is a little clumsy! Every command needs to be typed into the console through the I/O object that has been mapped to QEMU’s standard input, and after each command we need to call expect on the I/O where QEMU’s standard output is showing up. Otherwise, we can’t be sure that the script will wait long enough for the VM to process the command and provide a new shell prompt. Because of this clumsiness, I prefer to do as little interaction through that facility as I can get away with.

Accordingly, let’s write a couple more scripts that we can then run in the QEMU console.

First, we’ll want a script that sets up the disk volumes for the build: set up the swap volume if it’s not already set up (again, we want this to be idempotent), enable it, create a filesystem for the target system, mount it at the correct location, and make sure the lbl user can write to it.

You might notice the sync command there. We’ll run sync a few more times as well. If something goes wrong and the run-host-build script crashes, the QEMU process will get a kill signal as the ruby parent process is cleaned up. If the virtual machine has dirty pages that have not been written to the disk image files, this could potentially leave the filesystem in a bad or unexpected state. While I was writing and testing this blueprint, I had a lot of trouble figuring out why commands run through expect didn’t seem to have been completely processed, and had left lock files on the system that caused problems when restarting. Once I figured out that it was because QEMU was being killed before it flushed any buffers, it was trivial to fix by adding a sync command.

And second, we want a script that we can run as the lbl user to generate the CBL scripts and run the top-level script it produces.

Now let’s get everything onto the virtual host. Since we’ll be using TFTP for all the file copies, and it doesn’t provide any way to do recursive copies, we’ll create a tar file that contains everything we want to copy in to the VM. The -h directive tells tar to archive the files and directories that symbolic links point to, rather than the links themselves — when I set up build directories for use with this process, I often have symbolic links for the cbl and materials directories.

The curl program that is part of the basic LB Linux system can act as a TFTP client, so we can use it to copy the file in. When using user-mode networking, the host system is accessible by default at 10.0.2.2.

Everything we need for the build is now on the virtual machine, so we can run those handy scripts we just wrote to perform the host build.

If everything went well, we have a complete scaffolding environment on the target volume. If that’s the case, we should copy the scaffolding kernel and build logs off of the target volume, power down the host virtual machine, and proceed with the target side of the build.

If the host-side build did not run to completion, we’ll skip copying the scaffolding kernel and just power down the host virtual machine: the build failed, so the resulting system needs to be inspected to see how to proceed.

We can determine which of these happened by looking to see if the output includes the TOTAL SUCCESS line.

There’s one issue with just running poweroff and then waiting for the QEMU process to terminate: as long as the root console is active, the s6 shutdown process will hang. So we need to exit the root console as well.

Now that we’ve got that script written, we just need to run it.

If the host build succeeds, we’ll find the scaffolding kernel in the transfer directory, so we can put it where it belongs (and also take care of the other files that have been copied out to transfer). If not, we should bail out!

To run the target side of the CBL process, we need to run another QEMU process. This will be essentially the same command that is used in the default (real-computer host, QEMU-emulated target) CBL configuration, so this command is basically a copy of the one in the launch-qemu blueprint. If I were a better person, I’d refactor to have no duplication between blueprints.

As with the launch-qemu blueprint, if you haven’t built the host-prerequisites and are not using an LB Linux system, you’ll need to either build and install the yoyo program before you do this, or remove the reference to yoyo from this blueprint.

Once that QEMU command terminates, the entire CBL process is complete! The target-system/cbl.qcow2 disk image will contain the freshly-built Little Blue Linux system.