Custom Discovery Images for SecureBoot

In the last article “orcharhino and SecureBoot” I explained what SecureBoot is, how it works and what options there are for implementing it in orcharhino. Various options and solutions were shown at the end, with option 2 being discussed in more detail here. I recommend reading this blog post beforehand to better understand the following.

The Foreman Discovery Image

The Foreman Discovery Image (FDI) is a “small RedHat-based image which is loaded into the working memory via PXE, initializes all network interfaces and executes a script called “discovery-register” via systemd.” (see [1]). This script contacts the orcharhino server via a provided URL to the orcharhino server and uploads the host facts there in order to enter the host in the orcharhino server as a discovered host. If the host is then known to the orcharhino server, an installation of the target distribution can be started using kexec [2]. This type of installation does not require a reboot.

Ready-made FDIs can be obtained from Index of /discovery/releases .

These FDIs already support the extension of additional scripts/functionality at runtime, in which ZIP archives can be reloaded from a TFTP or HTTP server, for example.

Alternatively, an FDI can also be built locally.
Reasons for this are, for example, more up-to-date package versions or a more up-to-date kernel for the support of newer hardware.

With the upstream repository [3], anyone can build an FDI locally, but with the following two restrictions:

the built environment must be an Enterprise Linux 8 (EL) system, e.g. RedHat Enterprise Linux 8, Rocky Linux 8, or AlmaLinux 8
the resulting FDI is a corresponding EL-based system

In the context of SecureBoot, the latter means in simple terms that only one installation of the same distribution can be started using kexec because the kernel is in lockdown (see [4]).

This means that other distributions such as Ubuntu, Debian, or SUSE cannot be installed with SecureBoot activated in combination with an EL-based FDI.

Based on my experience with previous projects, I came up with a tool that can overcome these limitations: KIWI NG.

KIWI NG

KIWI Next Generation (KIWI NG) is an open source tool for creating custom images. It provides a command line program and uses configuration files that allow users to define the components and settings of a desired image. KIWI NG supports various image formats, including ISO and virtual hard disk images. It simplifies the process of creating reproducible and customized images for various purposes, such as installation media or deployment in virtualized environments. KIWI NG is used in the openSUSE Build Service (OBS) [5], among others.

KIWI NG can be installed on various Linux distributions either via the respective package manager [6] or via PyPI [7].

To run KIWI NG in a container, extended privileges are required, e.g. for the required mount syscalls.

A KIWI image is configured using the following files/directories:

An XML-based configuration file that defines various aspects of the image, such as the base operating system, packages to be installed, the output format (e.g. an ISO image), the file system structure, etc.
A bash script which is executed in the target image and can make adjustments such as the configuration of services.
A root directory with all additional files that should later exist in the root directory of the image.

It is recommended to configure your user-defined image based on sample configurations (see [8]), the so-called image descriptions.

In order to be able to start an Ubuntu installation, e.g. Ubuntu 22.04, via kexec from an FDI with SecureBoot activated, we need an Ubuntu-based FDI.

Build your own Foreman Discovery Image with KIWI NG

After analyzing the finished FDIs from Upstream and the EL-based build scripts, it became apparent that an FDI basically consists of a Foreman Proxy and additional scripts and services. A search revealed that the Foreman Proxy is available as package(s) for Ubuntu 20.04 (Focal) under Index of /dists/focal . Thus the choice of the base operating system also fell on Ubuntu 20.04 (Focal), the desired image format is an ISO. The XML-based configuration files were expanded to include the repository URL, the foreman-proxy package and other packages (see [9]).

The configuration of the image (see [10]) can largely be transferred to the bash script.
Among other things, all required services are configured and activated there so that they are active when the FDI boots.

All additional scripts that cannot be installed via packages are copied to the root directory. This includes the discovery menu, which is displayed at system startup and allows manual operation of the FDI at runtime.

Once you have everything together, the image can be built:

# kiwi-ng --debug system build --description ubuntu/x86_64/ubuntu-focal --target-dir /root/fdi-image

The build takes up to 30 minutes, depending on the build environment. The resulting ISO can now be used as a boot medium, in the simplest case provided as a virtual CD/DVD device. After the Ubuntu-based FDI has contacted the orcharhino server, an Ubuntu 22.04 installation can be started via kexec. This is possible because the FDI booted via shim (see [14]) holds a list of trusted keys. Simplified, this always includes the current manufacturer keys (e.g. from Canonical Ltd.), which were also used to sign the installation kernel.

The procedure described here has been summarized in a very simple way. The result for rebuilding an FDI using KIWI NG has been published on GitHub [11].In addition to Ubuntu, there is also an image description based on CentOS and AlmaLinux. The Ubuntu image description requires the HWE kernel [12] to ensure the latest hardware support.

By the way: The OBS cannot currently be used for building the FDI without further ado as access to external repositories (here for the Foreman packages) is blocked. You could try to build and package the required Foreman packages in OBS as well in order to be able to use them.

Deploy Foreman Discovery Image via PXE

FDIs are used in PXE environments (network boot) using DHCP and TFTP servers. For a standard setup (see [13]) in a PXE environment, a tar archive is used instead of the ISO file. In addition to the FDI kernel, this tar archive also contains an initrd, which contains the complete FDI root directory, and is created as an additional artifact when building with the upstream repository (see above).

With KIWI NG, we only receive the ISO file as an artifact. The initrd contained therein does not contain the complete FDI root directory but can download the ISO file via HTTP, mount the root directory contained therein, and start the system.

Incidentally, this variant is supported by all FDIs and offers considerable time savings in PXE environments with TFTP, as the FDI root directory, which is several hundred megabytes in size, is not loaded via the slow TFTP protocol but via the HTTP protocol.

The preparation on the respective orcharhino proxy is as follows:

mount /tmp/fdi.iso /mnt/
mkdir /var/lib/tftpboot/boot/fdi-image-slim
cp /mnt/isolinux/{vmlinuz,initrd.img} /var/lib/tftpboot/boot/fdi-image-slim/
umount /mnt
mkdir /var/www/html/pub/fdi-image-slim
mv /tmp/fdi.iso /var/www/html/pub/fdi-image-slim

The ISO is mounted, the kernel and Initrd are copied to the TFTP directory. The ISO itself is moved to the HTTP directory so that it can be downloaded later by Initrd.

Finally, the PXE template must be adapted (pxegrub2_discovery snippet):

common_slim="rootflags=loop root=live:http://YOURPROXY/pub/fdi-image-slim/fdi.iso rootfstype=auto ro rd.live.image acpi=force rd.luks=0 rd.md=0 rd.dm=0 rd.lvm=0 rd.bootif=0 rd.neednet=0 nokaslr nomodeset proxy.url= proxy.type=proxy BOOTIF=01-$net_default_mac ip=dhcp"

menuentry 'Foreman Discovery Image EFI Slim' --id discovery {
  linuxefi boot/fdi-image-slim/vmlinuz ${common_slim}
  initrdefi boot/fdi-image-slim/initrd.img
}

This entry allows booting an FDI with a small initrd and reloading the root directory from the ISO file (see root=live: Parameter).It is also conceivable to use several FDIs in parallel that are based on different distributions.

Conclusion

Ready-made FDIs can be downloaded from online sources. Building FDIs yourself enables the use of more up-to-date package versions including the kernel, which contributes to better hardware support. If SecureBoot is activated, you must build a corresponding FDI for each distribution to be installed, which is based on the respective distribution.

Building FDIs, especially for non-EL-based distributions, is possible with KIWI NG. Using the image descriptions from [11], anyone can relatively easily build an FDI locally that is based on Ubuntu, CentOS, or AlmaLinux and thus fulfills the respective restriction regarding SecureBoot.

[1]: https://github.com/theforeman/foreman-discovery-image

[2]: https://linux.die.net/man/8/kexec

[3]: https://github.com/theforeman/foreman-discovery-image

[4]: https://wiki.debian.org/SecureBoot#Secure_Boot_limitations

[5]: https://build.opensuse.org/

[6]: https://osinside.github.io/kiwi/installation.html#installation-from-obs

[7]: https://pypi.org/project/kiwi/

[8]: https://github.com/OSInside/kiwi-descriptions

[9]: https://github.com/theforeman/foreman-discovery-image/blob/master/20-packages.ks

[10]: https://github.com/theforeman/foreman-discovery-image/blob/master/22-discovery.ks

[11]: https://github.com/ATIX-AG/foreman-discovery-image-kiwi

[12]: https://www.thomas-krenn.com/de/wiki/Ubuntu_LTS_Hardware_Enablement_Stack