To recap, what I want to achieve during the Fellowship is to build an immutable image-based OSTree system with Garden Linux.
The first milestone in reaching this goal is to get a Garden Linux system to boot with OSTree. This does not need to be elegant, I just want to see it work so we can identify what needs to be done do it properly.
This sounds easier than it is.
First contact with OSTree
Making an initial OSTree repository is quite simple, I've done that in the past and I've even contributed the OSTree tl;dr page to the tldr-pages project. Side-note: If you don't know tldr-pages, check it out, it's a great resource for looking up how to use cli tools. I use it on a daily basis.
But building a bootable system with OSTree is much more complex.
As someone who was not involved in making those decisions, I think they show a lot of pragmatism. Debian is a rock solid base operating system that is being developed by an open source community and supported by a non profit organization. Garden Linux takes this foundation and swaps out parts that make sense for it's purpose. And it is available to the public, as open source software, allowing anyone to use it for their own purposes.
Because I could not find a OSTree builder that would work with Garden Linux, I decided to start from scratch.
Building a Garden Linux OSTree system
The goal was to get a Garden Linux system booting using OSTree, somehow.
OSTree does generate BLS entries, and with that a system can boot without an extra bootloader.
This does not look too bad.
But it does not boot successfully. I've had various issues while trying to boot this deployment, and I'll get to them later.
First, we need to investigate a question that became ever more apparent as I've continued my journey.
Well, how does Linux boot anyway?
This question took me down an interesting rabbit hole.
I have to clarify that we are only looking at UEFI-based systems here, and only at systemd-boot or the UEFI bootloader.
UEFI systems have a small FAT partition, the EFI System Partition. This contains an executable file in the PE format (actually, that's a Windows executable, regardless of the Operating System you are booting).
The bootloader is started by the UEFI firmware, and the bootloader then loads the kernel and the initramfs. The initramfs is a small filesystem that contains the initial root filesystem and the init system, in most modern Linux systems this is systemd. The init system then mounts the real root filesystem and switches to it.
This is a simplified flowchart of the boot process:
OSTree needs a modified boot procedure. For this, it includes a program that is being packaged in the initial ramdisk. This program is called ostree-prepare-root and it is responsible for mounting the OSTree root filesystem.
During the boot process, orchestrated by systemd, the ostree-prepare-root program is called and does what's needed by OSTree.
Accounting for the OSTree specific changes, this is how our flowchart looks like now:
We can also observe this in the journal of our CentOS OSTree system.
Once the system is booted, we can have a look at the mounted filesystems. Ha, that's unusual. We have three partitions, so far so expected. We have the efi partition as described above. We have a separate boot partition which contains the kernel and the initramfs. And we have a root partition which is mounted multiple times with different mount options. This is what gives us the properties of the OSTree system, where /usr is read-only and /etc is read-write.
Current status and next steps
During the last three weeks, I've made a lot of progress.
I've spent quite some time debugging, looking into the source code of OSTree, comparing my Garden Linux system to the CentOS system, analyzing the build logs of the CentOS vm, even injecting my own dracut module into the initrd for debugging purposes.
Over time, I've fixed errors in my build script as I discovered them.
Since we learn from errors, let's have a look at some of them:
Somehow I managed to create a cyclic link from /sysroot/ostree pointing to /sysroot/ostree
I'm still not sure if I got the relation between the sysroot, repo and ostree directories right, this lead to multiple failed attempts to build and boot the system
I had wrong parameters in the ostree deploy command leading to invalid BLS entries
I have to admit that creating a bootable OSTree system is harder than I anticipated, and both the documentation and the existing build scripts were hard to understand for me at times. I did open an issue in the ostree repo about the documentation, let's see if I can contribute something to that after my own understanding has improved.
This is what currently happens when I try to boot my Garden Linux OSTree system, and I don't know yet how to fix this:
It feels like I'm close to reaching the first milestone, a minimal bootable Garden Linux system based on OSTree, but I'm not there yet.
I'll keep you updated on my progress in this blog post series.
Pick of the month
I've came across this blog post by Colin Walters, the maintainer of OSTree. The article is a bit long, but I think it is the most comprehensive explanation of the benefits of immutable image-based systems I've read so far. I highly recommend reading it.
If you're interested in the topic, feel free to comment this blog post or reach out to me on LinkedIn.