frood, an Alpine initramfs NAS

My NAS, frood, has a bit of a weird setup. It’s just one big initramfs containing a whole Alpine Linux system. It’s delightful and I am not sure why it’s not more common.

• As long as the firmware can find the image, the machine comes up cleanly.
• Deploying a new version is just copying a single file.
• The system is defined declaratively in the git repo that builds the initramfs.
• Importantly to me, it’s not defined in some complex DSL: if I want a file to exist at /etc/example.conf I put it in root/etc/example.conf, and the rest is done by a few hundred lines of scripts I can (and have) read.
• Configuring it doesn’t look any different than configuring any regular Alpine system.
• I can test the next deploy with a qemu oneliner.
• There are very very few moving parts.

If this already sounds appealing, you can skip to the “How it works” section below.

I have updated this post for my 2025-onwards setup, which moved to a 64-core
Ampere Altra and removed the bootloader in favor of a single-file UKI.

But why

I’ve always liked running systems from memory: it’s fast and prevents wear on the system storage device, which is often some janky SD card, because the good drives are dedicated to the ZFS pool.

However, you immediately have the problem of how to persist configuration changes.

Alpine’s answer to this is “diskless mode” where any customization is kept in an overlay file. After boot, the stock system looks for a file matching *.apkovl in all available filesystems, applies it, and then installs any missing apk packages from a local cache.

The first problem with that is complexity: the tool to generate and manage the apkovl, lbu(1), is pretty good but that process has a lot of moving parts. Find the apkovl, apply it, mount the filesystems in the new fstab, install the missing apks, resume the boot process. Over the past year, I had this break multiple times, either because it couldn’t find the filesystem anymore or because the apks did not get installed. The boot process depends on the package manager!

The second problem is that I would really like the state of the system to be tracked in git. Graham Christensen has a very good pitch for declarative or immutable systems in “Erase your darlings”.

I erase my systems at every boot.

Over time, a system collects state on its root partition. This state lives in assorted directories like /etc and /var, and represents every under-documented or out-of-order step in bringing up the services.

“Right, run myapp-init.”

These small, inconsequential “oh, oops” steps are the pieces that get lost and don’t appear in your runbooks.

“Just download ca-certificates to … to fix …”

Each of these quick fixes leaves you doomed to repeat history in three years when you’re finally doing that dreaded RHEL 7 to RHEL 8 upgrade.

“Oh, touch /etc/ipsec.secrets or the l2tp tunnel won’t work.”

I used to solve that by making (most) changes via Ansible, but then I had a multi-layer situation where I needed to make a change in Ansible, then deploy it, then save it with lbu to the apkovl.

There are of course many alternatives for declarative systems: from NixOS (which just doesn’t sound fun) to gokrazy (which is not quite ready to ship ZFS) to embedded toolchains like buildroot or the newer u-root.

Thing is though, I really like Alpine: a simple, well-packaged, lightweight, GNU-less Linux distribution. What I don’t like are its init and persistence mechanisms.

How it works

When it boots, Linux expects an “initramfs” image. It’s a simple cpio archive of the files that make up the very first root filesystem at boot. Usually the job of this system is to load enough modules to mount the real rootfs and pivot into it. Nothing stops us from putting the entire system in it, though! Who needs a rootfs?

Building an initramfs

The starting point is alpine-make-rootfs, which is a short (~500 lines) script meant to build a container image. It’s really 90% of what we need.

#!/bin/sh
set -e

wget https://raw.githubusercontent.com/alpinelinux/alpine-make-rootfs/v0.7.0/alpine-make-rootfs \
    && echo '91ceb95b020260832417b01e45ce02c3a250c4527835d1bdf486bf44f80287dc  alpine-make-rootfs' | sha256sum -c && \
    chmod +x alpine-make-rootfs

ROOTFS_DEST=$(mktemp -d)

# Stop mkinitfs from running during apk install.
mkdir -p "$ROOTFS_DEST/etc/mkinitfs"
echo "disable_trigger=yes" > "$ROOTFS_DEST/etc/mkinitfs/mkinitfs.conf"

export ALPINE_BRANCH=edge
export SCRIPT_CHROOT=yes
export FS_SKEL_DIR=root
export FS_SKEL_CHOWN=root:root
PACKAGES="$(cat packages)"
export PACKAGES
./alpine-make-rootfs "$ROOTFS_DEST" setup.sh

alpine-make-rootfs will copy the files from the root directory, install the packages from the packages file, and run the setup.sh script in a chroot.

Then we package everything but the boot directory (which holds the kernel) into an initramfs archive, dropped right next to the kernel.

cd "$ROOTFS_DEST"
find . -path "./boot" -prune -o -print | cpio -o -H newc | gzip > "$ROOTFS_DEST/boot/initramfs-lts"

That’s truly very nearly it! It’s impressive how Alpine lends itself to this with practically no hacks.

Packages

The packages we install are the usual stuff you’d install on a server. Only a few are noteworthy.

• alpine-base is the metapackage that installs apk, busybox, openrc, and a few config files.
• linux-lts is the kernel, along with its modules. I considered thinning down the modules to only the ones I needed, but it’s ultimately a lot of hacks just to save a couple hundred MB. Note there is no modloop! The modules are always available.
• linux-firmware-none provides linux-firmware-any, otherwise linux-firmware gets installed, which installs hundreds of megabytes of Linux firmware.

Setup script

The setup.sh script is also nothing special. We set up the run-levels and the root password. (Yes, that’s my actual password hash. No you won’t break it.)

#!/bin/sh
set -e

rc-update add devfs sysinit
rc-update add dmesg sysinit

rc-update add hwclock boot
rc-update add modules boot
rc-update add sysctl boot
rc-update add hostname boot
rc-update add bootmisc boot
rc-update add syslog boot
rc-update add klogd boot
rc-update add networking boot
rc-update add seedrng boot

rc-update add mount-ro shutdown
rc-update add killprocs shutdown

rc-update add acpid default
rc-update add crond default
rc-update add local default
rc-update add openntpd default
rc-update add sshd default
rc-update add tailscale default

chpasswd -e <<'EOF'
root:$6$twsDxnP.TG2M8J4l$7lte7E/ImK4UwoursD7qQCC7XMUothIDb9FTH1MncxYbGQDUQPkC/9pxleTwPxEs3nbatApszxuwc4yj6ucdX1
EOF

In practice I set up a few more services here, but they are not needed to run the system. This is just where you declaratively specify how the system is configured.

Root skeleton

The root skeleton is similarly system-specific, and it’s so nice to be able to drop files into the image just by creating them. For example, if I want something to run at boot, I just add a file to root/etc/local.d/.

A few noteworthy files in the skeleton.

::sysinit:/sbin/openrc sysinit
::sysinit:/sbin/openrc boot
::wait:/sbin/openrc default
::shutdown:/sbin/openrc shutdown

tty1::respawn:/sbin/getty 38400 tty1
ttyAMA0::respawn:/sbin/getty -L 115200 ttyAMA0 vt100

root/etc/inittab is where busybox’s init picks things up: it runs the OpenRC run-levels and respawns a getty on tty1 and the serial console.

root/etc/network/interfaces and root/etc/hostname and root/etc/hosts get the network to work.

root/etc/ssh/ssh_host_ed25519_key and root/etc/ssh/ssh_host_ed25519_key.pub and root/root/.ssh/authorized_keys for obvious reasons.

sshd_disable_keygen=yes

root/etc/conf.d/sshd avoids generating non-Ed25519 host keys.

Finally, a bit of persistence for the two things that truly can’t do without it: the RNG seed (arguably not necessary with hardware randomness) and Tailscale (which really doesn’t know how to run without persistence, alas). Rigorously UUID mounted.

UUID=B61B-19E7   /media/usb   vfat   noatime,rw,fmask=177 0 0

root/etc/fstab

seed_dir=/media/usb/persist/seedrng

root/etc/conf.d/seedrng

TAILSCALED_OPTS="-state /media/usb/persist/tailscaled.state"

root/etc/conf.d/tailscale

Bootloader UEFI

In 2024 there was a section about bootloaders here. I have since switched to a
Unified Kernel Image assembled by ukify and loaded directly by UEFI from the
removable-media default path EFI/BOOT/BOOTAA64.EFI.

apk add --no-cache systemd-efistub ukify

# The default rdinit is /init, while the default init is /sbin/init.
CMDLINE="rdinit=/sbin/init console=tty1 console=ttyAMA0"

ukify build --output "$1" --cmdline "$CMDLINE" \
    --linux "$ROOTFS_DEST/boot/vmlinuz-lts" \
    --initrd "$ROOTFS_DEST/boot/initramfs-lts" \
    --os-release "@$ROOTFS_DEST/etc/frood-release"

Deploy is just

rsync images/$image root@frood:/media/usb/EFI/BOOT/BOOTAA64.EFI

qemu testing

Here’s something beautiful about this setup: you can meaningfully test it in qemu by just pointing it at the UKI image.

qemu-system-aarch64 -M virt -cpu cortex-a72 -m 4G -nographic \
    -bios QEMU_EFI.fd -kernel "images/$image" \
    -drive file=usb_disk.img,if=virtio,format=raw

This includes a persistence device that I formatted with the same UUID as the production one. ^{Since Tailscale configuration is in there, the qemu image comes up as a different Tailscale device, and I can SSH into it separately.}

A simple status service

I wanted a simple service to get the status of the system at a glance. There are a million ways to do this, but I chose to write a small Go server. It’s not needed to make this system work, but I am including it to show how easy it is to add a service.

Before the alpine-make-rootfs invocation, I added a couple lines to build all Go binaries in a local module into /usr/local/bin/. Note that even the Go toolchain is selected declaratively from the go.mod thanks to GOTOOLCHAIN=auto.

go env -w GOTOOLCHAIN=auto
go build -o "$ROOTFS_DEST/usr/local/bin/" ./cmd/...

Then I created root/etc/init.d/srvmonitor.

#!/sbin/openrc-run
# shellcheck shell=sh

description="Serve scripts from /etc/monitor.d"
command=/usr/local/bin/srvmonitor
command_background=true
pidfile="/run/$💬.pid"

depend() 💬

And finally I added one line to setup.sh.

rc-update add srvmonitor default

That’s it. The Go server listens on port 80 on the Tailscale IP, and serves the output of scripts I put in /etc/monitor.d/.

frood

The entire setup is open source, in my mostly-harmless repository. You might be interested in how I made ZFS imports work, which is not covered above.

I have not made it into a reusable project partially because there is so little to it. Adding hooks to configure things would easily double its size. I encourage you to just fork it if you’d like.

One thing I haven’t solved yet is how to inject secrets. For now ZFS unlocks with passkeys and the rest are just .gitignore’d. Maybe I’ll plug in a YubiKey and use age-plugin-yubikey to decrypt them, and yubikey-agent for the host key. Or maybe this board has a TPM and I can use the simplicity of this system to get a full Secure Boot chain that unlocks TPM keys. That’d be fun.

If you got this far, you might also want to follow me on Bluesky at @filippo.abyssdomain.expert or on Mastodon at @filippo@abyssdomain.expert.

The picture

The natural pools of Porto Moniz, in Madeira. They’re publicly accessible, made of volcanic rock, and filled by the ocean waves that crash spectacularly against them. I was not doing great that day, but it was an excellent place to not do great at.

Madeira is pretty cool. ^{Also one of the trickiest crosswind landings.}

My maintenance work is funded by the awesome Geomys clients: Interchain, Smallstep, Ava Labs, Teleport, SandboxAQ, Charm, Tailscale, and Sentry. Through our retainer contracts they ensure the sustainability and reliability of our open source maintenance work and get a direct line to my expertise and that of the other Geomys maintainers. (Learn more in the Geomys announcement.)

Here are a few words from some of them!

Teleport — For the past five years, attacks and compromises have been shifting from traditional malware and security breaches to identifying and compromising valid user accounts and credentials with social engineering, credential theft, or phishing. Teleport Identity is designed to eliminate weak access patterns through access monitoring, minimize attack surface with access requests, and purge unused permissions via mandatory access reviews.

Ava Labs — We at Ava Labs, maintainer of AvalancheGo (the most widely used client for interacting with the Avalanche Network), believe the sustainable maintenance and development of open source cryptographic protocols is critical to the broad adoption of blockchain technology. We are proud to support this necessary and impactful work through our ongoing sponsorship of Filippo and his team.

SandboxAQ — SandboxAQ’s AQtive Guard is a unified cryptographic management software platform that helps protect sensitive data and ensures compliance with authorities and customers. It provides a full range of capabilities to achieve cryptographic agility, acting as an essential cryptography inventory and data aggregation platform that applies current and future standardization organizations mandates. AQtive Guard automatically analyzes and reports on your cryptographic security posture and policy management, enabling your team to deploy and enforce new protocols, including quantum-resistant cryptography, without re-writing code or modifying your IT infrastructure.

Charm — If you’re a terminal lover, join the club. Charm builds tools and libraries for the command line. Everything from styling terminal apps with Lip Gloss to making your shell scripts interactive with Gum. Charm builds libraries in Go to enhance CLI applications while building with these libraries to deliver CLI and TUI-based apps.