Make Linux servers fun again!

Persistence by Choice

The immutable nature of Lightwhale offers clear advantages,
but in order to install, configure, run containers, and write data, a writable filesystem is required.
And for the system to be genuinely useful,
such changes must persist across reboots.

The Data Filesystem

Lightwhale provides both temporary and persistent writability
through an automated subsystem activated early during startup.
This mounts the data filesystem at /mnt/lightwhale-data.

All data written by Lightwhale is kept within a single subdirectory:
/mnt/lightwhale-data/lightwhale-state.
This in turn serves as the writable upper layer in an

overlayfs
stack,
with the immutable root as the lower layer.

By default, Lightwhale mounts a volatile tmpfs
as its data filesystem.

When persistence is enabled,
the data filesystem instead resides on a storage device
and is mounted accordingly.

Key Directories

The data filesystem overlay does not cover the entire root filesystem;
that would defeat the purpose of immutability
and Lightwhale altogether.
Instead, the writable overlays apply only to a few strategic directories:

/etc

For customizing system configuration,
including networking, password, and sshd settings.

/var

For log and other application data.

/home

For user account customization,
including authorized SSH keys,
and cloning Git repositories with Docker and Swarm stacks.

Docker Data

Docker is configured with its data root directory
located directly on the data filesystem,
where all Docker runtime data is stored,
including images, containers, volumes, and network state:

/mnt/lightwhale-data/lightwhale-state/docker

Enable Persistence

Persistence must be enabled explictly
by writing the magic header
to the storage device to be used,
e.g. /dev/sdx:

echo "lightwhale-please-format-me" | sudo dd conv=notrunc of=/dev/sdx

Multiple storage devices are supported to have a magic header written,
and will be assembled into a Btrfs RAID1 volume.

The next time Lightwhale boots up,
it will detect the magic disk,
format it, and make it the
data filesystem.

Managing Persistence

The
persistence subsystem
is initiated from /etc/init.d/S11persistence,
and proceeds through a sequence of detailed steps, executed fully automatically:

1. Find data filesystem

Scan all disks for a partition with the filesystem label lightwhale-data.

If found, use it as the data filesystem and jump to step 6;
otherwise proceed to step 2.

2. Find magic disks

Scan all disks for the magic header,
specifically this exact byte sequence at the very start of the device:
lightwhale-please-format-me.

If found, treat each as a magic disk and proceed to step 3;
otherwise jump to step 6.

3. Create magic partitions

For each magic disk, create a swap partition labeled
lightwhale-swap,
then create a Linux partition that uses the remaining space and label it
lightwhale-data.
Then proceed to step 4.

4. Find magic partitions

Scan all disks for swap partitions labeled
lightwhale-swap
and Linux partitions labeled
lightwhale-data.
Treat each as a
magic swap partition
or
magic data partition
and proceed to step 5.

5. Create data filesystem

All magic swap partitions are formatted
and labeled lightwhale-swap.

If only a single magic data partition exists,
format it with
btrfs --data single --metadata dup.
In case of multiple,
join them into a RAID1 and format with
btrfs --data raid1 --metadata raid1cn.
Subvolumes are created for
@lightwhale-data,
@lightwhale-state,
and @lightwhale-state-snapshots.

Label the data filesystem lightwhale-data,
so it can be detected in step 1 at next startup.

6. Mount data filesystem and prepare state directory

If a data filesystem was created or found,
mount its subvolume @lightwhale-data
at /mnt/lightwhale-data;
otherwise mount a tmpfs instead.

7. Mount overlays

Prepare the immutable lower layer:
Bind mount /etc
on /run/lightwhale/overlay/lower/etc,
and mirror the entire directory tree of the immutable root filesystem.

Prepare the writable upper layer:
If not present, create a directory on the writable data filesystem at
/mnt/lightwhale-data/lightwhale-state/overlay/upper/etc.

Finally use overlayfs to virtually merge
the two layers and mount the overlay filesystem at /etc.
This effectively replaces the immutable directory
with a writable version!

Repeat for remaining key directories /var and /home.