pg83/solo: Portable Linux binaries, solved · GitHub

Ship one musl-linked executable. At runtime, load the user’s existing
glibc-linked GPU driver. No container, no AppImage, and no second libc in the
process.

Static binaries are a wonderfully boring way to deploy software on Linux: one
file, no dependencies, nothing to break. We build ours with
IX, a source-first build system for producing
fully static Linux binaries. The boredom ends the moment the application needs
the GPU: Vulkan and OpenGL drivers are supplied by the host as shared objects,
usually built against glibc, and a fully static musl binary cannot normally
dlopen() them.

SoLo crosses that boundary. It provides a dlfcn-style source API backed by
its own ELF loader (x86-64 and aarch64) and a glibc ABI bridge implemented on
top of musl.
The result is still one ordinary static executable, but it can use the graphics
driver already installed on the machine.

The repository includes an end-to-end Vulkan proof: a fully static executable
loads the host’s unmodified Vulkan driver, runs a compute shader, and writes
the result to a PNG. Tested on AMD radv, radeonsi, Intel, and NVIDIA GPUs under
Linux, and on Apple M1 under Asahi Linux.

The host keeps the hardware-specific code. You ship everything else.

And not on a demo’s word alone: on every commit, CI loads the shared
libraries of the 1,000 most-installed Debian packages — over 2,100 host
objects — through SoLo, on both x86-64 and aarch64.

Grab the prebuilt binary — no clone, no toolchain, any Linux with a Vulkan
driver installed (mesa-vulkan-drivers is enough):

curl -LO https://github.com/pg83/solo/releases/latest/download/vulkan-x86_64
chmod +x vulkan-x86_64
./vulkan-x86_64 hello.png

vulkan-aarch64 is the same demo for arm64 machines. The command discovers
the distro-installed Vulkan ICD in the usual way and produces a 512×512 RGBA
image. This is how we build the
Shitty release binaries—a blazingly
fast terminal emulator, BTW! To force a particular driver:

./vulkan-x86_64 --driver /usr/share/vulkan/icd.d/radeon_icd.x86_64.json radeon.png
./vulkan-x86_64 --driver /usr/share/vulkan/icd.d/lvp_icd.json lavapipe.png

ICD manifest names vary slightly between distributions. Passing no --driver
lets the embedded Khronos loader perform its normal discovery.

You can verify that the executable itself is not dynamically linked:

readelf -lW ./vulkan-x86_64 | grep INTERP    # no output
readelf -dW ./vulkan-x86_64                  # "There is no dynamic section"

Or build the same demo from source, with Python 3 and a C/C++ compiler in
PATH:

git clone https://github.com/pg83/solo.git
cd solo
./build vulkan
./vulkan hello.png

This is not a toy call to vkCreateInstance. The demo:

1. enters the statically linked Khronos Vulkan loader;
2. loads the host’s Vulkan ICD and its non-glibc dependencies through SoLo;
3. creates a device, storage buffer, descriptor set, and compute pipeline;
4. dispatches a checked-in SPIR-V shader;
5. maps the result and writes it through statically linked libpng.

The complete example is in bin/vulkan, and the Vulkan program
itself is in main.cpp.

┌──────────────────── fully static executable ────────────────────┐
│                                                                 │
│  application → embedded Vulkan loader → SoLo dlopen/dlsym       │
│                                           ├─ x86-64 ELF mapper  │
│                                           └─ glibc ABI → musl   │
│                                           │                     │
└───────────────────────────────────────────┬─────────────────────┘
                                            │ maps at runtime
                                            ▼
                              system Mesa/Vulkan ICD.so + DSOs

elf_loader.cpp maps ELF segments, walks DT_NEEDED,
resolves versioned symbols, applies x86-64 relocations, supports ELF TLS and
TLSDESC, materializes IFUNCs, applies RELRO, and runs initializers. Dependencies
that are themselves ELF DSOs are loaded recursively.

glibc is deliberately not loaded. Imports such as malloc@GLIBC_2.2.5 are
resolved by glibc_shim.cpp to ABI-correct adapters over
the process’s existing musl runtime. Unsupported glibc functions have unique
generated stubs that fail loudly with the exact symbol and version if they are
ever called, instead of silently corrupting the process.

Because musl sizes its synchronization objects to the glibc ABI of each
architecture, the bridge does not shadow them: a pthread_mutex_t a driver
creates is used in place. A lock is therefore one lock for both the loaded DSO
and the static executable that may share it, and glibc’s static recursive and
error-check initializers are adopted on first use.

Before loading a DSO from disk, SoLo checks its static provider registry. This
lets an application satisfy a dependency—Wayland, for example—with functions
already linked into the executable. LD_LIBRARY_PATH and
DL_ELF_LIBRARY_PATH are honored for libraries outside the standard system
directories.

The interesting pieces are small enough to read:

The default target builds the standalone archive:

The published ./dlfcn symlink points to the resulting libdlfcn.a. Include
lib/dlfcn.h, link the archive into a musl-static application,
and ordinary dlopen()/dlsym() calls are redirected to SoLo. The source tree
is intentionally self-contained and suitable for copying into another static
build graph.

./build test          # load an Arch glibc DSO closure in the smoke test
./build vulkan_test   # build the static demo and verify a native Lavapipe PNG

CI performs the native build and test on Alpine/musl with GCC, Fedora with GCC,
and Ubuntu with Clang. The Vulkan test installs each distribution’s own
Lavapipe package; it does not run the driver from an Arch sysroot.

Every build input for the standalone Vulkan executable is vendored under
bin/vulkan. build.py compiles those sources directly: upstream
CMake, Meson, configure, and Make build systems are not invoked.

In the general case, only SoLo lets a static application tell the dynamic
loader: “for this system DSO’s libwayland dependency, use the symbols already
linked into my executable.” This lets the application embed the newest
libwayland instead of targeting the oldest version available on every
supported system.

And the boundary between the two worlds is not a thin dlsym shim — it carries
the parts that make foreign code actually behave:

• C++ exceptions cross it in both directions. A throw in the static world
  unwinds through glibc-compiled frames into a glibc catch, and the other
  way around, destructors running on both sides: the guests’ _Unwind_*
  imports are bound to the one unwinder in the executable, so there is a
  single exception machinery in the process instead of two fighting ones.
• All four TLS models, without wrappers or code patching. General- and
  local-dynamic through __tls_get_addr, TLSDESC through its custom-ABI
  resolver, and initial-exec — whose GOT slots are plain
  thread-pointer-relative offsets no loader can intercept — served from a
  surplus arena that rides in the executable’s own static TLS, so one
  process-wide offset is valid in every thread and unmodified musl does the
  per-thread layout.
• ld.so‘s binding semantics, not an approximation. Global-scope
  interposition, RTLD_DEEPBIND, DT_SYMBOLIC, symbol versioning with the
  unversioned-provider compatibility rule, lazy PLT binding with the
  argument registers preserved through the resolver, GNU and SysV hash
  lookups, ifunc resolvers handed their hwcaps, /etc/ld.so.cache.
• Cross-world introspection. backtrace() walks static and glibc frames
  alike and names both through one dladdr; dl_iterate_phdr, dladdr1,
  and the link_map facade let unwinders and profilers see every image; the
  file-backed mappings keep real paths in /proc/self/maps for debuggers.
• The stateful corners of glibc, for real. getcontext /
  makecontext / swapcontext in assembly against glibc’s mcontext
  layouts on both architectures, the pre-2.34 pthread ABIs, GNU obstacks,
  the fortified _chk family, and the inline-stdio ABI — musl’s FILE is
  deliberately laid out so glibc’s inlined putc_unlocked compiles against
  it — down to _IO_2_1_stdout_ resolving to musl’s own stream.

Every one of these is exercised by a conformance battery compiled against
real glibc headers at -O2, and by loading every shared object of the
thousand most-installed Debian library packages in CI, on x86-64 and
aarch64.

• gcompat is a distribution-level
  glibc API shim for running prebuilt glibc binaries on musl. Its loader stub
  re-executes the program through musl’s dynamic linker with libgcompat.so
  preloaded; using it from a musl program requires linking that shared library
  or adding it to the loaded DSO’s DT_NEEDED. It does not give a fully static
  musl process a dynamic loader. SoLo’s self-contained model is stronger: the
  executable embeds both the ELF loader and ABI bridge, loads unchanged host
  DSOs without a system compatibility package, preserves the versions of their
  glibc imports, and lets unused unsupported functions remain behind
  symbol-specific, fail-loud stubs instead of blocking the entire DSO.
• Detour bootstraps the system’s
  ld-linux and allows multiple C runtimes to coexist. SoLo takes the opposite
  route: it maps the required DSOs itself and translates their glibc imports
  onto musl, so a second libc and its TLS state never enter the process.
• Cosmopolitan Libc’s cosmo_dlopen()
  follows the same split-runtime scheme as Detour, with all of its advantages
  and drawbacks: it bootstraps the host’s ELF interpreter and libc, then
  delegates loading the target DSO to the host’s dlopen().
• ClickHouse’s experimental userspace dynamic loader
  currently maps ELF objects itself, but stops short of loading glibc. Its
  proposed path to real-world system libraries such as CUDA is Detour-like:
  bootstrap the system’s ld.so, keep a second libc runtime, and swap the
  musl/glibc thread pointer at every boundary. SoLo instead implements the
  glibc ABI over the host’s musl runtime and can satisfy DSO dependencies from
  providers already linked into the static executable.
• graphics.gd’s musl + dlopen experiment
  follows the same split-runtime model as Detour: an embedded helper brings in
  the host’s glibc loader, and assembly trampolines switch between musl and
  glibc TLS around foreign calls. This leaves two independent TLS worlds: every
  boundary crossing needs a trampoline, and a callback implemented in musl
  cannot be passed safely to glibc code because glibc invokes it while its own
  TLS is active. SoLo keeps a single musl TLS world instead.
• Flatpak, AppImage, and containers solve the problem by hiding a small Linux
  distribution inside or around your program. This works in roughly the same
  way that moving house solves a missing power adapter. The result is a huge
  blob full of duplicated libraries, mounts, namespaces, extraction tricks,
  and runtime indirection—all of which make profiling, debugging, and basic
  introspection worse. Shipping a distro because you need one system .so is
  not portability. SoLo ships one normal, inspectable executable and borrows
  the only component that genuinely belongs to the host: its hardware driver.

• Linux only, on x86-64 and aarch64. The loader, the TLSDESC and lazy-PLT
  resolvers, and the initial-exec arena cover both; the glibc symbol
  inventories are generated per architecture, so printf@GLIBC_2.2.5 on one
  is printf@GLIBC_2.17 on the other without a single translation rule in
  the code;
• focused on real Mesa/Vulkan ICD dependency closures, and driven by the top
  1000 Debian library packages by popcon votes: the 885 of them that ship
  glibc-linked shared objects — about 2100 objects — all load through SoLo
  in CI on both architectures. Loading is the floor, not the claim: calls
  into the symbols the bridge still stubs abort loudly, and
  dev/abi-demand.txt is the remaining work, ranked by
  how many installations demand each symbol;
• a load-once runtime (dlclose succeeds but does not unload an image);
• supporting all four TLS models. Initial-exec variables are placed in a
  16 KiB surplus arena that rides in the executable’s own static TLS, so one
  process-wide offset is valid in every thread without patching musl. The one
  restriction: threads created before a dlopen see zero-initialized TLS
  for the modules it loaded, so load initial-exec libraries before spawning
  the threads that use them. An initial-exec module that does not fit the
  arena fails to load with an error naming the image and the byte counts;
• explicit about missing ABI coverage: an unimplemented glibc call aborts and
  names itself.

The goal is to turn the hard wall between “fully static” and “uses the system
GPU” into a finite, testable compatibility layer. The Vulkan PNG is the first
proof that the wall has a door.