Kubernetes on Oxide: How Customer Needs Shaped Our Integrations

With clusters provisioned, reconciled with Oxide, and reachable from outside
their VPCs, storage for stateful workloads became the next layer to address.
Kubernetes users request persistent storage through PersistentVolumeClaim
objects and expect a
Container Storage
Interface (CSI) driver to create, attach, and mount the underlying volumes.
Oxide had disks, but Kubernetes had no native way to manage their lifecycle.

Without an Oxide CSI driver, customers could deploy a third-party Kubernetes
storage system such as Longhorn. Longhorn provides
its own CSI driver and replicates data across disks attached to Kubernetes
workers. However, using Longhorn meant backing its replicas with Oxide
distributed disks, which already store three replicas on distinct sleds.

Layering one replicated storage system on another can create substantial write
fan-out. When a three-replica Longhorn volume is backed by three-way-replicated
Oxide distributed disks, one application write can fan out to as many as nine
disk writes. The exact physical write amplification depends on the workload and
configuration, but customers wanted to avoid that duplicated replication.

The introduction of
Oxide local disks
provided a way to remove the second layer of replication. Local disks have no
built-in replication and remain tied to their sled, making them well suited
to systems such as Longhorn that replicate data across Kubernetes nodes. Our
Rancher showcase
uses this approach today. It avoids stacking two replicated storage systems,
though Longhorn still manages the storage lifecycle rather than a native Oxide
integration.

For a native integration, my teammate Luiz wrote
RFD 595 Oxide CSI Plugin.
The workflow seemed straightforward on paper. When a user creates a
PersistentVolumeClaim, the CSI controller creates an Oxide distributed disk.
After Kubernetes schedules the pod, the controller attaches that disk to the
selected Oxide instance, and the CSI node plugin formats and mounts it for the
pod. If the pod is rescheduled onto another node, the controller detaches the
disk and reattaches it to the new node.

Prototyping that workflow immediately exposed a blocker. Oxide requires an
instance to be stopped before attaching or detaching a disk. Kubernetes,
however, expects a CSI driver to attach storage to a running worker after
scheduling a pod. Stopping the worker would disrupt every other workload on the
node and could trigger cascading scheduling and attachment operations.

Before we can release our CSI plugin, we need to add support for disk hot-plug
throughout the Oxide stack, from the hypervisor all the way up to the API. What
began as a Kubernetes integration has turned into a project spanning multiple
layers of the Oxide software stack.

Disk hot-plug and the Oxide CSI plugin remain under active development as of
this writing. In the meantime, customers can use software such as Longhorn with
Oxide local disks for dynamically provisioned persistent storage without
stacking two layers of replication. When the native CSI plugin ships, customers
will be able to use familiar Kubernetes storage APIs backed directly by Oxide
distributed disks with replication and durability built in.