GitHub – gavamedia/deltafin: Run Kimi K3, a 2.8T-parameter Mixture-of-Experts LLM, on a single Apple Silicon Mac. Streams MXFP4 experts on demand over HTTP into a local disk cache — fused NEON kernels, Metal/MPS compute, exact reproducible decoding, and an OpenAI-compatible API server for local chat and coding agents. · GitHub

Three commands, then you’re generating. The only real decision is step 3.

# 1. environment (Python 3.12+, and Xcode CLT for clang)
python3 -m venv venv
./venv/bin/pip install torch numpy safetensors tiktoken ml_dtypes blobfile \
    "transformers==4.56.2" einops tokenizers

# 2. build the fused MXFP4 kernel
clang -O3 -mcpu=native -shared -DNO_MAIN -o tools/libmxfp4gemv.dylib tools/fused_gemv.c

# 3. download the model  (see the two modes below)
./venv/bin/python tools/setup_k3.py --full

Every token reads 16 experts × 92 layers = 25.8 GB of expert data. From local
disk that’s about 4 seconds; over the network it’s minutes. That single fact is
the whole difference between the two columns.

Run setup_k3.py with no flag and it picks --full when the disk allows,
otherwise falls back to streaming and tells you exactly how much space you’d
need to free.

Streaming is a fine way to try Deltafin without committing 1.7 TB. Whenever you
want the speed, one command finishes the job — no reinstall, no reconfiguration,
and it picks up whatever is already cached:

./venv/bin/python tools/fetch_experts_all.py          # resumable, run anytime
./venv/bin/python tools/fetch_experts_all.py --dry-run   # just show the numbers
./venv/bin/python tools/fetch_experts_all.py --layers 1-40   # partial is fine too

Deltafin prints a reminder at startup — both for the CLI and the API server —
whenever it’s still in streaming mode, showing how much of the pool is local and
what finishing would cost.

Halves per-token I/O for the non-expert weights, with no meaningful quality
change in our checks. Takes a few minutes:

./venv/bin/python tools/convert_spine_int8.py

# ask a question; generates until the model finishes its answer
./venv/bin/python tools/kimi_run.py --chat --prompt "What are the three largest moons of Saturn?"

# raw completion (no chat template); runs until you press Ctrl-C, or cap it
./venv/bin/python tools/kimi_run.py --prompt "The capital of France is" --max-new 16

Tokens print as they are generated, so you always see the text as it comes.
Ctrl-C stops cleanly at any point and prints the result so far; --max-new N
caps the length. One honest warning: K3 thinks before it answers, and at about
a token per minute a full chat answer can take a while — watching it stream is
part of the experience.

Router selections are logged to router_trace.jsonl if you want to study K3’s
routing behaviour.

Deltafin can serve the standard OpenAI API, so chat interfaces, the openai SDK
and coding agents can use it by changing a base URL:

./venv/bin/python tools/serve_openai.py --port 8000

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '🔥'

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")
r = client.chat.completions.create(
    model="deltafin-kimi-k3",
    messages=[{"role": "user", "content": "Hello!"}])

print(r.choices[0].message.content)            # the answer
print(r.choices[0].message.reasoning_content)  # K3's thinking, when present

/v1/chat/completions, /v1/completions and /v1/models are implemented, and
streaming ("stream": true) works. Most tools that read OPENAI_BASE_URL and
OPENAI_API_KEY will work by pointing those at the server.

Please read these caveats before pointing anything automated at it:

• Time. Answers arrive when they arrive — set your client’s timeouts to
  hours, not seconds. Omitting max_tokens lets the model finish its answer
  (recommended); raw completions, which never end on their own, default to 256.
  Operators can set a hard ceiling with K3_SERVER_MAX_TOKENS.
• Streaming installs are much slower here. A chat-template prompt is 60
  tokens or more and prefill touches many experts per layer, so on a partly
  filled cache a chat request can spend hours fetching. With a full install it’s
  just normal (slow) inference. The server prints a warning at startup when
  you’re in streaming mode.
• Greedy only. temperature and top_p are accepted and ignored, and one
  request runs at a time (a second concurrent request gets a 429).
• Agents are a curiosity, not a workflow. Coding assistants work in
  principle, but their long system prompts make prefill expensive.

Everything works with no configuration: Deltafin picks the GPU when there is
one and the int8 spine when it has been built, and says what it chose at
startup. These variables exist for overriding that:

• An Apple Silicon Mac. All published numbers are from an M1 Max with 64 GB —
  the slowest machine it has run on. More RAM is used automatically (a 128 GB
  machine pins several times more of the model), and newer chips bring higher
  memory bandwidth, more GPU cores and faster storage. See
  Why newer Macs should be faster.
• Xcode Command Line Tools, for clang (xcode-select --install).
• Python 3.12 or newer.
• Disk: ~1.7 TB for the full install, ~215 GB for streaming (see Install).
• Network access to Hugging Face.

K3’s weights total about 1.56 TB, which is more than this machine’s free disk, let
alone its RAM. The observation that makes local inference possible anyway is that a
Mixture-of-Experts model only touches a small fraction of itself per token.

• The resident spine (~114 GB: attention, shared experts, latent projections,
  embeddings) is downloaded once and read layer-by-layer from local NVMe each token,
  quantized to int8 and computed on the GPU.
• The 82,432 routed experts (~1.45 TB). For each token K3’s router picks 16
  experts per layer, and only those are read. Install them all locally if you can
  (recommended); otherwise Deltafin fetches them from Hugging Face on demand —
  one HTTP range request per expert — into a growing disk cache.
• The forward pass runs Moonshot’s own modeling code, unmodified. A small
  pure-PyTorch shim stands in for the CUDA-only fla kernels it expects.

flowchart LR
    subgraph HF["Hugging Face CDN"]
        W[("96 safetensors shards
1.56 TB · MXFP4")]
    end
    subgraph MAC["MacBook (M1 Max, 64 GB)"]
        subgraph DISK["NVMe"]
            SP[("resident spine
114 GB bf16 → 60 GB int8")]
            EC[("expert cache
raw shard spans")]
        end
        subgraph TOK["per token"]
            R{"router
top-16 of 896
× 92 layers"}
            L["93 decoder layers
2 shared GPU templates"]
            K["fused MXFP4 GEMV
NEON"]
        end
    end
    W -- "one range request
per missing expert" --> EC
    SP -- "double-buffered
layer loader" --> L
    EC -- "mmap" --> K
    R -- "selected experts" --> K
    K --> L
    L -- "logits" --> R

All numbers below were measured on one M1 Max (10-core CPU, 32-core GPU,
64 GB, internal NVMe) with the full model installed locally, greedy decoding,
on a quiet machine. This is the slowest machine Deltafin has run on — treat
these as a floor.

Roughly 3.75 tokens per minute. Where those 16 seconds go, per token:

Decode is now bound by disk bandwidth on the resident spine. Those 53 GB are
re-read every token, and at the ~7 GB/s this access pattern sustains that is about
7.5 s of the 15 — unavoidable without either more RAM (enough to hold the spine
without displacing the page cache the expert reads need) or a smaller spine.

Every one of those lines is bound by something the M1 Max is simply the weakest
at. Nothing here is tuned for a specific chip, so a newer machine should pick up
the difference without any configuration:

• Memory bandwidth. The M1 Max has 400 GB/s. An M3/M4 Max is meaningfully
  higher and an Ultra roughly doubles it — that lands directly on the spine load
  and the expert matmuls.
• GPU. More cores execute the same Metal kernels faster; the dequant shader
  and the attention path both scale with it.
• SSD. Expert reads are the single biggest slice, and they run at whatever
  the internal drive delivers. Later Macs ship faster NVMe.
• RAM. This matters most. The 53 GB spine does not fit alongside everything
  else on a 64 GB machine, so it is re-read from disk every token — about half
  the total time. On a 128 GB machine it can simply stay in the page cache, and
  that cost largely disappears. Deltafin also pins more of the model there
  automatically, with no flags.

We have only ever run this on the one machine. If you try it on an M3, M4 or
M5, or with 128 GB, we would genuinely like to see your numbers — open an
issue with the output of K3_PROFILE=1 and your chip.

When n-gram speculation accepts a draft, one forward pass emits two tokens, so
repetitive text runs proportionally faster. Speculation is lossless: accepted
drafts reproduce the reference sequence exactly, and a rejected draft restores
the model state bit-for-bit.

The gap between the two decode rows is the whole argument for the full install
(see above): with the experts local, every prompt runs at the
top-row speed instead of only the ones whose experts happen to be cached.

Output is greedy and reproducible: the same prompt yields the same tokens, run
after run.

The capital of France is → Paris. The Eiffel Tower is located in Paris. The Louvre Museum is also in Paris. The Louvre has…

To be clear about the limitations: this is a research artifact, not a practical
chat setup. Sixteen seconds a token is a long way from interactive, and long
prompts are expensive because prefill touches many experts. We think it is
interesting mainly as an existence proof, and as a testbed for
streaming-inference techniques.

Each of these was measured on real weights before we trusted it. The full
experiment log — including the ideas that didn’t survive measurement — is in
BRAINSTORM-SPEED.md, and the original plan in
PLAN.md. Little of this is novel on its own; most of it adapts ideas
from the projects credited below to K3’s particular shape.

• Coalesced expert fetch. Each expert’s six tensors happen to be contiguous in
  the shard files (we checked all 82,432), so a whole expert is a single 17.55 MB
  range request over a small pool of keep-alive connections. That measured about
  6.4× faster than fetching tensors individually. We also tried HTTP/2; it was
  slower than HTTP/1.1 keep-alive in this setting.
• Raw-span disk cache. Cache files are the shard bytes verbatim — no container
  format, no parsing.
• Parallel expert reads. A layer’s 16 selected experts are read together by a
  thread pool using pread with F_NOCACHE, rather than being demand-faulted
  page by page while the kernel computes. Measured cold: 0.87 GB/s faulting
  versus 6.85 GB/s reading. This was worth 40 s → 4.3 s per token on the read
  path alone, and F_NOCACHE keeps 25 GB/token of expert traffic from evicting
  the page cache the spine needs.
• Double-buffered layer loading. A worker thread reads the next layer’s spine
  data while the current layer computes.
• Previous-token prefetch. Consecutive tokens reuse about 31% of their expert
  selections, so each token’s set is fetched in the background for the next one.
  (An earlier 39.7% figure counted repeated prompts in our own trace; on a
  deduplicated holdout it is 30.8%, below the 41.3% colibri reported for GLM.)

• Fused MXFP4 dequant+GEMV (tools/fused_gemv.c) — a
  NEON kernel that dequantizes and multiplies in one pass using a 16-entry table
  lookup, with the e8m0 scale applied as integer arithmetic on the fp32 exponent.
  It matches the reference implementation bit-for-bit and replaced a much slower
  dequantize-then-matmul path. A Metal version exists as a validated prototype.
• Template-layer buffer reuse. All 69 KDA layers share one set of tensor
  shapes and all 24 MLA layers another, so two persistent GPU-resident “template”
  layers can receive each layer’s weights via copy_(). This avoids the allocator
  churn that profiling showed was a large share of per-token time.
• int8 resident spine. Halves the per-token resident I/O. In our checks the
  top-5 next-token candidates kept their order and the top logit moved by 0.07%.
• Custom Metal dequant kernel. Loading the spine spent most of its time in a
  row-broadcast multiply that MPS runs at 43 GB/s, against 334 GB/s for a plain
  copy of the same bytes. A small compile_shader kernel fusing int8→fp32, the
  row scale, and the copy reaches 297 GB/s; with persistent staging buffers and
  transfers hoisted out from between dispatches, per-layer load went 118 ms →
  21 ms. Bit-exact: max|diff| = 0 on every tensor.
• Pure-PyTorch KDA shim (tools/fla/) — Kimi Delta Attention’s
  recurrence, short convolution, and gated norm, ported from fla-core’s semantics.
  Chunked and step-by-step execution agree to about 1e-9. At decode the recurrence
  runs on CPU, where its small state fits better than a series of GPU dispatches.

• N-gram speculation. Drafts come free from suffix matching against the text
  so far, and are verified in a two-position batch whose fixed costs are shared.
  This is worthwhile here precisely because resident I/O and compute — not expert
  fetching — dominate a warm token. Accepted drafts reproduced the reference
  sequence exactly in our tests, and rejection restores the model state
  bit-for-bit (K3’s recurrent states are small enough to snapshot cheaply).

sequenceDiagram
    participant D as n-gram draft
    participant M as model (one T=2 pass)
    participant S as state snapshot
    D->>M: [last_token, draft]
    M->>M: 93 layers, shared cost
    alt draft verified
        M-->>D: 2 tokens accepted
    else draft wrong
        S-->>M: state restored (bit-exact)
        M-->>D: 1 token, nothing lost
    end

• Two modes. exact (default) is logit-faithful and reproducible; we use it
  for all correctness work. approx (K3_APPROX=1) uses fp16 weights, so output
  stays coherent but near-tie tokens can differ between runs. We haven’t measured
  approx to be reliably faster yet, which is why it isn’t called “fast”.

• At startup Deltafin reserves memory for the OS (max(10 GB, 18%)) and pins as
  many resident layers as the remainder allows. A 128 GB machine pins several
  times more than a 64 GB one without any configuration, and the expert cache
  additionally benefits from whatever page cache is free.

We kept the failures, since they may save someone else the time:

Roughly in order: the Metal expert kernels (already prototyped), a proper quality
harness — average NLL against the official API — so lossy speed/quality
trade-offs can be measured rather than argued about, smarter expert prefetching,
and eventually a native engine in the spirit of ds4, where most of the remaining
overhead should disappear. Details in PLAN.md.

Deltafin leans heavily on work that others published openly. In rough order of
influence:

• colibri (JustVugg, Apache-2.0) —
  showed that a 744B MoE can run in 25 GB of RAM, and is where we learned about
  router-lookahead prefetch, learned expert pinning, F_NOCACHE and F_RDADVISE
  discipline on macOS, and the shard-by-shard conversion pattern. Its M5 Max
  performance report — CPU spin-waits starving the GPU of a shared power budget —
  changed how we schedule work.
• ds4 / DwarfStar (Salvatore Sanfilippo,
  MIT) — the clearest expert-streaming design we studied: zero-copy expert
  buffers, masked dispatch, selection-based cache eviction, session persistence,
  and a quality methodology (average NLL against official API outputs) that we
  adopted outright. Its stated philosophy — correctness before speed, hide I/O
  behind compute — is the sensible one, and we tried to follow it.
• Moonshot AI — for releasing
  K3’s weights openly with readable modeling code, which Deltafin runs directly,
  and for the Kimi Delta Attention design, whose small recurrent state is what
  makes long context feasible on a laptop at all.
• flash-linear-attention
  (fla-org, MIT) — our KDA shim is a port of semantics from its kernels and
  reference implementations.
• llama.cpp / ggml — prior art for
  in-kernel dequantization and MXFP4 handling, and the foundation of most of what
  the local-inference community knows.
• PyTorch,
  Transformers,
  ml_dtypes (our bit-exactness
  reference for e2m1) and tiktoken.

Deltafin’s own code is MIT. Two things in this repository are not ours:

• tools/fla/ is a pure-PyTorch port of semantics from
  flash-linear-attention (MIT, © 2023–2026 Songlin Yang, Yu Zhang, Zhiyuan Li).
  The attribution is repeated in the file header and in LICENSE.
• Kimi K3’s weights and modeling code belong to Moonshot AI and are distributed
  under Moonshot’s own license. They are downloaded at setup, never vendored
  here — please read that license before using them.

Deltafin is an independent project with no affiliation to Moonshot AI.