every cliff moved 100x · Traceway

TL;DR

I spent six days working on and running a benchmark of observability data on DuckDB. I’ve done this with SQLite in the last blog post and I really wanted to see how DuckDB compares on a cheap CCX13 Hetzner instance. The way I’ve done the measurements is by implementing DuckDB as a valid storage engine for Traceway and then running its benchmarking suite. The benchmarks show that DuckDB’s columnar engine is able to query 100x more data points without any major backend changes. The write throughput is also 3x to 15x higher. The result is that you can now self-host the full OTel stack that can handle a large data volume on a pretty small server. Keep reading to find out how I’ve done the measurements!

This is what came back:

A read cliff, throughout this series, is the largest table size at which real dashboard pages still load: the median of three endpoint probes at or under 5 seconds, none timing out. It earns the name because of what sits one 10x step past it: queries don’t slow down, they stop coming back.

Each signal’s read cliff sits exactly 100x further out on DuckDB than on SQLite, at equal or better latency, on identical hardware. I checked that symmetry against the raw JSON twice before I believed it. The box also ingested a billion metric points in under an hour into 10.8 GB of disk and stayed up, a scale the SQLite benchmark never got within 100x of. Logs, the signal post 2 said to keep off SQLite entirely, are DuckDB’s biggest win.

What’s actually being compared

Post 2 ended promising a ClickHouse comparison, and that post is still coming. But every time I sat down to build it, the same question got in front of me: before reaching for a client-server OLAP database, with its own container, its own memory appetite, its own failure modes, how far does an embedded one go? Traceway ships a DuckDB telemetry backend as an opt-in build, same single binary, same deployment story as the SQLite build, different storage engine under the telemetry tables. If you self-host on one cheap box, these two builds are the actual decision in front of you, and I couldn’t find anyone who had published numbers for it. So I ran post 2’s entire methodology against the DuckDB build and put the results side by side.

Here’s the plan. Methodology first, including what changed in the harness since post 2 and one fix in the backend itself that these numbers depend on. Then writes, all three signals against their SQLite baselines. Then reads, same shape. Then the billion-row run, what queries cost past the cliffs, the fleet math for both builds, and which build I would actually run.

How I measured

The setup is post 2’s, so I’ll keep it short: a system under test (SUT) running the Traceway binary and a separate load generator on a private link, both in Hetzner’s Nuremberg datacenter, OTLP, the OpenTelemetry wire protocol, over gzipped protobuf, uniform-random data in bounded ranges. Traceway built from commit 14b4aa6e, Ubuntu 24.04, retention off during the bench.

Two scenarios per signal. The throughput ramp: batch-size ramp at fixed rate, then a request-rate ramp at the winning batch (this time with a finer rate ladder, 1 through 25 req/sec, because the interesting cliffs turned out to sit between post 2’s coarser steps), then a small-batch high-rate ramp shaped like an SDK fleet. A step passes at under 5% errors and at least 70% of target rate achieved. The read-probe: fill the table to 1M, 10M, 100M, 1B, then 5B rows, and at each level load three real dashboard endpoints per signal, with a level passing when the median is at or under 5 s and nothing hits the 6 s timeout. The 5-second bar is generous on purpose and post 2’s discussion of that still applies unchanged.

Three things differ from post 2’s harness, all disclosed:

1. The loadgen grew. Post 2 generated load from a second CCX13, which SQLite’s ceilings never stressed. On my first DuckDB spans run, that loadgen box died mid-ramp while the database sat at 0% errors, the traffic generator broke before the thing it was breaking. The loadgen is now a CCX23. The SUT, the box every number describes, is unchanged.
2. The digestion gate. DuckDB checkpoints its write-ahead log after ingest stops; probe during that and you measure a busy engine, not a query. The read-probe now polls the backend’s deep-health endpoint until the database and WAL file sizes are stable, records the wait per level (5 seconds at every level in the final runs), then probes. SQLite reports no engine gauges and skips the wait, so its post 2 numbers were produced identically.
3. Only accepted rows count. The fill counts items confirmed by 2xx responses, not items attempted, which matters once the backend learns to refuse work. Fills overshoot their targets, at the smallest level by as much as 2.6x, so I quote actual row counts wherever the difference matters.

Caveats. These are single-shot runs, one throughput run and one read ladder per signal. Post 2 promised medians of three reps starting with this post, and I’m walking that back one more time: the fix-and-rerun cycle below consumed the budget, and I’d rather publish single-shot numbers with that admission than sit on them. The key cells reproduced across the debugging runs (100M metrics reads landed between 2.7 s and 3.2 s across four runs on different days), but the protocol is single-shot. “Passing” still means three named endpoints on freshly written data on an otherwise idle box; read-under-ingest remains future work.

One more disclosure, this one in the backend rather than the harness. My first DuckDB runs produced numbers I could not trust: the box kept dying mid-benchmark, and the hunt led to a real bug in my own ingest path, which had no admission control and let a sustained burst of fat batches run the process out of memory. I fixed it with an ingest gate that caps concurrent processing and answers overload with a 503 plus Retry-After (the commit linked above), and reran everything. All numbers in this post are from the fixed backend; the pre-fix results still visible in the repo’s history predate it and understate DuckDB, spans by 23% and logs by almost half, because the crashes were cutting the ramps short. SQLite never surfaced the bug, its slower insert path errors early under pressure, which is post 2’s 5 req/sec logs wall doing its job.

Writes: 4x, 3x, 15x

Going in, my quiet bar for this table was “double SQLite and I’ll call it a win,” and the one cell I braced for was logs. Post 2’s starkest result was logs writing at 4,877/sec into a wall at 5 req/sec so sharp that 1.5x more requests took the error rate from 0% to 98%. That wall is the cell that moved most: the DuckDB build writes logs at fifteen times SQLite’s rate, and the wall became a slope of polite refusals. The signal with the lowest SQLite ceiling and the highest real-world volume is the one the columnar engine helps most, which is either obvious in hindsight or backwards from everything post 2 trained me to expect, and I held the second view until this table existed.

The “lowest failing step” column matters as much as the headline column, and lowest is the right word: the ladder bisects between the last passing and first failing rate, so those are the smallest rates that fail, not the first failures in wall-clock order. Every failure in the column is the backend refusing work rather than falling over: the run logs show 503s from the admission gate, the archived JSON shows zero insert failures and zero dropped rows behind them, and the container finished all three runs with zero restarts. That is the same graceful-cliff behavior SQLite showed in post 2, and DuckDB only exhibits it because of the fix I disclosed above; before it, this table could not be produced at all.

The bottom of each ramp holds one more first for the series: the SDK-fleet shape, hundreds of small batches per second rather than a few fat ones, which post 2 could only measure on SQLite. DuckDB sustains 53,091 metric points/sec, 36,577 spans/sec, and 30,442 log records/sec at batch 100. My early attempts never produced these numbers, the pre-fix backend died before that phase of the ramp could run, so seeing them appear at all was the first sign the reruns were measuring something the crashes had been eating.

Reads: every cliff, 100x later

Writes were the appetizer; reads are why anyone reaches for a columnar engine, and they’re where I had the most to lose. Post 2’s most-quoted line was about read cliffs, the last row count where the dashboard still loads, and if DuckDB only matched SQLite here, the whole build would be a curiosity. Before running it I told myself I’d be satisfied if each signal’s cliff moved 10x.

Each cell is the median of three real endpoint probes after fill and digestion; bold marks each signal’s last passing level. Two footnotes on the table itself. The levels are fill targets and the fills run over: the 1M level actually holds 1.7M rows for spans and logs and 2.6M for metrics, and every level above lands within 16% of target, so the exact row counts are in the JSON. And the over-60-s durations come from a second ladder run with the probe timeout raised to 60 seconds, which gets its own section below; the standard probe gives up at 6 s. I got 100x per signal, not 10x: SQLite’s last passing levels were 1M, 100k, and 100k, and all three moved by the same factor of one hundred. Same endpoints, same thresholds.

Metrics reads scale gently: 146 ms at 1M, 382 ms at 10M, 3.0 s at 100M. A hundred times the rows costs twenty times the latency, the scan gets cheaper per row as the table grows. The level SQLite barely passed is a hundredth of the level DuckDB passes with the same margin.

Spans pay for their percentiles, same as they did on SQLite, just later. This is the page doing the paying, shown on my local instance with the loadgen’s spans:

Every row in that table is a P50/P95/P99 aggregation over the spans behind it, and the stacked latency chart above it is a second aggregation. Those two queries cost 902 ms at 10M rows and blow past a full minute at 100M. Post 2 watched the same two pages die between 100k and 1M.

Logs are the numbers I trusted least and checked hardest, an early bad run had me convinced for a full day that DuckDB couldn’t read logs at all, and the truth turned out to be post 2’s most interesting pattern repeated at ten times the rows. Here’s the page in question, on my local instance with the loadgen’s log records:

At their passing levels logs are the fastest dashboard of the three (85 ms median at 10M), and their failure at 100M is uneven in the same way SQLite’s was at 1M: the trace-ID lookup still answers in 2 ms, the body search in 3.5 s, and only the severity filter, which matches and ranks a large slice of 100M rows, dies. The pages an on-call actually opens mid-incident, find this trace, search this error, remain usable at 100M rows; the full-table scan does not. Retention policy should still follow that split. The split just moved 100x.

A billion rows on an 80 GB disk

I put the 1B level on the ladder expecting to write a paragraph about how it died, because every earlier attempt at it had ended the same way: container gone, level unreadable. This time the final fill step alone ran for 52 minutes, adding its last 900M points at a sustained 287k points/sec with the gate shedding the excess, and brought the table to 1,001,472,000 rows in 10.8 GB on disk, about 10.8 bytes per point after columnar compression, under an hour of total ingest across the ladder. The box stayed up. Health checks answered the whole way through, and post-fill digestion took 5 seconds. Nothing in posts 1 or 2 gets within 100x of this level; on the fixed backend a billion rows is just a big table.

A big table you cannot look at: all three dashboard queries ran past 60 seconds at 1B, so the level fails, and 5B was never attempted since the ladder stops at the first failure. I left it that way on purpose. A 5B fill would prove the disk holds 54 GB of points, and I already know nothing could read them. The practical summary of the ladder’s top: this box can store a billion metric points, and it can show you a hundred million of them.

How slow is slow past the cliff

Post 2 could only say “it timed out” about anything past a cliff. This time I reran the entire ladder with a 60-second probe threshold to put real durations on the failing cells, and there were no durations to put: every failing query was still running when the new cap cut it off. Metrics at 1B, still running at 61 s. Spans aggregations at 100M, still running at 61 s. The logs severity filter at 100M, still running at 61 s while the body search finished in 3.2 s beside it.

So the cliffs are not slopes on either database. Queries go from a second or three to more than a minute across a single 10x step in rows, the signature of an aggregation outgrowing the 4 GB memory budget and spilling. Post 2 found the same absence on SQLite at one hundredth the scale, and it is why I keep publishing cliffs instead of curves: there is no slow-but-usable band above a cliff to plan around.

The 60-second run also caught a smaller flaw for the fix list: a cancelled dashboard query does not promptly release its read-pool connection, so after the two spans aggregations timed out, a 3 ms exceptions query queued behind them for 52 seconds. It affected no passing number, and it goes on the same list the admission gate came from.

What fits comfortably, on each build

Cliffs are ceilings; what matters day to day is how far under them a real fleet sits. The small-fleet shape from post 2, ten backends emitting 50 spans/sec, 200 log records/sec, and 10 metric points/sec each, held against both builds:

Writes were already a non-issue on SQLite for everything except logs, and on DuckDB they stop being a conversation entirely. Retention is still the real knob, with new units. At this fleet’s rates, logs reach their last passing read level (10M rows) in about 85 minutes and the failing level in about 14 hours; spans reach 10M in 5.5 hours; metrics reach 100M in 11 days. On SQLite, post 2 measured the logs window at 50 seconds. An incident-sized working set, hours of spans and logs, days of metrics, fits on the right side of every DuckDB cliff with an ordinary retention job, which is the thing the SQLite build could not offer.

What I deliberately didn’t measure

• Read under concurrent write. Fill, digest, settle, probe. Stricter isolation than post 2, and still not what a dashboard experiences at 3 PM. Top of the future-work list.
• Mixed-signal load. Each signal had the box to itself; a real deployment writes all three at once and these ceilings do not simply add.
• Result correctness. I timed responses, I did not diff their contents.
• Variance. Single-shot, as admitted in the methodology, with the median-of-three protocol still owed.
• Tuning. Stock config, meaning what the benchmark compose file ships by default: 4 GB DuckDB memory cap, 256 MB checkpoint threshold, no schema or query changes. The 1B read failure smells like it wants a memory-budget experiment, and that is a deliberate cliffhanger.
• ClickHouse. The comparison post 2 promised is still owed, and it gets its own post on these same ladders rather than a section squeezed in here.
• Durability over weeks. An hour-long billion-row fill says nothing about month three on the same disk.

What surprised me

I expected the reads to win, that’s what columnar storage is for, and I set my bar at 10x per signal. All three delivered 100x. But the thing that will stay with me from this week is that the database was never the hard part: every wrong-looking number in six days of benchmarking traced back to my own ingest path, not the engine. DuckDB, whenever my code got out of its way, was boring in the best way a database can be boring.

The other surprise was logs, again. Post 2’s verdict was “keep logs off SQLite,” and logs turned out to be the strongest single argument for the DuckDB build: 15x on writes, 100x on the read cliff, with the incident-workflow pages still answering at 100M rows.

The verdict

Post 1 claimed a $16 box could run your observability stack. Post 2 tested the claim and returned it with an exception: not logs, and mind the read cliffs. This post retires the exception. On the same box, the DuckDB build writes every signal faster than a small fleet emits, serves dashboards at row counts the SQLite build cannot approach, and turns logs from the signal post 2 told you to keep off the box into the best result in the dataset. A billion metric points cost 10.8 GB of disk. I came into the week hoping to double post 2’s numbers; the smallest improvement in the table is 3x.

So if you self-host Traceway on a single machine, run the DuckDB build (docker-compose.duckdb.yml in the repo brings it up). One config note before you do: that compose file ships a conservative 2 GB DuckDB memory cap and leaves the checkpoint threshold at DuckDB’s 16 MB default, while the benchmark ran on 4 GB and 256 MB, so set DUCKDB_MEMORY_LIMIT and DUCKDB_CHECKPOINT_THRESHOLD to match if you want these numbers. I would only reach for the SQLite build in two situations, and both are real. If you need a binary that compiles anywhere Go does, SQLite wins: the DuckDB build needs CGO, Go’s C bridge, and glibc, so its container image is Debian rather than Alpine. And if your volume lives comfortably under post 2’s numbers, SQLite’s storage engine doing all its work inline, no deferred checkpoints, no digestion window to wait out, remains the simplest thing that works. The moment logs matter, or retention past an hour matters, the comparison stops being close.

One asterisk belongs on the whole table: these numbers exist because the benchmark first found a crash bug in my ingest path, and I fixed it before rerunning everything. If you run the DuckDB build, run a version with the admission gate. The engine was never the problem. The code in front of it was, and finding that out is the most useful thing this comparison produced.

Raw data + workflow

Everything in this post, runnable from the repo:

Questions, pushback, or “your number is wrong because X”: jstojiljkovic941@gmail.com, or find me on GitHub.

Next post: the one post 2 promised. ClickHouse on the same box, same loadgen, same ladders, to find out what the full client-server stack buys you over the embedded engines and what it costs to get it.