The benchmarkpocalypse

There’s been a lot of talk about the vulnpocalypse, to which I don’t have much to add because I’m not a security person, but I haven’t seen much discussion on the closely related (and to be fair, less serious, issue), the benchmarkpocalypse.

While it’s become easier than ever to make serious performance gains, it’s also become easier than ever to reward hack a benchmark and make fake performance gains. The former is probably happening quietly across many different companies, but the latter is something I see at least once a week nowadays. Someone will claim they optimized X and got some huge performance improvement over existing software, but, when you look at it, what they did was make some optimization that improves benchmark performance without actually improving real-world performance. This is often some kind of “we rewrote X in Rust” project or a new startup that’s looking to either fundraise or sell something, but it happens on other kinds of projects as well.

Of course, people have always trumpeted unrepresentative microbenchmarks to show that their pet project is great. It’s always been easy to fake up an unrepresentative microbenchmark and that’s never going to change. What’s changed is that it used to take a lot of work to game a large benchmark suite, but an LLM and loop can just do it. There are quite a few famous examples of gaming large benchmark suites from back when this was hard. For example, way back when people cared about SPECint / SPECfp as proxies for workstation performance, CPU vendors would try to find compiler “optimizations” that would speed up the calculation in the benchmark, such as Sun finding a way to improve 179.art by 12x in SPECfp2000. Skilled engineers spent a lot of time trying to find benchmark hacks like that. LLMs make this trivial, making formerly trustworthy benchmarks meaningless unless you audit the result or trust someone who did.

Rather than point to someone’s bad claim, I’ll point to FRE, this regex engine I had an agent build, which I could claim is the world’s fastest regex engine because it beats the Rust regex crate at the fairly comprehensive rebar regex benchmark suite. But this was created by putting an agent in a loop for a month with instructions to not overfit to the benchmark but no real supervision. For the most part, getting an LLM to give you a good benchmark score is fairly easy, and this case was no different; it took a couple weeks to roughly match Rust regex crate performance and then another couple weeks to get to 1.4x faster on rebar. But agents are wont to reward hack and overfit unless you put serious guardrails in place to avoid that, which I didn’t do in this case as an experiment.

To check for overfitting, I somewhat arbitrarily used the ripgrep benchmark corpus as a holdout benchmark it was 10x slower on cases where the benchmark didn’t take forever due to an algorithmic blow-up, and there were cases where it took so long that it wasn’t reasonable to even wait for the benchmark to complete. So much for being 40% faster!

Andrew Gallant (aka BurntSushi)’s rebar benchmark suite is fairly comprehensive as benchmaark suites go, but even with a fairly comprehensive benchmark suite, agents have no problem getting a high score while overfitting in a way that doesn’t necessarily give good general performance.

The next step was using a trick we talked about before of not just telling the LLM not to cheat, but that there’s a holdout benchmark set that it’s judged against. After that, the LLM moderately generalized performance to the point where it’s about 2.4x slower overall on the holdout. That sounds pretty good considering that we’re comparing it to the fastest general purpose regex engine in existence. But, recall that these benchmarks were made by a coding agent. On looking at what the benchmarks measure, some of them really don’t make sense to include, at least at equal weight. If we only look at the benchmarks that seem like they matter, FRE is 4x slower on the holdout⁰, which is a lot better than before applying the good ole’ “tell them you have a holdout” trick, but still pretty far from being 40% faster.

There are a few things I thought were interesting about this:

1. It’s trivial to “win” a non-trivial benchmark in a meaningless way even when you instruct agents to not reward hack or overfit to win the benchmark
2. Once again, telling the LLM there’s a holdout set worked better than just telling the LLM to do generalized work or not overfit or cheat
3. Although the overall performance of FRE isn’t that good, it is actually performs better for some use cases; in general, the cost of writing specialized code that used to require people serious engineering experience for some specific use case has gone way down

On (1), no wonder I’m seeing so many bogus claims. In the past, to build something like FRE that fakes performance well enough to be able to bogusly claim a 40% speedup, you would need a fair amount of expertise. At a minimum, you’d need to have a pretty good understanding of string matching algorithms, regex engines, as well as decent general code optimization and SIMD optimization skills. FRE also has a mode where it compiles the regex to machine code, so you’d also need some compiler expertise. Now you can get that kind of benchmark cheating (whether or not you want the cheating) with a few minutes of typing.

On (2), I’m curious if this generalizes but haven’t tried enough examples to be able to tell.

On (3), there’s no reason to use a vibe coded regex library that was almost no human effort that’s slower than a robust, existing, well-tested, library, so I find the FRE artifact uninteresting. The thing I find interesting here is how much LLMs can substitute for what used to be rare, specialized, and expensive, knowledge.

In the past, even if you had the knowledge, you probably wouldn’t write a custom regex engine that’s optimized for your particular workload. There are some large-scale use cases where people would do that level of customization, e.g., when I worked on the Bing index, the code contained multiple different compilers because someone who worked on it wanted to eke out maximal performance; since you care about both compile time and compiled performance in a search engine and the trade-offs are different in different places, you get better performance by writing a custom compiler for each place where a normal project might just use an interpreter or directly walk some data structure with “normal code”. The person who wrote those compilers, working on regex-like code might also write multiple custom regex engines, but very few people have both the expertise and the inclination to do that, let alone the freedom to spend that kind of time on such specialized code for work. If you price out that Bing engineer (then a Partner-level engineer, promoted to Distinguished Engineer for their work on the search index) compared to the price of running an LLM in a loop, the cost of writing this kind of specialized code has gone down by many orders of magnitude.

Even though the overall FRE regex engine has worse performance than the Rust regex crate, the gains you can get for specializing to your workload or use case mean that, in some cases, it could be reasonable to insert your own specialized regex engine somewhere, and the same goes for various other kinds of low-level software. You don’t have to be an AI maximalist to think that it’s plausible that, within some number of years, we could see this kind of thing happening for larger things, like databases.

Thanks to Yossi Kreinin, Jamie Brandon, Peter Geoghegan, Luke Burton, John Spurling, Dennis Snell, and Max Bittker for comments/corrections/discussion.

P.S. Per the discussion here, with LLMs, the time it takes to poke at something for a bit and satisfy my curiosity has gone way down, while the time it takes to write something up and make it rigorous enough to publish on my blog hasn’t really changed (for a variety of reasons, I think it’s actually gone up). The result of this has been that I’m doing a lot more analyses than ever and sharing results with a few friends but not publishing them. As an experiment, I’m trying to write up some things very quickly, with a much lower standard for how cleaned up and rigorous things are than I’d normally have for something that appears on the blog; more like what I’d tell a friend in a casual conversation than what I’d normally put in a blog post. The goal for this post was to do the write-up in about half an hour, so it’s something I could do over lunch and not really take time on. If you have opinions on this, let me know what you think!

Of course, a caveat here is that all of the numbers have a higher risk of being wrong than usual. I looked at one benchmark for maybe a minute or two and found an issue, then I looked at another benchmark for a minute and found another issue. Both of those are fixed, but this implies there are other issues I haven’t taken the time to chase down. But, with respect to bad benchmark numbers, that’s highly realistic! Almost any time I look into benchmark numbers, such as here, or here, the numbers are wrong. Another aspect of the benchmarkpocalypse is that, at least for now, LLMs are good at doing bad benchmarking, so even if you have something that’s a real performance improvement, you generally can’t tell from some LLM-generated benchmark setup unless a significant amount of care has been taken to make sure that the benchmark setup is reasonable.

Appendix: more FRE benchmark details

One thing I found after I wrote the above but before publishing the post, was that the LLM’s claim that FRE is 40% faster than the Rust regex crate on rebar was also wrong. Or, if not wrong, at least misleading. It wasn’t actually running benchmarks in the same way rebar benchmarks were run. I checked this after spending a minute checking benchmark results found two issues. It turns out that, despite instructions to run rebar benchmarks as they’re run in https://github.com/BurntSushi/rebar, the LLM changed the interface to allow FRE to make some optimizations that improve performance. After fixing that, instead of FRE being 1.4x faster than Rust on rebar, it was 1.5x slower (and “only” twice as fast as re2), so the original result was doubly fake. Not only was FRE highly overfit to the rebar benchmarks, it the results also involved cheating.

But on the bright side, this means the difference in performance between FRE on rebar (1.5x slower than Rust) and on the holdout benchmarks (2.4x slower) isn’t as big as it looked before, so the “tell the LLM you have a holdout” trick worked even better than it seemed to before.

After that, I let an LLM hill climb for a few hours and it claimed that FRE was 1.28x faster, which sounds like a great improvement for only a few hours of LLM time, but then I decided to spend another minute looking for cheating and found multiple issues, including one case where a search for the count of matches of (?s)^(.*)$ returned the count without even looking at the haystack (data). Another case of cheating was doing a multi-line grep where the benchmark is supposed to be done line-by-line. Finding these isn’t surprising because this is the kind of thing that happens when you leave an agent in a loop for a month without defining strict guardrails. Whether this makes my point here stronger or undermines it isn’t clear, but after fixing another set of these issues, FRE was back to being 1.4x slower. After leaving an agent to run overnight, FRE was allegedly back to being 1.5x faster.

Since my original goal here was to see what happens when you run a current (public) SOTA agent in a loop (GPT-5.6 Sol) without much supervision on a non-trivial code optimization problem without any real supervision, rather than spend more time fixing things up to make the benchmarks fairer, I’ll just stop here and put a few plots of the results.

Overall, we can see that against Rust and RE2, FRE tends to outperform on the rebar benchmarks (and as noted above, much of this is due to overfitting), but not across the board (the graphs below don’t necessarily match the numbers mentioned in the post because an agent is constantly making changes, so any snapshot is a point-in-time estimate that becomes obsolete immediately):

If you’re curious about performance on specific benchmarks or specific classes of rebar benchmarks, we have the following table (ratios above one mean FRE is faster; below mean FRE is slower):

There’s also an AOT compiler mode that takes a long time to compile a regex to native code before running it. There isn’t AOT support for everything, but here are the results from the cases where it’s supported. As we can see, the AOT compiler is very slow (it loses very badly in the compilation time benchmarks) and, despite spending quite a bit of time compiling, results are often slower than with the standard FRE regex engine (though it’s also faster in many cases).

And then there are the holdout benchmarks. As noted above, for the non-AOT FRE code, performance on the holdout isn’t as good as on rebar. And as also noted above, considering that this is for a workload like ripgrep, the “hot search” set of benchmarks is probably more important than the others, so the FRE result is worse than the overall score would make it look.

One thing to note here is that, for the holdout benchmark cases where we don’t include compile time as part of the benchmark and we repeatedly run searches, AOT FRE outperforms on the benchmark. For a lot of use cases, you don’t want a regex that takes multiple seconds to compile, but there are plenty of cases where this is fine, e.g., for something like ripgrep or Silver Searcher, it could start running with a regex that can start matching right away and then compile in another thread and cut over to the faster matcher when it’s done compiling. Given how much of my CPU is spent on long ripgrep searches, it seems like a strategy like that could improve performance for work I personally do. Before LLMs, it probably wouldn’t have made sense to spend the effort to write an optimizing regex compiler, but this is now do-able with a few tokens.

Another thing to note here is that this comparison is arguably unfair because this was run on an ARM Graviton machine with SVE/SVE2 and FRE has SVE/SVE2 optimizations. Pre-LLM, it might not have been worth it to have regexes optimized for every combination of SIMD instructions out there, but with LLMs, it’s fairly easy to generate ok-ish SIMD optimizations. I know human experts who find that they can generally outperform LLMs here, e.g., Jay Stelly said that the last time he tried getting an LLM to produce SIMD code, it took 20-some iterations to get the code as good as he wanted. But, on the flip side, LLMs have the capability to try more optimizations than a human could possibly try in any given amount of time, so they can still perform pretty well overall even if any specific optimization isn’t as good as a human expert would produce.

There’s also the problem discussed in this post of overfitting. Depending on the context, that problem is somewhere from very easy to solve to a bit difficult to solve. I deliberately didn’t try very hard to solve the problem here to see what would happen, but I did manage to solve the problem without an outsized amount of effort when working on this Azul AI (just for example), but a lot of these big benchmark claims come when people spend little to no effort trying to avoid overfitting, or even negative effort. In the pre-LLM era, people would often pick highly unrepresentative microbenchmarks to show off how great their pet project is which, at least at a non-conscious level, involves negative effort to avoid overfitting to a benchmark. Due to how humans are, I don’t think people are going to stop making misleading claims and it’s become easier than ever to make misleading claims, so of course we see more of them.

Note that while this post has discussed non-AI software, everything said here goes double for AI software. For example, I’ve seen lots of people drop comments saying that Kimi K3 is Fable (5) level. But every single person I know who’s used it has found it to be substantially worse than GPT-5.6 Sol and Fable. I’m not saying it’s not an impressive engineering achievement, but the performance on a wide variety of real-world tasks isn’t up to the level it is in benchmarks. This even applies to various eval-y problems, such as when a friend tried different coding agents on the ICFP 2026 contest problems. It also applies to security issues, which are something that I have no doubt AI labs are putting into their evals, e.g., a colleague of mine tried using Kimi K3 to scan for vulns in our software and found that it found approximately a quarter of the vulns GPT-5.6 Sol found, found no vulns that GPT-5.6 Sol didn’t find, and didn’t have any advantages in any dimension other than on cost. The people I know who are using cheaper models to find real security issues are using other models, such as GLM-5.2, which perform worse on benchmarks but better in practice.

Back on the topic of FRE, one more note is that the holdout benchmark is an arbitrary subset of the ripgrep benchmark setup that was chosen by an agent for unknown reasons. I asked an agent to pull the entire benchmark suite, but that didn’t finish in time for this post, so I don’t know what the result will be once it’s done.