Evaluating LLMs via Esoteric Programming Languages

Current benchmarks for large language model (LLM) code generation primarily evaluate mainstream
languages like Python, where models benefit from massive pretraining corpora. This leads to
inflated accuracy scores that may reflect data memorization rather than genuine reasoning ability.
We introduce EsoLang-Bench, a benchmark of 80 programming problems across five
esoteric languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare) where
training data is 5,000 to 100,000x scarcer than Python.

We evaluate five frontier models using five prompting strategies and two agentic coding systems.
The best-performing model achieves only 3.8% overall accuracy, compared to
~90% on equivalent Python tasks. All models score 0% on problems above the Easy
tier, Whitespace remains completely unsolved (0% across all configurations), and self-reflection
provides essentially zero benefit. These results reveal a dramatic gap between benchmark performance
on mainstream languages and genuine programming ability, suggesting that current LLM code generation
capabilities are far narrower than headline metrics imply.