✨ Explore this must-read post from Hacker News 📖
📂 **Category**:
💡 **What You’ll Learn**:
We got pretty excited yesterday when OpenAI put a demo of GPT-6 Astra working on a circuit board in KiCad on the front page of its
launch post. It is cool to see electronics show up in a major model release like this.
We are obviously still some distance from asking an AI to build an entire phone in one prompt. The demo does
raise a question we have been thinking about for a while, though: how do we measure whether the electronics an
AI produces are actually any good?
The models know a surprising amount about electronics
Our experience has been that current models know much more about electronics than their output in
conventional design tools tends to show. They have read textbooks, datasheets, application notes and a lot of
code.
You can have an agent operate a graphical CAD tool, but it spends a lot of time clicking around and keeping
track of what is on screen. A lot of its context consists of coordinates, menus and application state.
EEBench uses atopile instead. The circuit
lives in declarative code, so the agent can work directly on components, connections and electrical
constraints. It can change the design, build it, run a simulation and inspect what failed without leaving the
project.
This has worked much better for us than asking a model to draw lines in a GUI. It also means the benchmark
can spend less time testing computer use and more time testing electronics.
.ELEC: @STD::Import 💬
.Submission: @type ⚡The real world is messy
One of the public tasks is based on a residential energy meter. When its 5 V supply disappears, the circuit
has to keep the processor alive for another 20 ms so it can save the accumulated reading. The protected rail
must stay above the processor’s 3.0 V brownout threshold during that window.
Most models intuitively jump to the right base conclusion: add a capacitor.
A real capacitor makes the task more interesting. A ceramic part may provide much less than its advertised
capacitance once it has voltage across it. Parts have tolerances. Adding more capacitance costs more, takes up
space and makes the rail slower to recharge when the power returns. A design that works with nominal values
can fail with the parts that arrive.
EEBench cuts the input power in simulation and measures what happens. It checks the voltage throughout the
outage, the effective capacitance at the operating point, the recovery after power returns and the limits on
package, dielectric, voltage rating and cost.
after 0.85 ms.
The meter is one of the easier tasks. In a harder analog task, the agent may have to synthesize a
multiple-feedback low-pass filter around an op-amp, solve the resistor and capacitor ratios for the required
poles, and keep its gain, cutoff frequency and Q inside their limits after every component is pushed to a
worst-case tolerance corner. The harness rebuilds the SPICE deck for those corners, runs the AC and transient
captures, binds measurements to named probes, and records each result against its lower and upper
specification limits.
But getting the equations right is only part of electronics engineering. EEBench uses real manufacturer
parts, with specifications extracted from their datasheets and carried into the SPICE model. The agent has to
find a combination that works across those tolerance corners while also choosing parts that exist, can be
ordered and are reasonably priced for the product. That trade-off between electrical performance, cost and
supply is much closer to designing real hardware than picking ideal values from a textbook.
This is the part we find most interesting, because it is what electrical engineering eventually boils down
to, just like every other engineering discipline: trade-offs.
How the grading works
EEBench checks are fully deterministic. It builds the submitted design, constructs the circuit
graph and bill of materials, and runs a set of SPICE simulations and design checks. Each requirement produces
a measurement with a limit.
For the energy-meter task, the harness measures the protected rail while the input drops out and returns.
Other tasks measure gain, thresholds, ripple, transient response and behavior at component-tolerance corners.
The technical score is combined with cost efficiency against a reference bill of materials. Cost only helps
once the circuit works.
This is similar to giving a coding agent a compiler and tests, except the tests are measuring voltages and
component behavior..
EEBench V1 covers analog and digital design through simulation. It does not yet tell us whether a model can
lay out, manufacture and bring up a complete product. We want to add those parts later. The current benchmark
concentrates on the requirements, design and verification loop because that is where we can already grade
useful engineering work objectively. The full methodology and sample result
explorer are public.
What we are seeing on the leaderboard
The September 1 results are encouraging. Claude Opus 5 scored 61.6% across the 13 tasks in EEBench V1. Grok
4.6 came second at 57.1%, just ahead of Claude Fable 5.1 at 56.4%. A few months ago we would not have expected
models to do this well.
1Claude Opus 561.6%
2Grok 4.657.1%
3Claude Fable 5.156.4%
4Claude Fable 554.3%
5Claude Opus 4.8 Max51.4%
See the full leaderboard and run details
There was another result we were especially happy to see: xAI included EEBench in the Grok 4.6 model
card. It appears in the section on “engineering acceleration,” alongside evaluations for 3D modeling and
parametric CAD. Their published run put Grok 4.6 at 60.0% with xhigh reasoning effort. Seeing a frontier lab
use EEBench to describe a new model’s engineering ability makes us think this is becoming a category people
care about.
Anthropic’s models have consistently done well in this environment. Grok’s rise is also interesting. In its
Grok 4.6 launch post, xAI says the
model received high-quality engineering data and RL training in domain-specific environments including
computer-aided design. Its EEBench result fits that story.
The OpenAI models we have tested so far sit further down the table. GPT-5.5 scored 42.3%, while GPT-5.6 Sol
scored 39.4%. We do not have a GPT-6 Astra result yet. After seeing it work on a board in KiCad, we would
really like to find out how it handles these circuit-design tasks.
Training environment
Once we had a simulation harness that could grade a circuit, we also had the beginnings of an RL environment
for electronics. The same checks can be used as reward signals during post-training.
A failed run contains useful information. We can see which voltage missed its limit, which operating corner
failed, or whether the model solved the problem with an unnecessarily expensive design. That gives a training
loop more to work with than a model saying that a schematic looks plausible.
EEBench is the small, public view of this work. We are also starting to work directly with frontier labs that
want to make their models better at electronics. If you work on evals or post-training, you can talk to us about the larger
evaluation suites and simulation-backed training environments we provide.
So, can it design a circuit board?
For a useful and growing set of circuit problems, we think the answer is already yes. The scores also make it
clear that there is plenty left to do. OpenAI choosing a PCB for one of Astra’s first demos and xAI publishing
EEBench in a model card both feel like early signs of the same thing: AI labs are starting to take electronics
seriously.
There may be another data point very soon. Elon Musk has said that Grok 4.7 is coming within weeks after additional training on a
large collection of SpaceX data, with the aim of making it especially good at engineering. The model is not
out and the schedule may move, but if it arrives as described, we will be keen to put it on EEBench.
It looks like we are in for an interesting few weeks. We will keep adding harder tasks as the models improve,
and we are looking forward to seeing how Astra and the next generation perform.
You can try the same approach in atopile
today. Give the agent a board you have been meaning to build and see how far it gets.
So, can AI design circuit boards yet? Some of them, yes. We still would not ask it to design a pacemaker and
blindly install the result. But we are on the way there.
EEBench is built and funded by the team behind atopile. We pay for the public
benchmark runs and do not sell benchmark scores.
⚡ **What’s your take?**
Share your thoughts in the comments below!
#️⃣ **#design #circuit #boards #EEBench**
🕒 **Posted on**: 1788553874
🌟 **Want more?** Click here for more info! 🌟
