EmbedAgent and friends: what the LLM-for-embedded benchmarks actually measure

Somebody drops EmbedAgent into a thread about whether LLMs can write firmware. Somebody else drops the CHI paper. Then the argument proceeds as if the two are competing scoreboards and one of them must be wrong.

They aren't competing. They aren't even playing the same sport.

Short version: EmbedAgent measures whether generated code compiles and passes functional cases across a component and platform matrix. Englhardt et al. measure whether an LLM's reasoning helps a human, including in the runs where the code was broken. The MDPI Future Internet benchmark sits in the same lineage as the second one. All three are worth reading. None of them run the firmware long enough to find the class of bug that keeps firmware people up at night, and that gap is the whole reason "LLMs are at 55% on embedded" is a sentence with no meaning attached to it.

EmbedBench: 126 cases and a ceiling near 55%

EmbedAgent is the most benchmark-shaped of the three.¹ The paper builds EmbedBench: 126 cases spanning 9 electronic components and 3 hardware platforms, run against 10 mainstream LLMs. The framing is a nice piece of scoping — instead of one blob task, it splits the work into the roles an embedded developer actually switches between: programmer, architect, integrator. Writing the driver, deciding how the parts connect, making the whole thing work on a given platform.

The headline number is where it gets interesting. DeepSeek-R1, given the schematic information, lands at 55.6% pass@1. Made to generate the schematic itself first, it drops to 50.0%. The paper describes these cases as simple.

Sit with that for a second. Simple cases. Top reasoning model in the set. Coin flip.

The delta between the two conditions is the part I didn't expect. Handing the model the wiring is worth 5.6 points. If figuring out how the parts connect were the hard step, removing it should have bought a lot more than that. It suggests the failures are concentrated somewhere after the design decision — in the driver code, the init sequence, the platform glue.

Then there's the migration result, which is the number I keep coming back to. Asked to move working code across platforms, the best model hits 73.8% pass@1 on MicroPython for the Raspberry Pi Pico and 29.4% on ESP-IDF. Same task class. Forty-four points apart.

That spread is not a statement about hardware difficulty. A Pico and an ESP32 are both small microcontrollers with the same kinds of peripherals. It's a statement about how much of each toolchain exists in public code, and in what shape. MicroPython is short, uniform and everywhere. ESP-IDF is large, versioned, and full of code that only compiles against one specific release. The model is good exactly where the corpus is thick and consistent. That's a corpus benchmark wearing a hardware costume.

The paper also splits the failure modes by model type, and this is the observation I'd promote to the abstract if it were mine. General-purpose chat models like DeepSeek-V3 often fail to use relevant knowledge they demonstrably have. Reasoning models do the opposite: they overthink, and in doing so overlook the efficient piece of knowledge that would have solved it. Two different ways to be wrong about the same GPIO.

EmbedAgent then proposes two fixes: retrieval-augmented generation, and compiler feedback. With both, DeepSeek-R1 goes from 55.6% to 65.1% pass@1 with schematics, and from 50.0% to 53.1% without. Arduino-to-ESP32 migration goes from 21.4% to 27.8%.

I want to be fair about how good that is. Those are real gains and the compiler-feedback half is the part I'd bet on, because it's the one that closes a loop instead of widening a prompt. It's also the whole thesis of this blog arriving in someone else's paper: the model gets better when you let it find out it was wrong. But look at where it lands. Two thirds on simple cases with the schematic handed over. Barely over a quarter on the migration. The remedy works and the ceiling is still low.

The CHI paper: measuring whether wrong code was still useful

Englhardt et al. are asking a different question, and if you read them as a pass-rate paper you will misread them completely.²

Their setup is an automated testbench across 450 trials, plus a 15-user pilot study. The finding that matters: even when the LLM failed to produce working code, it generated helpful reasoning about embedded design tasks and specific debugging suggestions that were useful to both novice and expert developers.

That is a success criterion with no overlap with pass@1. A trial can be a total failure by EmbedBench's scoring and a clear win by this one. The model wires the sensor to the wrong bus but correctly tells you your I2C address is probably shifted a bit, and the human — who was going to write the code anyway — saves an hour.

This is an HCI paper doing HCI things, and the softer shape is the point rather than a weakness. It's measuring the tool as it's actually used in 2026: a developer in the loop, reading suggestions, keeping the good ones. Nobody I know ships an LLM's first draft unread. Measuring the first draft in isolation tells you less about the workflow than measuring what the draft did to the human.

The MDPI paper: same lineage, newer volume

The third one is "Benchmarking Large Language Models for Embedded Systems Programming in Microcontroller-Driven IoT Applications," in MDPI's Future Internet.³ It positions itself on the Englhardt line, building on and surveying that 450-trial testbench approach.

I'm not going to quote pass rates from it, because the honest state of my reading is that I have the title and the framing and not the result table. What's worth noting is the lineage itself. Two of the three papers here trace back to the same testbench design. When you see three citations in a thread and assume three independent measurements, you're counting one methodology twice.

Why stacking them up is a category error

Put the three side by side and the comparison collapses.

EmbedAgent's question is "did the machine produce artifacts that compile and pass a functional check." Englhardt's is "did the machine make a human more effective, even when it was wrong." Those aren't harder and easier versions of one test. A single trial can score 0 on the first and high on the second, and neither number is lying.

So when a summary says LLMs are at 55% on embedded, the only correct follow-up is: 55% of what, on which platform, with the schematic given or not, and scored by a compiler or by a person. Every one of those choices moves the number more than swapping the model does.

The thing none of them measure

Here's the part that made me want to write this up, and it applies to all three.

None of them run the generated firmware against real interrupt timing. None of them stress peripheral register races. None of them soak a build for days to see what drifts. They are testing "does this compile and pass a functional check," which is necessary and nowhere near sufficient for firmware anyone ships.

The bugs that own my calendar don't fail a functional check. They pass it every single time. A missing memory barrier that works until the compiler inlines differently. A DMA buffer read one cycle before the transfer completes, which is fine at room temperature and not fine at 70°C. An interrupt priority inversion that only bites when two peripherals go active in the same millisecond. A counter that wraps on day nine. All of those produce code that compiles, runs, and passes the check on the bench.

Which is the recurring point around here: the code takes minutes, proving it takes months. These benchmarks are almost entirely measuring the minutes. That's not a flaw in the papers — it's a flaw in how the results get repeated, because "55% pass@1 on simple cases" gets read as a progress bar toward shipping firmware, and it isn't one. It's a progress bar toward a first draft.

The same confusion shows up one layer up the stack, which I got into in Copilot for firmware is solved, and that was the easy half. Generation is the part that's going fine. Every benchmark here is scoring generation, with a compile step attached, and then people extrapolate from it to the validation problem sitting downstream. The extrapolation is where it goes wrong, not the measurement.

What I'd want measured

If I were extending EmbedBench, I'd keep the role split and the platform matrix and add one axis: how long the firmware ran before it was scored, and under what conditions. Pass at first boot is one number. Pass after ten thousand interrupt cycles is another. Pass with two peripherals contending for the same bus is a third. Those are three different levels of "working," and right now the literature collapses them into one bit.

I'd also want the ESP-IDF gap treated as a first-class result rather than a footnote. A forty-four point swing by toolchain, at fixed hardware difficulty, says the useful lever today is making a platform legible to the model, not making the model larger. That should be a paper.

Not a takedown

I'll be clear, because criticism of academic work compresses badly. These are good papers. EmbedAgent's contribution is a real matrix over real components with a real failure-mode analysis, and I'd rather have its 55.6% than another vibes-based thread about whether models "get" embedded. Englhardt et al. asked a question nobody else was asking and got a genuinely useful answer about wrong code still being valuable. The MDPI paper is continuing a line of work that deserves continuing.

The complaint is about the citation layer, not the research. Three papers measuring three things get repeated as one score. And the bar all three stop at — compiles, passes a functional check — is exactly where firmware validation starts getting expensive.

Somebody should benchmark that half. Until then, when you see a pass rate for LLMs on embedded, ask what happened after the check passed. Usually the answer is: nothing, the trial ended.

Sources

Footnotes

1. "EmbedAgent: Benchmarking Large Language Models in Embedded System Development," arXiv preprint arXiv:2506.11003v3, 2025. https://arxiv.org/html/2506.11003v3 ↩

2. Zachary Englhardt et al., "Exploring and Characterizing Large Language Models for Embedded System Development and Debugging," CHI EA '24 (Extended Abstracts of the CHI Conference on Human Factors in Computing Systems), ACM, 2024. https://zachary.englhardt.com/pdfs/llm_embedded.pdf ↩

3. "Benchmarking Large Language Models for Embedded Systems Programming in Microcontroller-Driven IoT Applications," Future Internet 18, no. 2 (2026): 94, MDPI. doi:10.3390/fi18020094. https://www.mdpi.com/1999-5903/18/2/94 ↩