The LLM wrote the skills. The ESP-IDF score went down.

The default fix, once an LLM fails at firmware, is to give it more text. A RAG corpus. A skills folder. The datasheet in the prompt. Something in, better code out.

A March 2026 paper measured that move. The skills the model wrote for itself made ESP-IDF worse.

What they ran

IoT-SkillsBench is a hardware-in-the-loop benchmark. Three platform-framework pairs: ATmega2560 with Arduino, ESP32-S3 with ESP-IDF, nRF52840 with Zephyr. 42 tasks, three difficulty bands, 23 peripherals. Each task under three agent setups: no skills, LLM-generated skills, human-expert skills. 378 runs, checked on real boards, not in QEMU.¹²

I have already argued that most LLM-for-embedded benchmarks do not measure the thing people cite them for. This one is closer to a firmware test than most, because the pass condition is behavior on silicon.

The skills themselves follow the usual pattern: YAML header, markdown body, one peripheral or framework concern per file. The LLM set was produced by asking Claude Sonnet 4.6 to summarize the relevant knowledge from parametric memory only — no datasheets, no error logs, no extra docs. The human set was written by people who had the failing code, the compiler errors, and the runtime behavior in front of them.¹

The expert skills are not a mystical prompt. They are a short note written after watching the model fail on the bench.

The number

On ESP-IDF, no skills: 31 out of 42 tasks. LLM-written skills: 27 out of 42. Human skills: 41 out of 42.¹

Level 3 — interrupts, multi-device work, scheduling — is where the drop concentrates. No skills resolved 7 of 14. LLM skills resolved 4 of 14. The paper's own reading is that synthesized skills "can sometimes reinforce incorrect assumptions about complex ESP-IDF-specific behavior."¹

Arduino barely moved. 42/42 with no skills, 41/42 with LLM skills, 42/42 with human skills. Zephyr went 28/42 to 27/42 with LLM skills, then 41/42 with human skills. The only human misses the paper calls out are a 5 V RTC on a 3.3 V ESP32-S3, and a rotary encoder whose direction is not standardized. Those are hardware ambiguities, not prompt failures.¹

Token cost went the wrong way too. No-skills sat around 300 input tokens per task. LLM skills ran 8,500 to 9,500, plus the model talking to itself about the skill before emitting code. Human skills landed in the middle and almost cleared the suite.¹

So the expensive condition was also the worse one.

"Just add RAG" is a retrieval story, not a truth story

This paper did not kill retrieval. It killed a lazier claim: that any extra structured text about the chip is an upgrade.

The LLM skills were not retrieved from a good corpus. They were generated from the same weights that were already failing the task. That is closer to asking the student to write the cheat sheet, then grading the exam they take with it. If the student is wrong about ESP-IDF init sequences, the cheat sheet is wrong in a more confident font.

I have made the adjacent argument before: LLMs write plausible firmware and cannot prove it, and what they get wrong is usually hardware context, not C syntax. SkillsBench adds a measurement. Injecting ungrounded "knowledge" is not a neutral act. It can overwrite the cases the model would have gotten right.

The human skills worked because they were grounded in the failure. Compiler log, runtime behavior, one peripheral, short prose, almost no sample code. That is closer to a lab notebook than to a scraped SDK dump.

What I am not claiming

This is one paper, one agent scaffold, one model used to write the skills, three boards. It is not a law of RAG. A corpus built from datasheets and errata, retrieved against the actual error, is a different experiment. SkillsBench did not run that experiment, and I am not going to cite it as if it had.

It also does not say "skills are useless." The human set almost maxed the benchmark. The variable was the source of the text, not the presence of a skills folder.

If you are going to add skills to an embedded agent, the test is simple. Keep the no-skills baseline. Measure the suite again after the skills land. If the score drops, you did not add knowledge. You added a second, worse model of the chip, and you paid tokens for the privilege.

Sources

Footnotes

1. Li et al., "Skilled AI Agents for Embedded and IoT Systems Development," arXiv:2603.19583, 2026-03-20. https://arxiv.org/abs/2603.19583 ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

2. IoT-SkillsBench, GitHub repository, accessed 2026-08-14. https://github.com/iot-agent/iot-skillsbench ↩