The build passed. Why doesn't it run?

The firmware compiles. No warnings. No errors. The embedded CI pipeline turns green.

Then you flash it to the board and nothing happens.

Not a crash. Not a fault. Nothing. The UART is silent. The LED doesn't blink. The state machine never starts. The code is there—the linker proved it—but the chip isn't running it the way you expected.

This is the gap nobody talks about in embedded development—the space between "the build passed" and "it works on hardware." Most firmware bugs live here.¹

The compile-green trap

The compiler is ruthless about syntax. It catches stale APIs, missing dependencies, broken types, and the language rules you forgot existed. But the compiler has no opinion about whether your code actually does what you intended.

An address is an address. A register write is a register write. If you tell the compiler to initialize USART1 instead of USART2, it will happily compile that. If you forget to enable the clock for a peripheral, the compiler doesn't care. If you configure the wrong GPIO pins, the linker will place the code in flash and call it done.

All of these compile clean. All of them produce silent failures on hardware.

The problem is asking the compiler to be the final gate.

What the run catches

A virtual board run is the next layer. You upload the ELF, the simulator boots the chip, executes the firmware for a fixed budget of virtual time, and captures what the UART says. That catches a different class of failure: the ones that compile but don't behave.

We demonstrated this in our earlier post. A small language model's nRF52840 attempt compiled without a single warning. It used the modern UARTE peripheral, satisfied the borrow checker, and looked plausible on review. But when it ran on the virtual board, it produced nothing. The firmware had written to a DMA buffer in flash—which the peripheral couldn't read—so the UART stayed silent. The run said so immediately: five seconds of virtual CPU time, zero bytes on the wire.

That's the exact failure class the compiler can't catch, and it's unambiguous the moment the firmware actually runs.

Here's what we saw when we tested a broken STM32F4 firmware that uses the wrong UART peripheral:

[chiplab] run complete (board: stm32f4_discovery)
[chiplab] uart0: (no output captured)

The firmware booted. The code executed. The linker placed everything correctly. But the UART peripheral it was trying to write to wasn't the one connected to the board's debug connector—a silent failure the run caught in seconds.

Then we fixed it—switched to the correct peripheral—and ran it again:

[chiplab] run complete (board: stm32f4_discovery)
[chiplab] uart0: Hello world!

Same board, same toolchain, same linker. Different peripheral. The run is the receipt.

The failure classes and where they die

Not every bug needs a physical board to catch. The layers protect each other:

Failure class	Compiler	Virtual board	Physical hardware
Stale API or missing dependency	✓ Catches	Not needed	Not needed
Wrong peripheral mode or clock gate	✗ Misses	✓ Catches	Can confirm
Firmware boots but produces no output	✗ Misses	✓ Catches	Can confirm
Code hangs waiting for an event	✗ Misses	✓ Catches	Can confirm
Exact timing, pin routing, electrical behavior	✗ Misses	✗ Insufficient	✓ Final authority
Real sensor, actuator, and fault injection	✗ Misses	✗ Insufficient	✓ Final authority

The goal is fewer bad binaries reaching the expensive layer. Automotive research has been formalizing exactly this move, building systematic ways to decide which test cases can shift off a hardware-in-the-loop bench and onto a simulation instead.²

Why this matters now

Embedded teams used to accept a slow feedback loop because firmware changes were slow too. A human wrote the code, flashed the board, and watched the output. The whole loop was human-paced.

AI can generate firmware faster than a bench can be booked. If an agent can propose ten changes before lunch, a team needs a way to reject the obviously broken ones before those changes compete for a shared bench. The verification system has to handle candidate code at machine speed, even when the final answer still comes from real hardware.³

That means the pyramid becomes a funnel: many changes enter at the top, few make it to the bottom.

Run the cheap checks first. Keep the bench for what needs a bench.

We've made the harder version of this case already: HIL alone can't scale to how fast an agent generates firmware. This post is about the gate that runs before the bench ever sees the code.

The boundary is real

Simulation doesn't get a free pass here. In our own probes, a byte-level simulation model accepted clock and configuration mistakes that real hardware wouldn't forgive. We wrote that down instead of editing it out, because the boundary matters.

A virtual target is a test environment with a scope, not a magic copy of reality. So is HIL. So is a vehicle. The point is to know what each one can prove.

The simulator catches structural lies—wrong peripheral, wrong pin, code that hangs—but doesn't model exact timing, electrical behavior, or the analog quirks that only show up under load. That's what the bench is for, and in a safety context the standards name it explicitly: ISO 26262 leans on HIL for software integration testing and for verifying software safety requirements.⁴ But the bench shouldn't be the first place you find out that the firmware never enabled the UART clock.

We've hit this same boundary from two other angles. Renode and QEMU ran the same STM32 firmware and diverged on what counted as a valid boot. And when we planted a real crash on purpose, Chiplab booted straight through a HardFault it doesn't model yet. Same lesson every time: know what a layer can prove before you trust it.

Can firmware CI run without hardware?

For most failure classes, a virtual board catches them before a physical board gets touched. Wrong peripheral, wrong clock gate, firmware that hangs waiting for an event: all caught by simulation. Firmware CI without hardware isn't a replacement for the bench. It's a filter that keeps bad binaries from reaching it. That ordering is the standard recommendation: run everything you can without devices first, and only let jobs touch real hardware once the cheap checks pass.⁵

The bench still handles exact timing, electrical behavior, and failures that only show up under real load. A HIL rig exists precisely because testing everything against the complete embedded system is impractical on safety, availability, or cost grounds.⁶ The point isn't to skip hardware. It's to stop spending it on bugs a simulator would have caught in five seconds.

What an embedded CI pipeline needs now

Embedded continuous integration used to be: build, link, ship to the bench.

It needs to be: build, link, run on virtual hardware, then ship to the bench.

The compiler is ruthless and cheap. The virtual board is the next gate. The bench is the final sign-off.

That's how you keep the expensive resource valuable in the age of generated code: not as the place every change begins, but as the place the surviving changes earn their final sign-off.

The build passed. Now make it run.

Sources

Footnotes

1. Stack Overflow, "Continuous Integration on hardware-centric firmware," 2023. https://stackoverflow.com/questions/70303010/continuous-integration-on-hardware-centric-firmware ↩

2. Keil et al., "Evaluation of SiL Testing Potential—Shifting from HiL by Identifying Compatible Requirements with vECUs," Vehicles, 2024. https://doi.org/10.3390/vehicles6020044 ↩

3. Sevenhuijsen, Patil, Nyberg, and Ung, "Generating Safety-Critical Automotive C-programs using LLMs with Formal Verification," PMLR 284, 2025. https://proceedings.mlr.press/v284/sevenhuijsen25a.html ↩

4. Himmler, A. (dSPACE GmbH), "Hardware-in-the-Loop Testing of Safety-Relevant Functions in the Context of ISO 26262," ERTS 2012. http://web1.see.asso.fr/erts2012/Site/0P2RUC89/TC-2.pdf ↩

5. Semaphore, "What CI/CD strategies work for embedded or IoT projects that require hardware testing?" 2024. https://semaphore.io/blog/what-ci-cd-strategies-work-for-embedded-or-iot-projects-that-require-hardware-testing ↩

6. NI, "HIL Test System Architectures." https://www.ni.com/en/solutions/transportation/hardware-in-the-loop/hardware-in-the-loop--hil--test-system-architectures.html ↩