"The code takes minutes. Proving it takes months." — anatomy of a firmware validation cycle

I repeat this line often enough that people nod at it without making me prove it: the code takes minutes, proving it takes months.

So here is the proof, stage by stage.

The short version, if you're only here for one paragraph: in a firmware validation cycle, every stage that involves typing takes minutes to hours, and every stage that involves physical hardware takes days to months. There is almost nothing in between. The gap isn't caused by firmware engineers being slow. It's caused by the number of boards in the lab being a fixed, small integer.

The cycle we inherited

Embedded didn't arrive at this by accident. Microgenesis, writing about CI/CD for embedded systems, describes the starting point plainly: development "often followed a waterfall model — long design phases, months of coding, and extensive manual testing before release."¹ The consequences they list are the ones every firmware team recognizes: "slow time-to-market, late detection of defects, and painful certification cycles."

Their argument is that two forces are now breaking that model. One is compliance: ISO 26262 in automotive, IEC 62304 in medical. The other is that customers expect over-the-air updates, which means the release you shipped is not the last release you ship. Both of those turn a slow validation cycle from an annoyance into a structural problem.

I agree with all of it. I want to zoom in on the part their piece treats as a single block and take it apart, because "extensive manual testing" is doing an enormous amount of work in that sentence.

Anatomy of a firmware validation cycle

Here's a realistic cycle for a nontrivial change. Say you're fixing a bug in a CAN driver's error-frame handling on an automotive ECU. Nothing exotic. The kind of ticket that closes every week somewhere.

Write the code — minutes to hours

You read the datasheet section, you understand the error counter behavior, you change maybe forty lines. If you're pairing with an agent, this part is faster than that, and getting faster every month.

This stage is not the problem. This stage has never been the problem.

Compile and unit checks — minutes

Cross-compile, run whatever host-side unit tests exist, check the map file didn't blow past your flash budget. Static analysis if your build runs it locally.

Fast. Fully automated. Runs on any machine. Also, notably, tells you almost nothing about whether the fix works, because a CAN error counter's behavior is a property of a peripheral you just compiled against, not a property of the code.

Board bring-up and flash — hours, if a board is free

Now you need a target. This is where the cycle stops being about software.

You need a board of the right revision, a JTAG or SWD probe attached to it, a power supply, a CAN transceiver on the other end, and a machine with the right toolchain and probe drivers talking to all of it. In practice, that combination doesn't sit idle waiting for you. It's on someone's desk, or it's in the rack running a different test, or the probe walked off with whoever needed it last.

The flashing takes seconds. Getting to the point where you can flash commonly takes the better part of a day. I've watched engineers schedule their week around who has the debug probe.

Integration testing on real hardware — days

Your fix touches the CAN driver, so now you care about the rest of the system. Does the scheduler still meet its deadlines with the new error path? Does the bootloader still hand off cleanly? Does anything else on the bus notice?

This runs on shared hardware, in a lab, against other people's schedules. Every iteration costs a round trip to the bench. Every failure sends you back to read a register dump over a serial line at whatever baud rate someone picked in 2019. Days, and most of that is queueing, not testing.

HIL rig testing for timing and fault cases — days to weeks

The interesting failures don't show up in nominal operation. They show up when the bus floods, when a node drops mid-transmission, when a supply browns out during an interrupt. That's what the hardware-in-the-loop rig is for: real firmware, real MCU, simulated plant and simulated faults around it.

There is one rig. Maybe three, if you're at a large OEM. It's booked. It's booked because every team needs it, and because the setups it runs are physical wiring harnesses that take real effort to swap. Getting a fault-injection slot on a HIL rig is a calendar negotiation, not a command you type.

This is where weeks start appearing, and it's the single least elastic stage in the whole cycle.

Regression across hardware revisions — weeks

Your product doesn't ship on one board. It ships on rev B, rev C, and the pre-production rev D that only exists in quantities of four. Silicon errata differ between them. Clock trees differ. Sometimes a peripheral you depend on got moved.

So the regression suite is the matrix of every test you care about times every revision you support, and it runs on physical units of which you own a countable number. When a board revision changes mid-cycle — and in hardware development it does, because the hardware team has its own deadlines — you don't extend the matrix, you re-run it. Weeks, and the ones you didn't plan for.

Compliance and certification paperwork — months

If you're in a regulated industry, this is the stage that dwarfs everything above it.

ISO 26262 and IEC 62304 don't just want working firmware. They want evidence: requirements traced to design, design traced to tests, tests traced to results, coverage reports, static analysis findings dispositioned one by one with a human signature, tool qualification arguments for the compiler and the test harness. Documented, reviewed, archived.

None of this is fake work. I'd genuinely rather my brake controller went through it. But it's measured in months, it's largely manual, and it's the reason Microgenesis calls certification cycles "painful" without needing to elaborate.

Field validation and release-candidate soak — weeks

Last stage. You put the candidate on real units in something resembling real conditions and you leave it there, because a class of firmware bugs only appears after three hundred hours: a counter that wraps, a heap that fragments, a watchdog that fires once a fortnight.

You cannot compress a soak test by working harder. You can only start it earlier.

Add up the bands

Two stages measured in minutes and hours. Six measured in days, weeks, and months.

And look at what separates them. It isn't difficulty. Writing correct interrupt-priority code is harder than running a regression matrix. It's whether the stage needs a physical object that exists in limited quantity: a board, a probe, a rig, a reviewer's signature.

That's the whole shape of it. The fast stages scale with how many engineers you have. The slow stages scale with how many boards you have. Hiring doesn't fix the second number, and neither does anyone's productivity tooling.

CI/CD is real, and it optimizes the fast part

I don't want this read as a swipe at embedded CI/CD. Microgenesis is right about the direction, and the benefits they list are the correct benefits: "accelerated feedback loops" and "improved software quality," achieved by automating builds and tests on simulators so developers get "immediate feedback" instead of waiting on hardware prototypes or manual QA.¹ Every firmware team should be doing this. Many still aren't.

But watch which stages a typical embedded pipeline actually covers. It builds. It runs unit tests. It runs static analysis. It maybe boots a smoke test somewhere. In other words, it takes the two stages that already took minutes and makes them take fewer minutes, on a schedule, without a human remembering to press the button.

That's worth having. It is not where the months are.

I've written the longer version of this argument in firmware CI ends at the linker: the pipeline proves the artifact was produced, then hands it to a human to find out whether it works. Everything after the linker — the six slow stages above — stays gated behind physical rigs and physical boards, even in teams with a beautiful green pipeline. Automating the fast half of a cycle whose time lives in the slow half improves the developer experience and barely moves the release date.

What actually shrinks the slow stages

If the constraint is "this stage needs a physical object that exists in limited quantity," there are exactly two ways out. Buy more physical objects, or stop needing them for some of the work.

The first one is what teams do today, and it doesn't scale: boards cost money, rigs cost more, and pre-production silicon can't be bought at all because there are twelve units in the world and they're on a shelf in a different country.

The second one is simulation, and the reason it matters isn't that it's cheaper. It's that a simulated target scales horizontally. Fifty parallel instances of a virtual MCU is a scheduling decision. Fifty parallel instances of a physical board is a purchase order, a rack, and someone to wire it.

That changes which stages can move earlier:

• Peripheral-register-level tests. Does the CAN error counter increment the way the reference manual says? That's a question about registers, and a simulator that models the peripheral can answer it before a board is involved.
• Interrupt-timing and priority behavior. Does the new error path still let the high-priority ISR preempt? Deterministic on a simulator, and repeatable, which is more than you can say for a bench.
• Fault injection. Bus errors, unexpected resets, corrupted frames. On a rig this is a wiring change and a booked slot. On a simulator it's a line of test setup.
• The revision matrix. Different memory maps and different peripheral configurations are configuration files, not four more boards you don't own.

Simulation does not replace the HIL rig. I've said this often enough that I wrote the routing procedure down: simulation vs HIL, what belongs in CI is the decision rule for which checks move earlier and which ones have to stay on real silicon. Analog behavior, real timing under real electrical conditions, anything where the plant model is the point — that stays on the rig.

The win isn't deleting the slow stages. It's arriving at them with the boring failures already found. A HIL slot spent discovering that you mishandled an error counter is a wasted HIL slot. There's one rig. Don't send it your typos.

What this does not fix

Two honest limits, because a post that claims to delete the months would be lying.

Compliance paperwork is still paperwork. Traceability matrices, static analysis dispositions, tool qualification, review signatures. Simulation can produce more test evidence earlier and produce it repeatably, which is not nothing when a report has to cite results. But the process is human, it's regulated, and nobody at Veecle has eliminated a certification cycle. Anyone telling you otherwise is selling.

And the soak test is still a soak test. Three hundred hours is three hundred hours. The only lever is starting it on a candidate that's already survived everything cheaper.

The part I actually care about

I mostly write about coding agents, so here's the connection.

An agent writing firmware is fine at the minutes-long stages. It's fine at the code, and increasingly good at it. Then it hits the first stage that needs a board and it stops having a feedback loop at all. It can't queue for the rig. It can't ask who has the probe. It generates a patch, and the loop that would tell it whether the patch was right is a physical process measured in weeks.

That's the same wall the human team hits, just more visibly, because a human will at least wait patiently and an agent will confidently hand you something unverified.

Which is why I keep coming back to the same sentence. The code takes minutes. Proving it takes months. And every hour of that gap that lives in "no board was free" is an hour we put there ourselves.

Sources

Footnotes

1. Chandra Narayanan, "CI/CD for Embedded Systems: Tools, Pipelines, and Strategies," Microgenesis, Oct 23, 2025. https://mgtechsoft.se/blogs/ci-cd-for-embedded-systems-tools-pipelines-and-strategies ↩ ↩²