RTOS trace in a simulator: the format was never the hard part

I wanted a scheduler timeline out of a simulated board. No probe, no bench, just a run in CI that tells me which thread held the CPU and when it gave it up.

The tracing format turned out to be the easy part. The pipe the bytes come out of is where the whole thing lives or dies, and that is the part nobody writes about.

Three unrelated things are called "trace"

RTOS event trace is the kernel telling you about itself. Zephyr's hooks fire on thread switches, semaphore operations, ISR entry and exit, and the format layer serializes each one into an event. This is the only one of the three that knows what a thread is.

CPU instruction trace is the simulator recording what the core executed. Renode does this with cpu CreateExecutionTracing, in PC, Opcode, PCAndOpcode or Disassembly mode, and the output feeds coverage tooling.¹ It is a superb receipt for which lines ran. It has no idea that a scheduler exists, because a context switch is just more instructions.

Waveform capture is pin state over time, the thing a logic analyzer draws. Renode does not have it. A GPIO logic analyzer view has been an open feature request since June 2023.²

Three different questions. Teams conflate them constantly, then get frustrated that the tool they picked cannot answer the question they actually had.

Tracealyzer and SystemView are wired to a debug probe

Search tracealyzer renode and you find people trying to point a commercial RTOS visualizer at a simulated machine. The obstacle is transport.

Percepio's TraceRecorder ships as a Zephyr module, and the stream ports exposed in Zephyr's configuration system are Ring Buffer, RTT, ITM and Semihost.³⁴ Ring Buffer means you dump RAM from a debugger. RTT means a host-side reader polls a control block in target RAM, commonly through J-Link tooling. ITM means the Cortex-M trace unit and a fast probe on SWO. Semihost writes a host file and is scoped to QEMU. There is no UART stream port in that list.

SEGGER SystemView is blunter still. Zephyr's own documentation says the payload "relies on RTT as a transport", and that newer SystemView versions support other transports such as UART or snapshot mode, both "still not supported in Zephyr".⁵ SEGGER shipped UART and TCP/IP acquisition in SystemView V3.10 in December 2019.⁶ The host tool has had a serial recorder for over six years. The Zephyr integration still asks for the probe.

That is the crux for anyone running without hardware. Delete the probe and something still has to poll the RTT control block while the core runs. Neither integration ships that compatible host-side reader for a simulated target.

Zephyr's own CTF format does speak UART

The in-tree path avoids the problem entirely. Zephyr's Common Trace Format layer is transport-agnostic, and the supported backends are UART, USB, File on the POSIX targets, RTT, and RAM.⁵ The tracing sample carries a config for exactly this combination:

CONFIG_TRACING=y
CONFIG_TRACING_CTF=y
CONFIG_TRACING_BACKEND_UART=y
CONFIG_TRACING_BUFFER_SIZE=4096

That is prj_uart_ctf.conf, verbatim, from the sample in Zephyr 4.4.1.⁷ The sample README adds the detail that decides whether any of this works: you set the zephyr,tracing-uart property under the devicetree chosen node, which is how you keep trace bytes off the console UART.⁸ Decoding is Babeltrace or TraceCompass pointed at a directory holding the CTF metadata file next to the captured channel.

No probe, no vendor license, no RTT. On paper this is the simulator-friendly option, so I ran it.

What actually came back

Zephyr 4.4.1, the tracing sample built for the nRF52840 DK with prj_uart_ctf.conf, trace backend bound to UART1 so the console could keep UART0. Chiplab ran it on Renode 1.16.1 for a bounded simulation window.

The run produced exactly one artifact: combined stdout.

$ python3 -c "b=open('stdout.bin','rb').read(); \
print(len(b), sum(1 for c in b if c < 9 or 13 < c < 32 or c > 126))"
16264 3750

$ strings -n 4 stdout.bin | grep -E 'thread_a|thread_b|idle|tracing_thread' | sort -u
idle
thread_a
thread_b
tracing_thread

Sixteen kilobytes, of which 3,750 bytes fall outside printable ASCII. Those are UART1 binary fragments, interleaved with console text in one stream. The visible strings are the giveaway that this is real trace payload rather than log output: thread_a is the K_THREAD_DEFINE name, thread_b is set by k_thread_name_set in the sample's own source,⁹ and tracing_thread is the tracing subsystem's worker, named the same way in the tracing core.¹⁰ Those names ride in CTF thread events. The sample's printk lines say "Hello World from" and look nothing like that.

Enough of the chain held to put trace-shaped payload into capture. Hooks compiled in, the UART backend transmitted binary data, and trace-specific thread names survived. Without a successful decode, I cannot claim that a complete valid CTF stream survived intact.

What I do not have is a clean channel file. The trace reached capture mixed into combined stdout rather than delivered as a separate decoder-ready UART stream, and I am not going to fabricate a Babeltrace session I never ran. Partial result, reported as partial.

A clean UART stream is necessary, not sufficient

This is the part worth internalizing before you blame the emulator. A CTF decoder wants the stream and nothing but the stream. Anything else in the byte sequence is an event header as far as the parser is concerned.

Zephyr issue 49889 shows the next wall on a physical reel_board with a real serial cable. The reporter commented out every printk in main.c so only trace data would go down the UART, captured with the in-tree trace_capture_uart.py, then ran a second capture specifically to dodge the boot banner. Babeltrace 1.5.8 still answered [error] Event id 32 is unknown, and Babeltrace 2 produced the structured equivalent.¹¹ No simulator was involved anywhere in that report. Separating the streams removes one source of corruption; it does not guarantee that the remaining bytes decode.

The Renode side of the same class shows up in issue 699, where semihosting UART output simply never appeared on the analyzer the user was watching.¹² Getting a second byte stream out of a simulated machine, into its own file, is configuration work that nobody's quickstart covers, because every quickstart assumes one console and one human reading it.

The fix is unglamorous and entirely mechanical: bind each UART to its own sink, keep console and trace separated from the peripheral all the way to disk, then decode. That is the next run, and it is a plumbing change, not a tracing change.

Why this matters more for agents than for humans

A human debugging with a board on the desk works around a contaminated stream without noticing. Squint at the console, ignore the garbage, move on.

An agent cannot. It gets an artifact, and the artifact either parses into events or it does not. "Mostly the right bytes, in the wrong container" is worthless to a machine reader and looks identical to success from the outside, which is the failure mode I care about most. The run exits zero. Something got captured. Only a decode attempt tells you whether observability actually happened.

That is the same line I keep drawing between what simulation proves and what it merely runs. Instruction trace and event trace both come free of hardware, and neither covers what simulation still cannot catch. Deciding which of these belongs in a pipeline at all is the argument in simulation versus hardware in the loop, and the habit of checking a tool's claims against its own issue tracker is the one from the simulator comparison audit.

Trace was supposed to be the observability story for firmware without hardware. It half is. The formats are open, the hooks are upstream, the decoders are free. The transport is where six years of a shipped UART recorder still has not reached the integration, and where a working RTOS trace turns into a file no decoder will read.

Sources

Footnotes

1. Renode documentation, "Execution tracing." https://renode.readthedocs.io/en/latest/execution-tracing/execution-tracing.html ↩

2. Renode issue #478, "GPIO logic analyzer view," opened 2023-06-21, still open. https://github.com/renode/renode/issues/478 ↩

3. Percepio, "Getting Started with TraceRecorder on Zephyr." https://percepio.com/getstarted/latest/html/zephyr.html ↩

4. Zephyr documentation, "Percepio TraceRecorder and Stream Ports" (stream port list stated as of July 2024). https://docs.zephyrproject.org/latest/services/tracing/index.html ↩

5. Zephyr documentation, "Tracing" — SEGGER SystemView support and Transport Backends. https://docs.zephyrproject.org/latest/services/tracing/index.html ↩ ↩²

6. SEGGER, "SystemView adds data acquisition via UART and TCP/IP," 2019-12-18. https://www.segger.com/news/seggers-systemview-adds-data-acquisition-via-uart-and-tcpip/ ↩

7. Zephyr v4.4.1, samples/subsys/tracing/prj_uart_ctf.conf. https://github.com/zephyrproject-rtos/zephyr/blob/v4.4.1/samples/subsys/tracing/prj_uart_ctf.conf ↩

8. Zephyr v4.4.1, samples/subsys/tracing/README.rst. https://github.com/zephyrproject-rtos/zephyr/blob/v4.4.1/samples/subsys/tracing/README.rst ↩

9. Zephyr v4.4.1, samples/subsys/tracing/src/main.c. https://github.com/zephyrproject-rtos/zephyr/blob/v4.4.1/samples/subsys/tracing/src/main.c ↩

10. Zephyr v4.4.1, subsys/tracing/tracing_core.c (worker thread named tracing_thread via k_thread_name_set). https://github.com/zephyrproject-rtos/zephyr/blob/v4.4.1/subsys/tracing/tracing_core.c ↩

11. Zephyr issue #49889, "ctf trace: unknown event id when parsing samples/tracing result on reel board," opened 2022-09-04. https://github.com/zephyrproject-rtos/zephyr/issues/49889 ↩

12. Renode issue #699, "Access semihosting UART from Rust," opened 2024-11-13. https://github.com/renode/renode/issues/699 ↩