HIL Pipeline Setup Checklist

This checklist assumes you already have the pieces: target hardware on a bench, a CI system that runs jobs, and a test suite that passes somewhere. Nothing here is about choosing a framework or writing your first test.

It is about the gap between a rig that runs and a rig you can trust — one where red means the firmware is broken and green means it is not. Most new rigs fail that long before the firmware does.

Work through it once when the rig goes up, and again every time the DUT hardware revision changes. Cabling, power and recovery paths are what a board spin quietly invalidates.

1. The device under test

• DUT power is switched by the runner — a relay board, a programmable supply, or a USB hub with per-port control — so a hung device can be recovered without hands
• The reset line is driven by the runner too, so a soft recovery is tried before a full power cycle
• There is a recovery path that does not depend on the firmware running: a ROM bootloader, a strap that forces DFU, or JTAG
• A known-good image sits on the runner and can be flashed by a single job when a bad build leaves the DUT unresponsive
• The DUT hardware revision and serial number are read at the start of every run and printed into the test report

2. The runner

• The rig has its own runner, physically cabled to the DUT — not a shared cloud runner reaching hardware over someone's VPN
• The runner carries a tag (hil, hil-<board>) and rig jobs require it, so a hardware job can never land on a machine with no hardware attached
• Concurrency is one job per rig, enforced by the CI system — concurrent = 1 in the runner's config.toml, or a resource group on the job. Two jobs sharing one DUT produce results that mean nothing
• The runner comes back after a host reboot with nobody logging in: service enabled at boot, device permissions from a udev rule rather than a manual chmod
• The runner's configuration — tags, config.toml, udev rules, instrument addresses — is in version control, not only on the host's disk

3. Flashing

• Flashing is one scripted command end to end (probe-rs, openocd, pyocd, esptool, or the vendor CLI), with no IDE step and no button to press
• The flash is verified by reading the image back or querying the running firmware for its version — never inferred from a zero exit code — and that version is checked against the commit under test
• A failed flash fails the job loudly and immediately. It never falls through to testing whatever firmware happened to be on the device already
• A flash is retried at most once, with a power cycle in between, and the retry appears in the log rather than being swallowed

4. Stimulus and measurement

• Every instrument is driven programmatically — supply, DAQ, logic analyser, signal generator — over SCPI/VISA, a vendor Python API, or a documented serial protocol
• No instrument depends on a front-panel setting someone restores by hand; a rig whose state lives on a knob drifts the first time it is borrowed
• Instruments are in calibration, and the calibration date is recorded in the report next to the measurements it underwrites
• Measurement timestamps are aligned to the DUT's clock, by a shared trigger line or a sync marker the DUT emits at test start. A trace you cannot line up against the firmware's own log is worse than no trace

5. The test suite

• Every test begins by putting the DUT into a known state — reset, then wait for a boot banner or heartbeat — rather than inheriting what the previous test left behind
• The suite passes in a randomised order (pytest-randomly or equivalent), which is the most reliable practical check that no test depends on running after another
• Every bus read has a timeout — serial, CAN, TCP, debug probe — because a blocking read with none stalls the pipeline for the full job limit instead of failing in seconds
• Fixtures separate talking to the DUT from asserting on the result: one layer returns values, the tests judge them. Tests that parse serial bytes inline do not survive a transport change

6. Fault injection

A rig that only exercises the happy path confirms the happy path still works. That is worth something, but it is not what the field breaks on.

• The supply can brown out the DUT to a set voltage and drop it entirely, including mid-write to flash rather than only at idle
• Corrupt, truncated and out-of-order frames can be injected on the bus the product actually uses — CAN, RS-485, Ethernet — not just well-formed traffic
• A sensor can be disconnected, shorted, or made to return a stuck value while the test asserts what the firmware does about it
• The network link can be dropped mid-transfer, so an OTA update interrupted at the worst moment is a test case rather than a field report
• Each injected fault has a written expected behaviour — degrade, enter a safe state, recover — and the test asserts that, not merely that the DUT is still alive

7. Reporting

• The suite emits JUnit XML (pytest --junitxml=...) and CI ingests it, so failures render in the merge request instead of hiding in a job log
• Traces, captures, serial logs and instrument readings are kept as artefacts per run and linked from the failing test, so a failed voltage assertion arrives with the capture that produced it
• Every report records the commit SHA, the firmware version the DUT reported, the rig identifier and the DUT hardware revision — enough to reproduce the run
• Flaky tests go to a quarantine with a named owner and an expiry date. A skip with no expiry is a permanent hole in coverage that nobody is looking at

Where this usually goes wrong

Rig flakiness gets blamed on the firmware. An intermittent rig destroys trust in the suite faster than having no suite at all, because every red build becomes an argument instead of a bug. The cause is almost always power or cabling — a marginal USB hub, a long unshielded harness, a supply shared with a motor. Chase the physical layer before the test code.

Nothing recovers unattended. The rig works beautifully until the DUT hangs at two in the morning, and the pipeline stays dead until someone walks to the bench. Recovery has to be a job step, not a person.

The rig only one person can fix. The wiring is in one engineer's head and the instrument addresses are on a sticky note. You have built a single point of failure into the system you built to remove single points of failure.

When this needs outside help

Most teams can work through this list themselves, and should. It is worth bringing in help when the problem is not the checklist:

• The instrumentation is outside the team's experience — a bus interface, an RF chamber, a motor dyno — and nobody has driven that class of equipment programmatically before
• The rig's output is a required certification artefact (DO-178C, IEC 61508, EN 50128), so the rig itself needs qualification and its own evidence, not just green pipelines
• The DUT has no debug or recovery path designed in: no external reset, no bootloader entry that survives bad firmware, no accessible test points. The fix is a hardware change, and no amount of test engineering substitutes for one

If any of those describe where you are, this is work we do: contact@mnpsystems.ca.