Booting every board, every night

Key points

Armbian describes its automated test facility: a rack of real single-board computers that installs each night's published packages, switches kernels, reboots and measures, to catch changes that build cleanly but leave a board unbootable. The post also sets out the next goal, running these hardware tests on pull requests before they merge, and what is missing to get there.

The fleet

• 65 boards under test: 62 distinct models from 22 vendors across 20 SoC families, including Rockchip, Amlogic, Allwinner, NXP, Marvell, Broadcom, TI, Samsung, Qualcomm, SpacemiT and x86.
• When the article was written, 56 boards were active and 9 were in a failed state.
• Every board is registered in NetBox, which the orchestrator reads to decide what each board can be tested with. Power control is derived from the cabling model; a board without managed power skips the tests that need a hard power cut.

What each bench needs

• Remote power (required): PoE switch ports, a switched rack PDU, or a 24-channel relay board driven over GPIO.
• Wired Ethernet on a managed switch (required): control over SSH, throughput measurement and, on PoE benches, power.
• Power metering (strongly recommended): shows whether a board really rebooted or the SoC never stopped.
• Serial console (recommended): several boards lack one, which stalls investigations of boot hangs.
• Clean-flash path (not yet wired on any bench): an SD-card switcher or USB vendor recovery. This is the biggest constraint on what comes next.
• Shared infrastructure: three core and four access switches, a rack PDU, DC and USB supplies, a UPS, console hosts with a KVM device, and six servers (two Ampere-class ARM) running CI.

The nightly cycle

• Power on, scan and reconcile the inventory, sync maintainers' SSH keys to the boards they own, run the board pipeline in parallel, scan again, and always power off, even after a failure.
• Per board: upgrade to the nightly repository, reboot, then for each configured kernel branch (current, edge and so on) switch kernel, do warm reboots and a cold power cycle, test CPU, memory, disk and temperature, check that the governor reaches its claimed frequencies (dvfs), measure throughput with iperf on each cabled interface and record installed versions. The board is then restored to stable.
• A full board pipeline takes about half an hour.
• Results are public at docs.armbian.com/status/board-tests.

What it has caught

• One board fails every reboot on both kernel branches; the power trace shows the SoC never stopped, which points at firmware. It is unresolved, and that bench has no serial console.
• The frequency-scaling check expected 95% of the advertised maximum, which is wrong for Intel and AMD drivers. Every x86 board failed until the test was fixed to detect the driver.
• Self-hosted runners sometimes drop mid-job. The retry logic re-runs only jobs whose test step did not fail, so real failures stay red.

AI in the tooling

• Much of the test modules, hardware control scripts, inventory reconciliation and documentation is now written and maintained with AI assistance.
• The post gives an example of a confident but wrong analysis based on a partial sample. The working rule: AI may propose anything, but hardware evidence is the gate.

Next: validation at pull-request time

• Today only nightly builds are validated, after the change is already published. The goal is to boot the affected board family before a PR merges.
• Open problems: mapping a diff to the right boards; publishing PR builds to an isolated repository, which needs more build capacity and storage; a short PR profile, since thirty minutes is too slow for a merge gate; remote recovery of boards bricked by a bad kernel; and keeping bench failures from blocking merges.
• PR testing will start on benches that can be reflashed remotely; the others stay on nightly testing.

How to help

• Bench infrastructure: SD-card switchers, managed PoE switches with 2.5 GbE ports and 10 GbE uplinks, and PoE splitters.
• Build capacity: hosting a build server; requirements are documented.
• Hardware partnerships: vendors can place devices in the permanent test fleet under support and maintenance agreements.

Summary by DistroNews, based on the original post.