Alex
74f0d3393a
The first run went green but the best-effort tiers silently didn't exercise
anything:
- state.sh: the mpd `audio_output { ... }` block was on one line, which mpd
rejects ("Unknown tokens after '{'"), so mpd never started and the mpd
sub-tier (the #5183 empty-queue repro) skipped. Split the block across lines.
Also move teardown into a cleanup() with `|| true` so a dead-pid kill under
`set -e` no longer makes the step exit 1.
- hardware.sh: umockdev-run's LD_PRELOAD lands ahead of the linked-in ASan
runtime, so ASan aborted before main() ("ASan runtime does not come first")
and waybar never started -- the backlight/slider (#5179) path wasn't tested.
Set verify_asan_link_order=0 so the instrumented binary runs under the preload.
- leakcheck.sh: abort_on_error=1 turned LSan's exit report into a core dump;
force abort_on_error=0 for this report-only tier.
- lib.sh: assert_clean now also flags "ASan runtime does not come first" so a
future preload/link-order regression fails loudly instead of passing.
Smoke / render / interaction tests
The smoke workflow builds Waybar with
AddressSanitizer and runs the real binary inside a headless compositor
(software-rendered, no GPU) to exercise it end to end. All helpers share
lib.sh.
What it checks
| Tier | Check | Script |
|---|---|---|
| 0 | Launches, stays alive, and logs no ASan / Gtk-CRITICAL / crash — for every launch below |
lib.sh assert_clean |
| — | Coverage tripwire — every Factory module must be classified (source-only, runs before the build) |
coverage.sh |
| 1 | Renders — deterministic config screenshotted with grim, bar strip not blank |
run.sh |
| 1 | Per-module matrix — each headless-safe module rendered standalone and inside a group, each torn down via SIGINT | modules.sh |
| 2 | Golden — screenshot vs reference.png (fuzz 8%, 800px) |
workflow |
| 2 | Interaction — inject pointer clicks via sway; assert on-click side effect + format-alt toggle |
interact.sh |
| 2 | Accessibility — assert module labels via the AT-SPI tree (robust to fonts) | a11y.sh / a11y.py |
| 3 | Layout matrix — top / bottom / left (vertical) + HiDPI (scale 2) | positions.sh |
| 3 | Second compositor — same launch under labwc |
workflow (COMPOSITOR=labwc) |
| 4 | Lifecycle / teardown — clean SIGINT exit, runtime output hotplug, toggle + reload churn, fast-interval teardown (the C++ destructor path a kill never runs, #5182) |
lifecycle.sh |
| 4 | Custom-backend fuzz — empty / non-zero-exit / invalid-JSON / huge / non-UTF8 / empty-format exec output | fuzz.sh |
| 5 | State transitions (best-effort) — real mpd driven through an empty queue (#5183) + stop/clear, pulseaudio + slider on a null sink |
state.sh |
| 6 | Fake hardware (best-effort) — backlight, backlight/slider (#5179) and battery via umockdev |
hardware.sh |
| — | Leak check (report-only) — clean-exit run under LSan | leakcheck.sh |
AddressSanitizer (plus -D_GLIBCXX_ASSERTIONS) runs under all of the above, so
any memory bug in the real render/teardown path aborts the run. Tiers 5–6 and the
leak check are best-effort (continue-on-error): they depend on daemon setup /
mock devices that vary by runner. The interaction step is likewise best-effort — a
headless seat exposes no pointer capability, so injected clicks may be dropped
(hardening it needs a virtual pointer, uinput/wlr-virtual-pointer).
Because the teardown path (~Bar → window unmap → toggleSuspend) is where the
exit-time crashes live, tests assert on a clean SIGINT exit via
smoke::assert_clean_exit rather than killing with SIGTERM, and re-check the log
for sanitizer reports emitted during destruction.
Updating the reference image
reference.png must be generated by the CI environment (local rendering differs
from Ubuntu's). Two ways:
- One click: Actions → smoke → Run workflow with
bless = true. The job regenerates the screenshot and commits it to your branch. - Manual: download the
waybar-smokeartifact from a run and commit itsbar.pngastest/smoke/reference.png.
Do this whenever an intended visual change trips the golden comparison.
Run locally
On a machine with sway, grim, imagemagick, jq and Waybar built:
export WAYBAR_BIN=$PWD/build/waybar
./test/smoke/coverage.sh # no compositor needed
./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css /tmp/bar.png
./test/smoke/modules.sh /tmp/module-shots
./test/smoke/lifecycle.sh
./test/smoke/fuzz.sh
./test/smoke/interact.sh /tmp/interaction-shots
./test/smoke/positions.sh /tmp/layout-shots
dbus-run-session -- ./test/smoke/a11y.sh
# best-effort (need extra daemons / tools):
./test/smoke/state.sh # needs mpd + mpc, pulseaudio
./test/smoke/hardware.sh # needs umockdev
./test/smoke/leakcheck.sh