Alex
9a10949938
The smoke test only exercised steady-state rendering and killed waybar with SIGTERM, so whole crash classes were invisible: exit-time use-after-free (#5182), module state transitions (#5183) and hardware-backed modules (#5179). Add tiers that hit those paths under ASan (+ _GLIBCXX_ASSERTIONS): - lib.sh: assert_clean_exit (SIGINT teardown -> checks segfault/abort + ASan report emitted during destruction), output hotplug helpers, signal/reload, WAYBAR_WRAP hook, opt-in leak detection. - lifecycle.sh (gating): clean exit, runtime output hotplug (Bar/module destroy), toggle/reload churn, fast-interval teardown race (#5182 class). - fuzz.sh (gating): pathological custom-backend output (empty, nonzero exit, invalid JSON, huge, non-UTF8, empty format). - coverage.sh (gating): every Factory module must be classified for smoke coverage; a new unlisted module fails the job (#5179 slipped through). - modules.sh: also render the whole matrix inside a group; SIGINT teardown per module instead of SIGTERM. - state.sh (best-effort): real mpd driven through an empty queue (#5183) + stop/clear; pulseaudio + slider on a null sink. - hardware.sh (best-effort): backlight, backlight/slider (#5179) and battery via umockdev. - leakcheck.sh (report-only): clean-exit run under LSan.
Smoke / render / interaction tests
The smoke workflow builds Waybar with
AddressSanitizer and runs the real binary inside a headless compositor
(software-rendered, no GPU) to exercise it end to end. All helpers share
lib.sh.
What it checks
| Tier | Check | Script |
|---|---|---|
| 0 | Launches, stays alive, and logs no ASan / Gtk-CRITICAL / crash — for every launch below |
lib.sh assert_clean |
| — | Coverage tripwire — every Factory module must be classified (source-only, runs before the build) |
coverage.sh |
| 1 | Renders — deterministic config screenshotted with grim, bar strip not blank |
run.sh |
| 1 | Per-module matrix — each headless-safe module rendered standalone and inside a group, each torn down via SIGINT | modules.sh |
| 2 | Golden — screenshot vs reference.png (fuzz 8%, 800px) |
workflow |
| 2 | Interaction — inject pointer clicks via sway; assert on-click side effect + format-alt toggle |
interact.sh |
| 2 | Accessibility — assert module labels via the AT-SPI tree (robust to fonts) | a11y.sh / a11y.py |
| 3 | Layout matrix — top / bottom / left (vertical) + HiDPI (scale 2) | positions.sh |
| 3 | Second compositor — same launch under labwc |
workflow (COMPOSITOR=labwc) |
| 4 | Lifecycle / teardown — clean SIGINT exit, runtime output hotplug, toggle + reload churn, fast-interval teardown (the C++ destructor path a kill never runs, #5182) |
lifecycle.sh |
| 4 | Custom-backend fuzz — empty / non-zero-exit / invalid-JSON / huge / non-UTF8 / empty-format exec output | fuzz.sh |
| 5 | State transitions (best-effort) — real mpd driven through an empty queue (#5183) + stop/clear, pulseaudio + slider on a null sink |
state.sh |
| 6 | Fake hardware (best-effort) — backlight, backlight/slider (#5179) and battery via umockdev |
hardware.sh |
| — | Leak check (report-only) — clean-exit run under LSan | leakcheck.sh |
AddressSanitizer (plus -D_GLIBCXX_ASSERTIONS) runs under all of the above, so
any memory bug in the real render/teardown path aborts the run. Tiers 5–6 and the
leak check are best-effort (continue-on-error): they depend on daemon setup /
mock devices that vary by runner. The interaction step is likewise best-effort — a
headless seat exposes no pointer capability, so injected clicks may be dropped
(hardening it needs a virtual pointer, uinput/wlr-virtual-pointer).
Because the teardown path (~Bar → window unmap → toggleSuspend) is where the
exit-time crashes live, tests assert on a clean SIGINT exit via
smoke::assert_clean_exit rather than killing with SIGTERM, and re-check the log
for sanitizer reports emitted during destruction.
Updating the reference image
reference.png must be generated by the CI environment (local rendering differs
from Ubuntu's). Two ways:
- One click: Actions → smoke → Run workflow with
bless = true. The job regenerates the screenshot and commits it to your branch. - Manual: download the
waybar-smokeartifact from a run and commit itsbar.pngastest/smoke/reference.png.
Do this whenever an intended visual change trips the golden comparison.
Run locally
On a machine with sway, grim, imagemagick, jq and Waybar built:
export WAYBAR_BIN=$PWD/build/waybar
./test/smoke/coverage.sh # no compositor needed
./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css /tmp/bar.png
./test/smoke/modules.sh /tmp/module-shots
./test/smoke/lifecycle.sh
./test/smoke/fuzz.sh
./test/smoke/interact.sh /tmp/interaction-shots
./test/smoke/positions.sh /tmp/layout-shots
dbus-run-session -- ./test/smoke/a11y.sh
# best-effort (need extra daemons / tools):
./test/smoke/state.sh # needs mpd + mpc, pulseaudio
./test/smoke/hardware.sh # needs umockdev
./test/smoke/leakcheck.sh