Alex
9a10949938
The smoke test only exercised steady-state rendering and killed waybar with SIGTERM, so whole crash classes were invisible: exit-time use-after-free (#5182), module state transitions (#5183) and hardware-backed modules (#5179). Add tiers that hit those paths under ASan (+ _GLIBCXX_ASSERTIONS): - lib.sh: assert_clean_exit (SIGINT teardown -> checks segfault/abort + ASan report emitted during destruction), output hotplug helpers, signal/reload, WAYBAR_WRAP hook, opt-in leak detection. - lifecycle.sh (gating): clean exit, runtime output hotplug (Bar/module destroy), toggle/reload churn, fast-interval teardown race (#5182 class). - fuzz.sh (gating): pathological custom-backend output (empty, nonzero exit, invalid JSON, huge, non-UTF8, empty format). - coverage.sh (gating): every Factory module must be classified for smoke coverage; a new unlisted module fails the job (#5179 slipped through). - modules.sh: also render the whole matrix inside a group; SIGINT teardown per module instead of SIGTERM. - state.sh (best-effort): real mpd driven through an empty queue (#5183) + stop/clear; pulseaudio + slider on a null sink. - hardware.sh (best-effort): backlight, backlight/slider (#5179) and battery via umockdev. - leakcheck.sh (report-only): clean-exit run under LSan.
70 lines
4.2 KiB
Markdown
70 lines
4.2 KiB
Markdown
# Smoke / render / interaction tests
|
||
|
||
The [`smoke`](../../.github/workflows/smoke.yml) workflow builds Waybar **with
|
||
AddressSanitizer** and runs the real binary inside a **headless** compositor
|
||
(software-rendered, no GPU) to exercise it end to end. All helpers share
|
||
[`lib.sh`](lib.sh).
|
||
|
||
## What it checks
|
||
|
||
| Tier | Check | Script |
|
||
| --- | --- | --- |
|
||
| 0 | Launches, stays alive, and logs **no ASan / `Gtk-CRITICAL` / crash** — for every launch below | [`lib.sh`](lib.sh) `assert_clean` |
|
||
| — | **Coverage tripwire** — every `Factory` module must be classified (source-only, runs before the build) | [`coverage.sh`](coverage.sh) |
|
||
| 1 | **Renders** — deterministic config screenshotted with `grim`, bar strip not blank | [`run.sh`](run.sh) |
|
||
| 1 | **Per-module matrix** — each headless-safe module rendered standalone **and inside a group**, each torn down via SIGINT | [`modules.sh`](modules.sh) |
|
||
| 2 | **Golden** — screenshot vs [`reference.png`](reference.png) (fuzz 8%, 800px) | workflow |
|
||
| 2 | **Interaction** — inject pointer clicks via sway; assert `on-click` side effect + `format-alt` toggle | [`interact.sh`](interact.sh) |
|
||
| 2 | **Accessibility** — assert module labels via the AT-SPI tree (robust to fonts) | [`a11y.sh`](a11y.sh) / [`a11y.py`](a11y.py) |
|
||
| 3 | **Layout matrix** — top / bottom / left (vertical) + HiDPI (scale 2) | [`positions.sh`](positions.sh) |
|
||
| 3 | **Second compositor** — same launch under `labwc` | workflow (`COMPOSITOR=labwc`) |
|
||
| 4 | **Lifecycle / teardown** — clean SIGINT exit, runtime **output hotplug**, toggle + reload churn, fast-interval teardown (the C++ destructor path a `kill` never runs, #5182) | [`lifecycle.sh`](lifecycle.sh) |
|
||
| 4 | **Custom-backend fuzz** — empty / non-zero-exit / invalid-JSON / huge / non-UTF8 / empty-format exec output | [`fuzz.sh`](fuzz.sh) |
|
||
| 5 | **State transitions** *(best-effort)* — real `mpd` driven through an **empty queue** (#5183) + stop/clear, `pulseaudio` + slider on a null sink | [`state.sh`](state.sh) |
|
||
| 6 | **Fake hardware** *(best-effort)* — `backlight`, `backlight/slider` (#5179) and `battery` via **umockdev** | [`hardware.sh`](hardware.sh) |
|
||
| — | **Leak check** *(report-only)* — clean-exit run under LSan | [`leakcheck.sh`](leakcheck.sh) |
|
||
|
||
AddressSanitizer (plus `-D_GLIBCXX_ASSERTIONS`) runs under *all* of the above, so
|
||
any memory bug in the real render/teardown path aborts the run. Tiers 5–6 and the
|
||
leak check are best-effort (`continue-on-error`): they depend on daemon setup /
|
||
mock devices that vary by runner. The interaction step is likewise best-effort — a
|
||
headless seat exposes no pointer capability, so injected clicks may be dropped
|
||
(hardening it needs a virtual pointer, uinput/wlr-virtual-pointer).
|
||
|
||
Because the teardown path (`~Bar` → window unmap → `toggleSuspend`) is where the
|
||
exit-time crashes live, tests assert on a **clean SIGINT exit** via
|
||
`smoke::assert_clean_exit` rather than killing with SIGTERM, and re-check the log
|
||
for sanitizer reports emitted *during* destruction.
|
||
|
||
## Updating the reference image
|
||
|
||
`reference.png` must be generated by the CI environment (local rendering differs
|
||
from Ubuntu's). Two ways:
|
||
|
||
- **One click:** Actions → *smoke* → **Run workflow** with `bless = true`. The
|
||
job regenerates the screenshot and commits it to your branch.
|
||
- **Manual:** download the `waybar-smoke` artifact from a run and commit its
|
||
`bar.png` as `test/smoke/reference.png`.
|
||
|
||
Do this whenever an **intended** visual change trips the golden comparison.
|
||
|
||
## Run locally
|
||
|
||
On a machine with `sway`, `grim`, `imagemagick`, `jq` and Waybar built:
|
||
|
||
```sh
|
||
export WAYBAR_BIN=$PWD/build/waybar
|
||
./test/smoke/coverage.sh # no compositor needed
|
||
./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css /tmp/bar.png
|
||
./test/smoke/modules.sh /tmp/module-shots
|
||
./test/smoke/lifecycle.sh
|
||
./test/smoke/fuzz.sh
|
||
./test/smoke/interact.sh /tmp/interaction-shots
|
||
./test/smoke/positions.sh /tmp/layout-shots
|
||
dbus-run-session -- ./test/smoke/a11y.sh
|
||
# best-effort (need extra daemons / tools):
|
||
./test/smoke/state.sh # needs mpd + mpc, pulseaudio
|
||
./test/smoke/hardware.sh # needs umockdev
|
||
./test/smoke/leakcheck.sh
|
||
```
|