ci(smoke): sanitizer build + module/interaction/a11y/layout coverage

Turn the smoke test into a real runtime safety net. Waybar is now built with
AddressSanitizer and exercised end to end in a headless compositor:

- Tier 0: run everything under ASan; fail on ASan reports and Gtk/GLib
  criticals in the log (lib.sh assert_clean)
- Tier 1: per-module render matrix (modules.sh) — each headless-safe module
  rendered in isolation
- Tier 2: pointer interaction (interact.sh) — inject clicks via sway, assert
  on-click side effect and format-alt toggle; AT-SPI assertions (a11y.sh/.py)
  check module labels semantically instead of by pixels
- Tier 3: layout matrix (positions.sh) — top/bottom/left + HiDPI scale 2;
  second compositor run under labwc

Shared helpers extracted to lib.sh. Adds a workflow_dispatch `bless` input to
regenerate the golden reference in one click. Trigger push only on master to
avoid duplicate PR runs. AT-SPI and labwc steps are continue-on-error.
This commit is contained in:
Alex
2026-07-05 10:25:49 +02:00
parent 57c1ac789f
commit eb10a511f0
10 changed files with 524 additions and 135 deletions
+35 -20
View File
@@ -1,34 +1,49 @@
# Smoke / screenshot test
# Smoke / render / interaction tests
The [`smoke`](../../.github/workflows/smoke.yml) workflow launches the real
Waybar binary inside a **headless sway** compositor (software-rendered, no GPU)
and checks three things:
The [`smoke`](../../.github/workflows/smoke.yml) workflow builds Waybar **with
AddressSanitizer** and runs the real binary inside a **headless** compositor
(software-rendered, no GPU) to exercise it end to end. All helpers share
[`lib.sh`](lib.sh).
1. **It starts and stays alive** — with real modules (`clock`, `cpu`, `memory`,
`disk`) loaded, Waybar must not crash or log a fatal error.
([`modules-config.jsonc`](modules-config.jsonc))
2. **It renders** — a deterministic config ([`config.jsonc`](config.jsonc) +
[`style.css`](style.css)) is captured with `grim`; the bar strip must not be
blank.
3. **It looks right** — the screenshot is compared to `reference.png` with an 8%
fuzz and an 800-pixel tolerance. Regressions fail the job, and the screenshot
and diff are uploaded as artifacts.
## What it checks
| Tier | Check | Script |
| --- | --- | --- |
| 0 | Launches, stays alive, and logs **no ASan / `Gtk-CRITICAL` / crash** — for every launch below | [`lib.sh`](lib.sh) `assert_clean` |
| 1 | **Renders** — deterministic config screenshotted with `grim`, bar strip not blank | [`run.sh`](run.sh) |
| 1 | **Per-module matrix** — each headless-safe module rendered in isolation | [`modules.sh`](modules.sh) |
| 2 | **Golden** — screenshot vs [`reference.png`](reference.png) (fuzz 8%, 800px) | workflow |
| 2 | **Interaction** — inject pointer clicks via sway; assert `on-click` side effect + `format-alt` toggle | [`interact.sh`](interact.sh) |
| 2 | **Accessibility** — assert module labels via the AT-SPI tree (robust to fonts) | [`a11y.sh`](a11y.sh) / [`a11y.py`](a11y.py) |
| 3 | **Layout matrix** — top / bottom / left (vertical) + HiDPI (scale 2) | [`positions.sh`](positions.sh) |
| 3 | **Second compositor** — same launch under `labwc` | workflow (`COMPOSITOR=labwc`) |
AddressSanitizer runs under *all* of the above, so any memory bug in the real
render path aborts the run. The interaction step is best-effort
(`continue-on-error`): a headless seat exposes no pointer capability, so injected
clicks may be dropped — hardening it needs a virtual pointer (uinput/wlr-virtual-pointer).
## Updating the reference image
`reference.png` must be generated by the CI environment (local rendering differs
from Ubuntu's). To (re)bless it:
from Ubuntu's). Two ways:
1. Push your change and open the run of the `smoke` workflow.
2. Download the `waybar-screenshot` artifact.
3. Commit its `bar.png` as `test/smoke/reference.png`.
- **One click:** Actions → *smoke***Run workflow** with `bless = true`. The
job regenerates the screenshot and commits it to your branch.
- **Manual:** download the `waybar-smoke` artifact from a run and commit its
`bar.png` as `test/smoke/reference.png`.
Do this whenever an **intended** visual change makes the golden comparison fail.
Do this whenever an **intended** visual change trips the golden comparison.
## Run locally
On a machine with `sway`, `grim` and `waybar`:
On a machine with `sway`, `grim`, `imagemagick` and Waybar built:
```sh
./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css /tmp/bar.png
export WAYBAR_BIN=$PWD/build/waybar
./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css /tmp/bar.png
./test/smoke/modules.sh /tmp/module-shots
./test/smoke/interact.sh /tmp/interaction-shots
./test/smoke/positions.sh /tmp/layout-shots
dbus-run-session -- ./test/smoke/a11y.sh
```