ci(smoke): sanitizer build + module/interaction/a11y/layout coverage
Turn the smoke test into a real runtime safety net. Waybar is now built with AddressSanitizer and exercised end to end in a headless compositor: - Tier 0: run everything under ASan; fail on ASan reports and Gtk/GLib criticals in the log (lib.sh assert_clean) - Tier 1: per-module render matrix (modules.sh) — each headless-safe module rendered in isolation - Tier 2: pointer interaction (interact.sh) — inject clicks via sway, assert on-click side effect and format-alt toggle; AT-SPI assertions (a11y.sh/.py) check module labels semantically instead of by pixels - Tier 3: layout matrix (positions.sh) — top/bottom/left + HiDPI scale 2; second compositor run under labwc Shared helpers extracted to lib.sh. Adds a workflow_dispatch `bless` input to regenerate the golden reference in one click. Trigger push only on master to avoid duplicate PR runs. AT-SPI and labwc steps are continue-on-error.
This commit is contained in:
Alex
+69
-28
@@ -1,15 +1,31 @@
|
||||
name: smoke
|
||||
|
||||
# Launch the real Waybar binary inside a headless sway compositor and verify it
|
||||
# starts, renders, and matches a reference screenshot. Complements the unit
|
||||
# Launch the real Waybar binary (built with AddressSanitizer) inside a headless
|
||||
# compositor and exercise it: render, per-module matrix, pointer interaction,
|
||||
# accessibility-tree assertions, and a layout matrix. Complements the unit
|
||||
# tests and build jobs, which never actually run the bar.
|
||||
|
||||
on: [push, pull_request]
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
pull_request:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
bless:
|
||||
description: "Regenerate and commit the golden reference screenshot"
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
WAYBAR_BIN: ${{ github.workspace }}/build/waybar
|
||||
|
||||
jobs:
|
||||
smoke:
|
||||
runs-on: ubuntu-latest
|
||||
@@ -20,7 +36,8 @@ jobs:
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install --no-install-recommends -y \
|
||||
sway grim imagemagick fonts-dejavu-core \
|
||||
sway labwc grim imagemagick fonts-dejavu-core jq \
|
||||
at-spi2-core python3-pyatspi dbus-x11 \
|
||||
meson ninja-build g++ pkg-config scdoc wayland-protocols \
|
||||
gobject-introspection libdbusmenu-gtk3-dev libevdev-dev libfmt-dev \
|
||||
libgirepository1.0-dev libgtk-3-dev libgtkmm-3.0-dev libinput-dev \
|
||||
@@ -28,53 +45,77 @@ jobs:
|
||||
libpulse-dev libsigc++-2.0-dev libspdlog-dev libwayland-dev \
|
||||
upower libxkbregistry-dev libxkbcommon-dev
|
||||
|
||||
- name: Build & install Waybar
|
||||
- name: Build Waybar with AddressSanitizer
|
||||
run: |
|
||||
meson setup build -Dman-pages=disabled
|
||||
meson setup build -Dman-pages=disabled -Db_sanitize=address
|
||||
ninja -C build
|
||||
sudo ninja -C build install
|
||||
|
||||
# --- Level 1: smoke with real modules ---
|
||||
- name: Smoke — real modules load without crashing
|
||||
run: ./test/smoke/run.sh test/smoke/modules-config.jsonc
|
||||
|
||||
# --- Level 1 + 2: launch deterministic config and screenshot ---
|
||||
- name: Launch + screenshot (deterministic config)
|
||||
# --- Tier 0 + 2: render + sanitizer/critical gate ---
|
||||
- name: Render deterministic config
|
||||
run: ./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css bar.png
|
||||
|
||||
# --- Level 2: the bar actually rendered something ---
|
||||
- name: Check the bar is not blank
|
||||
run: |
|
||||
sd=$(convert bar.png -crop 1920x30+0+0 +repage -colorspace Gray \
|
||||
-format '%[fx:standard_deviation]' info:)
|
||||
echo "bar strip stddev = $sd"
|
||||
awk "BEGIN{ exit !($sd > 0.01) }" || {
|
||||
echo "::error::the top bar strip looks blank — Waybar rendered nothing"
|
||||
exit 1
|
||||
}
|
||||
awk "BEGIN{ exit !($sd > 0.01) }" || { echo "::error::bar looks blank"; exit 1; }
|
||||
|
||||
# --- Level 3: golden comparison against the committed reference ---
|
||||
- name: Compare against reference screenshot
|
||||
# --- Manual re-bless of the reference image ---
|
||||
- name: Bless reference screenshot
|
||||
if: github.event_name == 'workflow_dispatch' && inputs.bless
|
||||
run: |
|
||||
cp bar.png test/smoke/reference.png
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git add test/smoke/reference.png
|
||||
git commit -m "ci(smoke): re-bless reference screenshot [skip ci]"
|
||||
git push
|
||||
|
||||
- name: Golden comparison
|
||||
run: |
|
||||
if [ -f test/smoke/reference.png ]; then
|
||||
ae=$(compare -metric AE -fuzz 8% test/smoke/reference.png bar.png diff.png 2>&1 || true)
|
||||
ae=$(printf '%s' "$ae" | grep -oE '^[0-9]+' || echo 0)
|
||||
echo "differing pixels (fuzz 8%): $ae"
|
||||
if [ "$ae" -gt 800 ]; then
|
||||
echo "::error::visual regression: $ae pixels differ from the reference (threshold 800). See the 'diff.png' artifact."
|
||||
exit 1
|
||||
fi
|
||||
echo "✓ within tolerance"
|
||||
[ "$ae" -gt 800 ] && { echo "::error::visual regression: $ae px (see diff.png artifact)"; exit 1; } || echo "✓ within tolerance"
|
||||
else
|
||||
echo "::warning::No reference image yet. Download the 'waybar-screenshot' artifact from this run and commit bar.png as test/smoke/reference.png to enable golden comparison."
|
||||
echo "::warning::No reference image; run this workflow via 'Run workflow' with bless=true to create it."
|
||||
fi
|
||||
|
||||
- name: Upload screenshots
|
||||
# --- Tier 1: per-module matrix ---
|
||||
- name: Per-module render matrix
|
||||
run: ./test/smoke/modules.sh module-shots
|
||||
|
||||
# --- Tier 2: pointer interaction (best-effort) ---
|
||||
# Headless seats may not expose a pointer capability, so injected clicks
|
||||
# can be dropped; keep this non-blocking while it is hardened.
|
||||
- name: Interaction (click + format-alt)
|
||||
continue-on-error: true
|
||||
run: ./test/smoke/interact.sh interaction-shots
|
||||
|
||||
# --- Tier 2: accessibility assertions ---
|
||||
- name: Accessibility (AT-SPI) assertions
|
||||
run: dbus-run-session -- ./test/smoke/a11y.sh
|
||||
|
||||
# --- Tier 3: layout matrix (positions + HiDPI) ---
|
||||
- name: Layout matrix (top/bottom/left + scale 2)
|
||||
run: ./test/smoke/positions.sh layout-shots
|
||||
|
||||
# --- Tier 3: second compositor ---
|
||||
- name: Launch under labwc
|
||||
run: COMPOSITOR=labwc ./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css labwc.png
|
||||
|
||||
- name: Upload screenshots & diffs
|
||||
uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: waybar-screenshot
|
||||
name: waybar-smoke
|
||||
path: |
|
||||
bar.png
|
||||
diff.png
|
||||
labwc.png
|
||||
module-shots/
|
||||
interaction-shots/
|
||||
layout-shots/
|
||||
if-no-files-found: warn
|
||||
|
||||
Reference in New Issue
Block a user