From cf2f120ad77ca81e4ad145ac457bfb947044b6ac Mon Sep 17 00:00:00 2001 From: Alex Date: Sat, 4 Jul 2026 13:15:05 +0200 Subject: [PATCH 1/3] ci: add headless smoke + screenshot test for the bar MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Launch the real Waybar binary inside a headless, software-rendered sway compositor and verify it actually runs and renders — something the unit tests and build jobs never do. - test/smoke/run.sh: boots sway (WLR_BACKENDS=headless, pixman), starts waybar, asserts it stays alive with no fatal log, optionally grabs a screenshot with grim - level 1: real modules (clock/cpu/memory/disk) load without crashing - level 2: deterministic config is screenshotted and checked to be non-blank - level 3: screenshot compared to test/smoke/reference.png (fuzz 8%, 800px tolerance); screenshot + diff uploaded as artifacts - .github/workflows/smoke.yml runs it on push/PR The reference image must be blessed from a CI artifact (see test/smoke/README.md). Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/smoke.yml | 80 +++++++++++++++++++++++++++++++++ test/smoke/README.md | 34 ++++++++++++++ test/smoke/config.jsonc | 26 +++++++++++ test/smoke/modules-config.jsonc | 19 ++++++++ test/smoke/run.sh | 77 +++++++++++++++++++++++++++++++ test/smoke/style.css | 27 +++++++++++ 6 files changed, 263 insertions(+) create mode 100644 .github/workflows/smoke.yml create mode 100644 test/smoke/README.md create mode 100644 test/smoke/config.jsonc create mode 100644 test/smoke/modules-config.jsonc create mode 100755 test/smoke/run.sh create mode 100644 test/smoke/style.css diff --git a/.github/workflows/smoke.yml b/.github/workflows/smoke.yml new file mode 100644 index 00000000..439d8aae --- /dev/null +++ b/.github/workflows/smoke.yml @@ -0,0 +1,80 @@ +name: smoke + +# Launch the real Waybar binary inside a headless sway compositor and verify it +# starts, renders, and matches a reference screenshot. Complements the unit +# tests and build jobs, which never actually run the bar. + +on: [push, pull_request] + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +jobs: + smoke: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install --no-install-recommends -y \ + sway grim imagemagick fonts-dejavu-core \ + meson ninja-build g++ pkg-config scdoc wayland-protocols \ + gobject-introspection libdbusmenu-gtk3-dev libevdev-dev libfmt-dev \ + libgirepository1.0-dev libgtk-3-dev libgtkmm-3.0-dev libinput-dev \ + libjsoncpp-dev libmpdclient-dev libnl-3-dev libnl-genl-3-dev \ + libpulse-dev libsigc++-2.0-dev libspdlog-dev libwayland-dev \ + upower libxkbregistry-dev libxkbcommon-dev + + - name: Build & install Waybar + run: | + meson setup build -Dman-pages=disabled + ninja -C build + sudo ninja -C build install + + # --- Level 1: smoke with real modules --- + - name: Smoke — real modules load without crashing + run: ./test/smoke/run.sh test/smoke/modules-config.jsonc + + # --- Level 1 + 2: launch deterministic config and screenshot --- + - name: Launch + screenshot (deterministic config) + run: ./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css bar.png + + # --- Level 2: the bar actually rendered something --- + - name: Check the bar is not blank + run: | + sd=$(convert bar.png -crop 1920x30+0+0 +repage -colorspace Gray \ + -format '%[fx:standard_deviation]' info:) + echo "bar strip stddev = $sd" + awk "BEGIN{ exit !($sd > 0.01) }" || { + echo "::error::the top bar strip looks blank — Waybar rendered nothing" + exit 1 + } + + # --- Level 3: golden comparison against the committed reference --- + - name: Compare against reference screenshot + run: | + if [ -f test/smoke/reference.png ]; then + ae=$(compare -metric AE -fuzz 8% test/smoke/reference.png bar.png diff.png 2>&1 || true) + ae=$(printf '%s' "$ae" | grep -oE '^[0-9]+' || echo 0) + echo "differing pixels (fuzz 8%): $ae" + if [ "$ae" -gt 800 ]; then + echo "::error::visual regression: $ae pixels differ from the reference (threshold 800). See the 'diff.png' artifact." + exit 1 + fi + echo "✓ within tolerance" + else + echo "::warning::No reference image yet. Download the 'waybar-screenshot' artifact from this run and commit bar.png as test/smoke/reference.png to enable golden comparison." + fi + + - name: Upload screenshots + uses: actions/upload-artifact@v4 + if: always() + with: + name: waybar-screenshot + path: | + bar.png + diff.png + if-no-files-found: warn diff --git a/test/smoke/README.md b/test/smoke/README.md new file mode 100644 index 00000000..f181e793 --- /dev/null +++ b/test/smoke/README.md @@ -0,0 +1,34 @@ +# Smoke / screenshot test + +The [`smoke`](../../.github/workflows/smoke.yml) workflow launches the real +Waybar binary inside a **headless sway** compositor (software-rendered, no GPU) +and checks three things: + +1. **It starts and stays alive** — with real modules (`clock`, `cpu`, `memory`, + `disk`) loaded, Waybar must not crash or log a fatal error. + ([`modules-config.jsonc`](modules-config.jsonc)) +2. **It renders** — a deterministic config ([`config.jsonc`](config.jsonc) + + [`style.css`](style.css)) is captured with `grim`; the bar strip must not be + blank. +3. **It looks right** — the screenshot is compared to `reference.png` with an 8% + fuzz and an 800-pixel tolerance. Regressions fail the job, and the screenshot + and diff are uploaded as artifacts. + +## Updating the reference image + +`reference.png` must be generated by the CI environment (local rendering differs +from Ubuntu's). To (re)bless it: + +1. Push your change and open the run of the `smoke` workflow. +2. Download the `waybar-screenshot` artifact. +3. Commit its `bar.png` as `test/smoke/reference.png`. + +Do this whenever an **intended** visual change makes the golden comparison fail. + +## Run locally + +On a machine with `sway`, `grim` and `waybar`: + +```sh +./test/smoke/run.sh test/smoke/config.jsonc test/smoke/style.css /tmp/bar.png +``` diff --git a/test/smoke/config.jsonc b/test/smoke/config.jsonc new file mode 100644 index 00000000..433b7007 --- /dev/null +++ b/test/smoke/config.jsonc @@ -0,0 +1,26 @@ +// Deterministic Waybar config for the CI smoke/screenshot test. +// Uses only static custom modules (interval "once") so the rendered bar is +// reproducible and can be compared against a reference image. +{ + "layer": "top", + "position": "top", + "height": 30, + "modules-left": ["custom/left"], + "modules-center": ["custom/center"], + "modules-right": ["custom/right"], + "custom/left": { + "format": "Waybar", + "exec": "echo", + "interval": "once" + }, + "custom/center": { + "format": "CI smoke test", + "exec": "echo", + "interval": "once" + }, + "custom/right": { + "format": "12:34", + "exec": "echo", + "interval": "once" + } +} diff --git a/test/smoke/modules-config.jsonc b/test/smoke/modules-config.jsonc new file mode 100644 index 00000000..d6a7b9a8 --- /dev/null +++ b/test/smoke/modules-config.jsonc @@ -0,0 +1,19 @@ +// Waybar config for the CI smoke test: exercises real modules that work in a +// headless environment (no hardware/daemon dependency). Its content is dynamic +// (time, cpu, ...) so it is NOT screenshot-compared — it only checks that these +// modules load and render without crashing the bar. +{ + "layer": "top", + "position": "top", + "height": 30, + "modules-center": ["clock", "cpu", "memory", "disk", "custom/hello"], + "clock": {}, + "cpu": {}, + "memory": {}, + "disk": {}, + "custom/hello": { + "format": "ok", + "exec": "echo", + "interval": "once" + } +} diff --git a/test/smoke/run.sh b/test/smoke/run.sh new file mode 100755 index 00000000..0b26790d --- /dev/null +++ b/test/smoke/run.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# Launch Waybar inside a headless sway compositor and verify it starts and +# renders. Used by the `smoke` CI workflow. +# +# Usage: run.sh [style] [screenshot.png] +# - always: assert waybar launches, stays alive, logs no fatal error +# - if given: capture the bar with grim +# +# Requires: sway, grim, waybar (on PATH). Software-rendered, no GPU needed. +set -euo pipefail + +CONFIG="$1" +STYLE="${2:-}" +SHOT="${3:-}" + +export XDG_RUNTIME_DIR="$(mktemp -d)" +chmod 700 "$XDG_RUNTIME_DIR" +export WLR_BACKENDS=headless +export WLR_RENDERER=pixman +export WLR_LIBINPUT_NO_DEVICES=1 +export LIBGL_ALWAYS_SOFTWARE=1 + +LOG="$(mktemp)" +SWAYCFG="$(mktemp)" + +wb="waybar -c $CONFIG" +[ -n "$STYLE" ] && wb="$wb -s $STYLE" + +cat > "$SWAYCFG" < $LOG 2>&1" +EOF + +echo "::group::Starting sway (headless) + waybar" +sway -c "$SWAYCFG" & +SWAY_PID=$! + +cleanup() { + swaymsg -q exit 2>/dev/null || kill "$SWAY_PID" 2>/dev/null || true +} +trap cleanup EXIT + +# Wait for the Wayland socket to appear. +sock="" +for _ in $(seq 1 40); do + sock="$(find "$XDG_RUNTIME_DIR" -maxdepth 1 -name 'wayland-*' ! -name '*.lock' 2>/dev/null | head -1 || true)" + [ -n "$sock" ] && break + sleep 0.5 +done +[ -n "$sock" ] || { echo "::error::compositor did not create a Wayland socket"; exit 1; } +export WAYLAND_DISPLAY="$(basename "$sock")" +echo "compositor up on WAYLAND_DISPLAY=$WAYLAND_DISPLAY" +echo "::endgroup::" + +# Give modules time to poll/settle. +sleep 6 + +echo "::group::Waybar log" +cat "$LOG" || true +echo "::endgroup::" + +# --- Level 1: smoke --- +if ! pgrep -x waybar >/dev/null; then + echo "::error::waybar is not running — it crashed on startup (config: $CONFIG)" + exit 1 +fi +if grep -iqE 'critical|segfault|\bfatal\b|terminate called' "$LOG"; then + echo "::error::waybar logged a fatal error (config: $CONFIG)" + exit 1 +fi +echo "✓ waybar launched and stayed alive under headless sway" + +# --- Level 2: screenshot --- +if [ -n "$SHOT" ]; then + grim "$SHOT" + echo "✓ captured screenshot -> $SHOT" +fi diff --git a/test/smoke/style.css b/test/smoke/style.css new file mode 100644 index 00000000..bf842ae1 --- /dev/null +++ b/test/smoke/style.css @@ -0,0 +1,27 @@ +/* Deterministic stylesheet for the CI smoke/screenshot test. + * A pinned font (DejaVu Sans, installed in CI) and fixed colors keep the + * rendered bar reproducible across runs for the golden comparison. */ +* { + font-family: "DejaVu Sans"; + font-size: 14px; + border: none; + border-radius: 0; + min-height: 0; +} + +window#waybar { + background-color: #1e1e2e; + color: #cdd6f4; +} + +#custom-left, +#custom-center, +#custom-right { + padding: 0 16px; + color: #cdd6f4; +} + +#custom-left { + font-weight: bold; + color: #89b4fa; +} From e8638e761e40a7496cba90c27413810a6d53dbb7 Mon Sep 17 00:00:00 2001 From: Alex Date: Sat, 4 Jul 2026 13:22:08 +0200 Subject: [PATCH 2/3] ci(smoke): make custom modules emit text so the bar is not empty The static custom modules used `exec: echo` (empty output), so the modules rendered nothing and the bar strip was blank. Have exec print the label text and use the default `{}` format. --- test/smoke/config.jsonc | 9 +++------ test/smoke/modules-config.jsonc | 3 +-- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/test/smoke/config.jsonc b/test/smoke/config.jsonc index 433b7007..461b04d8 100644 --- a/test/smoke/config.jsonc +++ b/test/smoke/config.jsonc @@ -9,18 +9,15 @@ "modules-center": ["custom/center"], "modules-right": ["custom/right"], "custom/left": { - "format": "Waybar", - "exec": "echo", + "exec": "echo Waybar", "interval": "once" }, "custom/center": { - "format": "CI smoke test", - "exec": "echo", + "exec": "echo 'CI smoke test'", "interval": "once" }, "custom/right": { - "format": "12:34", - "exec": "echo", + "exec": "echo 12:34", "interval": "once" } } diff --git a/test/smoke/modules-config.jsonc b/test/smoke/modules-config.jsonc index d6a7b9a8..108c51fc 100644 --- a/test/smoke/modules-config.jsonc +++ b/test/smoke/modules-config.jsonc @@ -12,8 +12,7 @@ "memory": {}, "disk": {}, "custom/hello": { - "format": "ok", - "exec": "echo", + "exec": "echo ok", "interval": "once" } } From 0c059db0fd670837ba87d97d04501caa5425333c Mon Sep 17 00:00:00 2001 From: Alex Date: Sat, 4 Jul 2026 13:28:48 +0200 Subject: [PATCH 3/3] ci(smoke): add reference screenshot to enable golden comparison MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Blessed from the CI artifact of the passing headless run (Waybar / CI smoke test / 12:34 on a pinned DejaVu Sans bar). Regenerate from a new artifact when an intended visual change trips the comparison — see test/smoke/README.md. --- test/smoke/reference.png | Bin 0 -> 9817 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 test/smoke/reference.png diff --git a/test/smoke/reference.png b/test/smoke/reference.png new file mode 100644 index 0000000000000000000000000000000000000000..8d3165e9a2c9865825da41c764c3decf80a450b0 GIT binary patch literal 9817 zcmeI2`8S(s|Hsp5i!MrA%d}H5X4*2f)xI;;MN!%y6eaduQcEnUNT;QWX|>c&w4$Q6 z*keyojIDM-hz?>8A|xb{e6{EK2fjZ%KRxar?)!bNb3XUyT(9fg@9TP>bH_d~HxfK~ z@gx8M5H!Af%K`x4e+mE`(-ipWV8pZBS_A<26<~bphE-VB+H`m@c(jRs6S*#U;^t2$ zl)EkoJkwN?`N>D#1a!=Iv^h@GS#(-sQGfB4KM8p0pgjQaZAQKZ zEB!-$CL03;zpKwssi%D3g%e)*&!gW(nZ3aKBj3fi{IP-WPr#bkPhLMnNIWm~NU+>S z*{U>zHPE@34R?#U-+vqcs5z$wd`xN(LN^f1MFD_j(4;w9(szVc{sDVQWHdt4c_g{A z$5Kl5ADxZ`^a{}LVI z$$igk@y~ScDC&t@xR<@nQh`NS)k@_tFDY>s`%owvToGrozHA#Qo4yadXy7&hPUS!s z6yV)F7cBtbNlbw=UL__w(9O}&u_tT1N{4S+|3l#L+lO;U`Gj=JAsY&H^2(Sv3gjMw zy*$&bU*u=s7uZPsyc&};USdY?NMDXsi&l{)Y;9I<^xEf4Ze?kg+;3)Tv^5a-9*?AB zW=hC=vFm<-Y&pnvjBfY?OjW5o$+H(-E%l{gjr74$#PB5*4}Lxztb_Gv!JcW;AbYMBtcQG#2hwMLsII5gxxUaDoR7g zhm(#1p6dN-Qj}Q(ZHcnsF}j#WXC_#8S8pOV8ho}a2}neMZLTfTQc;hcLTaU?zJ}D7 zybDIQ%$J|%K5VSG<}<>0S=rybwT^(Gxu56ZyuaTM>trMMgnSp|iuJr~r0Xk>1m6`~ z#62{q0zH4h4p|Y&RFhTJahrJos)zuad6?5*vpe&VS>5YnxyHJAg(eFqw1A+UVB^jv zFnmHw>%s-vElM1iPVPqS#72Mgrha7&zt?ByrEnP0IzU;|7s6_Kn_D)`#4AmR4jdX8 zOwNyQXL!RoFcxrHqXBw*qSup*qSaAWEu0%%t)m~3oe5$2HmkcY5>RxBr!jd=Yyc=3yn;Gs8M{`t$NuzF9}H(@0kaPB`+`}GT#Q|^T$ zw%c(1q<7v}{Cdz>GFF0warX$M<|dFMW}5?s3C*3_W3|L&jYq?t%NU~Wrn$43+~U(D zMLpMnvl5x3B?|j4-L?GO%}K1NJyp_pL(652(zJT#U%c!6Gg3;-I#U#DDDe*I6N`1O z!zzngz4a4G9suATz0z~lfHpoayt2Le_O!j8gn|;iZqcvDlLhM-8 z9ZKc%P1saF?sXJ2yof9$G4|2SZ!7CnxQ`Wqbiqc_X8RwtNT=U-ylUCm=1d`By*J}`?@xxHi~ zpdG43rN3Wprm)&H1y?eD1I!3$!|CboP2o!S*@=bqW_Iuu5g!c+Z_ac2YXs$ArQg@T zE=9J0+hDA4TynAsx@oSEV%wxxhuH1s`&#?>k+brx9ia>Q3oH7jTuqPNyj<<@=XV}O z%t@4=6>qL82#Jr+!|O8YR~6T;)6qSWY_HqDxdm&_twSPUU6PXZoaXlA9lH6?A?*@2 zews`M-o|fm$%g0Jo3H_^rKqSXYiCze3rEyb9;nO}Q4YT1o69@OE@~8CSq=4%aP~BJ z8wlFKnMXh1%ty5(oGy&y zc#Le`nyn@tpUTeNLI3uYF-v`n1mv1c{%ZK8z@m**0yR-btCFc*37wQh-fNH*A= z^ypxljd)UyL1Jz9Zrf{59FqKo*!74Y`*Q=*7k;HwZ!LE3=5{ z`gV-lo;1qVPSntEIJ}xpcC6CA(8?O(Fo_%|LPvQGyQ=t?I9HNH5IttTF`zP~*fd)B z+-S|gz8rzSJrmJQyb{l4SaZADHo`x6a_7@kux6RtEmO(hUlQJl%5ef+4`$F1;!o{< z7M+}F*HP_DI31oYm|J#r5`Qx@tOc@IJRN))#A(l}kgB6rx(-|CA1vBeB~0X~Q~PcX zWPfPNY?Ll99|pF220*uT6e5w7JDoN}tc)#HP*r{D7xE)C6;jGlX5JFLrpe z6BFfv?!&+={UBqrF{Y3K+$lu&7D&XfP?<+LE-g`x8XI(A$rUHrXqbrfX0rM;&aFT0 zHS>|RCJLaB02n8`5?d*1JESTYFtetQnD1wlyP2ICb8lm)$difYNAlv5zG^L2Lmanz zJ%17M9}T>kWUuym(3)2x4OT+=IFpr{60~+p_L7Lu3~Le7-f%`Py;E#`%8^R7@%KHE zv)(%n0B}0v)Yc=p)4#vA|})S977IF|KGw{!I|qG0#Z@$wl4sjU~jMGD0|zv zcb+#l(DW^`(oC6<3F_?g^Fc1J5eR!zX*x!awnsQMMkt(VL<|=8hizk7S&hTuS3wO2 zWpYJOY6nEYw#p-@(xGxUSqB#OnDQ1+*5qv7DHk@pE7m7p&_x|AqG-U!yKY_kGMl-u zB?u-xY_WAloh4d zy**ag&}=YEE%tY2tVGy9Z;F&t7OGZB2ChYa`Is0)ROijO%PX%$TkTYQf+hQ_@vN?4 zU<63>SkZ5KMaJfGCf0r70U-goS0`y{2WMgjiz0*FeZ3fR_J8Hk+JB zmFhfwY5ZHi_9Qiy$0t zFe64<6uT&RF(8n1d~PGVygek=s&6(R^i|xQ*FdCe?Y|2~`K1mT-cz;F_$(#X><*K_ zNXX6LGqVrXD>Pc0HtOu3U|l7wR1B%vvLz<#Y+Hr|)qbggh}JYk;ojL;J)xWcd`9IJ zI9965Uu+*{`U|%YWrX%p%cMY#RuApclzkgX5I6M++98dsfWIjqWmV#6(O7>i z>dhd17KC+g)MqvS0m(OP>V2Ku^`Ig5alqgiyakM;aC|@tq>j1+zJ_-cu>e=brH;H1HmvZPc$VnKPAPw5B8PUY zpf!BFzrT~vM?e|!^0V)+Bx=!yZO~?dde}WNl3L;wScLu}wok}U*k@}o(qcO~=h*1* zMkIFrWwQ}ea8%aR02wlSYOdQ-Dq{Hi61g8#%!E=ZBsp$N<(}jSlVLtuA^3JBwvo5F zkq?zdo{`E^w;G#(Ep4{gX!xf&HXsIjL!pRouX5eIIR1+1g;}(JUOp)*)uv7w#dusL z%4Kj?btPnf|I1tP*9cBl{UB#iaevvAR5Zyr*J{&qDg<%uKWB~(i$ z-W-kJXG{Mm-#s%qm+Mia(te