Vitest: integration test of the Electron API server (health endpoint, 404 handler).
Playwright driving Electron: login flow against ruv.wtf with the dedicated giteaci account, plus a failed-login case that must show the error without crashing the app.
Failure artifacts: screenshot, main-process and renderer console logs, HTML report uploaded on failure only.
Gitea Actions workflow (xvfb-run, single worker) on pull requests.
No credential in the repo. The password comes from the E2E_PASSWORD secret / env var; E2E_HOMESERVER and E2E_USER default to ruv.wtf and giteaci.
Review panel applied before opening: fork guard on the e2e job, dev server pinned to loopback, token value never printed on failure, authenticated state proven by a token-authenticated request instead of a localStorage string, DevTools window no longer mistaken for the app window, custom globalSetup deleted in favour of Playwright webServer.
Checks run: npm run test:unit (2 passed), npm run test:e2e (2 passed, ~18s).
Verdicts follow as comments.
Closes #109.
Two layers:
- Vitest: integration test of the Electron API server (health endpoint, 404 handler).
- Playwright driving Electron: login flow against ruv.wtf with the dedicated giteaci account, plus a failed-login case that must show the error without crashing the app.
- Failure artifacts: screenshot, main-process and renderer console logs, HTML report uploaded on failure only.
- Gitea Actions workflow (xvfb-run, single worker) on pull requests.
No credential in the repo. The password comes from the E2E_PASSWORD secret / env var; E2E_HOMESERVER and E2E_USER default to ruv.wtf and giteaci.
Review panel applied before opening: fork guard on the e2e job, dev server pinned to loopback, token value never printed on failure, authenticated state proven by a token-authenticated request instead of a localStorage string, DevTools window no longer mistaken for the app window, custom globalSetup deleted in favour of Playwright webServer.
Checks run: npm run test:unit (2 passed), npm run test:e2e (2 passed, ~18s).
Verdicts follow as comments.
Playwright's webServer replaces the custom globalSetup. The e2e job is gated to same-repo PRs because a fork PR controls both the test files and the workflow steps while the E2E_PASSWORD secret is in the environment. The dev server is pinned to loopback, the login screen is asserted before filling, the authenticated state is proven by a token-authenticated request, the token value is never printed, the failed-login test no longer asserts a tautological exitCode, and the DevTools window is no longer mistaken for the app window.
Blocking finding was the fork guard. on: pull_request runs fork PRs, and a fork PR controls both the test files and the workflow steps, so anything a fork controls runs with E2E_PASSWORD in its environment. Gitea omits repo secrets for fork runs today, but that gate is per-user-per-repo and permanent after one approval, so it is not a control. Fixed: the e2e job now runs only when github.event.pull_request.head.repo.full_name == github.repository. The unit job stays open to forks.
Applied:
playwright.config.mjs — dev server pinned to loopback. cinny/vite.config.js binds 0.0.0.0 with fs.allow: ['..'], which is reachable from other jobs on a shared runner and can serve files above the project root.
tests/e2e/login.spec.mjs — the access token is no longer printed on failure. Only the HTTP status is asserted, so a failure cannot put a live token into the CI log or the uploaded report.
Left as accepted risk:
E2E_HOMESERVER still comes from env. With the fork guard, the value is set by the repo, not by a fork.
Tracing is deliberately NOT enabled. A trace would capture the login POST body, which contains the password. Failure artifacts are screenshot + console logs only.
The failure screenshot only leaks the password if the reveal toggle is clicked; the field is masked. No main or renderer log prints the password or token.
cinny/vite.config.js open CORS relay (src/app/vite proxy, logs POST bodies) is in the submodule, out of scope here.
No trace, no video, no storageState uploaded. The Electron profile lives in os.tmpdir() and is not uploaded.
Still needs: the E2E_PASSWORD secret on the repo before the e2e job can pass.
Security panel: BLOCK, applied.
Blocking finding was the fork guard. `on: pull_request` runs fork PRs, and a fork PR controls both the test files and the workflow steps, so anything a fork controls runs with `E2E_PASSWORD` in its environment. Gitea omits repo secrets for fork runs today, but that gate is per-user-per-repo and permanent after one approval, so it is not a control. Fixed: the e2e job now runs only when `github.event.pull_request.head.repo.full_name == github.repository`. The unit job stays open to forks.
Applied:
- `playwright.config.mjs` — dev server pinned to loopback. `cinny/vite.config.js` binds `0.0.0.0` with `fs.allow: ['..']`, which is reachable from other jobs on a shared runner and can serve files above the project root.
- `tests/e2e/login.spec.mjs` — the access token is no longer printed on failure. Only the HTTP status is asserted, so a failure cannot put a live token into the CI log or the uploaded report.
Left as accepted risk:
- `E2E_HOMESERVER` still comes from env. With the fork guard, the value is set by the repo, not by a fork.
- Tracing is deliberately NOT enabled. A trace would capture the login POST body, which contains the password. Failure artifacts are screenshot + console logs only.
- The failure screenshot only leaks the password if the reveal toggle is clicked; the field is masked. No main or renderer log prints the password or token.
- `cinny/vite.config.js` open CORS relay (`src/app/vite` proxy, logs POST bodies) is in the submodule, out of scope here.
- No trace, no video, no storageState uploaded. The Electron profile lives in `os.tmpdir()` and is not uploaded.
Still needs: the `E2E_PASSWORD` secret on the repo before the e2e job can pass.
criterion 1: the login screen is asserted (server, username, password, submit visible) before anything is filled. It was a waitForSelector('input') wait, which any input satisfied.
criterion 3: the session read is polled, and the token is proven by a token-authenticated request to /_matrix/client/v3/capabilities. A truthy accessToken only proved a string existed. /versions and /account 404 on ruv.wtf, so /capabilities is the endpoint that exists and requires a valid token.
criterion 4: asserts an authenticated route (#/(inbox|home|room|space)) instead of "not a login route". A post-login crash to a blank screen used to pass.
criterion 5: exitCode() was near-tautological — if the process had exited, the earlier getByText would have failed first. Replaced with a renderer liveness check plus the form still being usable.
missing E2E_PASSWORD no longer produces two TypeErrors over the clear message.
temp Electron profile is removed after each test.
Not applied, and why:
no shell-element assertion. cinny has no data-testid; asserting a class selector would be fragile. The authenticated state is proven by route plus a token-authenticated request instead.
the ServerPicker input is still selected by position. Naming it needs a submodule edit, out of scope. Noted in the test comment.
no --no-sandbox: Playwright already adds it for Electron on Linux.
Rate limiting: the re-login after the failed login was dropped. It was the main login-budget cost; the suite now costs 2 login requests per run. retries: 1 in CI only. This is a real flake source against production Synapse — the wrong-password case can fail on the rate-limit message rather than the app.
Found during verification, not in the panel: DevTools opens in dev mode and can be the first window registered, so firstWindow() sometimes returns the DevTools page. The test now takes the non-DevTools window.
Still misses, by design: renderer crash during post-login init, single-instance lock regression, API server inside the running app. The Vitest test is a smoke test of Express plumbing and covers no acceptance criterion.
Test-quality panel: BLOCK, applied.
What the suite proves now:
- criterion 1: the login screen is asserted (server, username, password, submit visible) before anything is filled. It was a `waitForSelector('input')` wait, which any input satisfied.
- criterion 3: the session read is polled, and the token is proven by a token-authenticated request to `/_matrix/client/v3/capabilities`. A truthy `accessToken` only proved a string existed. `/versions` and `/account` 404 on ruv.wtf, so `/capabilities` is the endpoint that exists and requires a valid token.
- criterion 4: asserts an authenticated route (`#/(inbox|home|room|space)`) instead of "not a login route". A post-login crash to a blank screen used to pass.
- criterion 5: `exitCode()` was near-tautological — if the process had exited, the earlier `getByText` would have failed first. Replaced with a renderer liveness check plus the form still being usable.
- missing `E2E_PASSWORD` no longer produces two TypeErrors over the clear message.
- temp Electron profile is removed after each test.
Not applied, and why:
- no shell-element assertion. cinny has no `data-testid`; asserting a class selector would be fragile. The authenticated state is proven by route plus a token-authenticated request instead.
- the ServerPicker input is still selected by position. Naming it needs a submodule edit, out of scope. Noted in the test comment.
- no `--no-sandbox`: Playwright already adds it for Electron on Linux.
Rate limiting: the re-login after the failed login was dropped. It was the main login-budget cost; the suite now costs 2 login requests per run. `retries: 1` in CI only. This is a real flake source against production Synapse — the wrong-password case can fail on the rate-limit message rather than the app.
Found during verification, not in the panel: DevTools opens in dev mode and can be the first window registered, so `firstWindow()` sometimes returns the DevTools page. The test now takes the non-DevTools window.
Still misses, by design: renderer crash during post-login init, single-instance lock regression, API server inside the running app. The Vitest test is a smoke test of Express plumbing and covers no acceptance criterion.
tests/e2e/dev-server.mjs (whole file). It reimplemented Playwright's webServer: poll loop, reuse check, detached-process kill. process.kill(-child.pid) was POSIX-only, so it broke on Windows dev machines. webServer handles teardown cross-platform and reuseExistingServer already defaults to !process.env.CI. The DEV_URL export was used nowhere.
outputDir and open: false in playwright.config.mjs, environment: 'node' in vitest.config.mjs — all defaults.
the submodule URL rewrite step in the workflow. .gitmodules already has url = ../cinny.git, so submodules: true is enough. That step was a copy of the build.yml workaround from when .gitmodules pointed at public GitHub HTTPS.
the npm cache step: the unit job has no cache, so having one only here was inconsistent.
page.goto(DEV_URL): electron/main.js already loads the dev server, and the extra navigation made the page origin differ from the app's own.
Kept, because it earns its keep:
the manual afterEach hook. Config-based screenshot and trace do not fire for a page taken from app.firstWindow(), and console capture is not covered by config at all.
include in vitest.config.mjs — load-bearing, the default pattern would try to run the Electron spec.
workers: 1 and timeout — the single-instance lock and Electron startup.
xvfb step, failure-only artifact upload.
Not applied: root test-api.js and test-api.sh are superseded by tests/unit/, but deleting them is out of scope for this PR.
README gained one line: the E2E test logs in to a live homeserver over the network, so it needs network access and a working test account.
Simplicity panel: BLOCK, applied.
Deleted:
- `tests/e2e/dev-server.mjs` (whole file). It reimplemented Playwright's `webServer`: poll loop, reuse check, detached-process kill. `process.kill(-child.pid)` was POSIX-only, so it broke on Windows dev machines. `webServer` handles teardown cross-platform and `reuseExistingServer` already defaults to `!process.env.CI`. The `DEV_URL` export was used nowhere.
- `outputDir` and `open: false` in `playwright.config.mjs`, `environment: 'node'` in `vitest.config.mjs` — all defaults.
- the submodule URL rewrite step in the workflow. `.gitmodules` already has `url = ../cinny.git`, so `submodules: true` is enough. That step was a copy of the build.yml workaround from when `.gitmodules` pointed at public GitHub HTTPS.
- the npm cache step: the unit job has no cache, so having one only here was inconsistent.
- `page.goto(DEV_URL)`: `electron/main.js` already loads the dev server, and the extra navigation made the page origin differ from the app's own.
Kept, because it earns its keep:
- the manual `afterEach` hook. Config-based screenshot and trace do not fire for a page taken from `app.firstWindow()`, and console capture is not covered by config at all.
- `include` in `vitest.config.mjs` — load-bearing, the default pattern would try to run the Electron spec.
- `workers: 1` and `timeout` — the single-instance lock and Electron startup.
- xvfb step, failure-only artifact upload.
Not applied: root `test-api.js` and `test-api.sh` are superseded by `tests/unit/`, but deleting them is out of scope for this PR.
README gained one line: the E2E test logs in to a live homeserver over the network, so it needs network access and a working test account.
The login form only renders after the auth-flows request succeeds. On the runner the form never appeared within the expect timeout, so the login request to ruv.wtf (192.168.10.4) is not reachable from the runner network. The check turns a 30s locator timeout into a clear message.
CI blocker: the runner cannot reach the homeserver.
Evidence, run 927, job e2e: curl: (28) Connection timed out after 20002 milliseconds and 000 from the reachability step.
ruv.wtf resolves to 192.168.10.4. The runner host is on a different network — the git remote in the job log is http://synbox.ruv.wtf:8418 (192.168.1.130). The container reaches github.com and archive.ubuntu.com, so it has internet but no route to 192.168.10.0/24. DNS resolves, the connection is black-holed.
Why run 926 looked like a test failure instead of a network failure: cinny renders the password form only after mx.loginFlows() succeeds (cinny/src/app/components/AuthFlowsLoader.tsx), and autoDiscovery treats a fetch error as IGNORE and falls back to the entered host. So an unreachable homeserver shows up as a 30s toBeVisible timeout on input[name="usernameInput"], never as a network error. The reachability step now names the cause before the test runs.
Options:
Give the runner a route to 192.168.10.4, or run the e2e job on a runner inside that network.
Point E2E_HOMESERVER at a homeserver reachable from the runner and move the giteaci account there.
Until one of those is done, the e2e job cannot pass. The unit job passes.
Other CI fixes landed on this branch: npx playwright install-deps (Electron would not launch on the runner image), apt lists refreshed before installing xvfb, and upload-artifact@v3 because @v4 rejects Gitea as GHES.
CI blocker: the runner cannot reach the homeserver.
Evidence, run 927, job e2e:
`curl: (28) Connection timed out after 20002 milliseconds` and `000` from the reachability step.
`ruv.wtf` resolves to `192.168.10.4`. The runner host is on a different network — the git remote in the job log is `http://synbox.ruv.wtf:8418` (`192.168.1.130`). The container reaches `github.com` and `archive.ubuntu.com`, so it has internet but no route to `192.168.10.0/24`. DNS resolves, the connection is black-holed.
Why run 926 looked like a test failure instead of a network failure: cinny renders the password form only after `mx.loginFlows()` succeeds (`cinny/src/app/components/AuthFlowsLoader.tsx`), and `autoDiscovery` treats a fetch error as IGNORE and falls back to the entered host. So an unreachable homeserver shows up as a 30s `toBeVisible` timeout on `input[name="usernameInput"]`, never as a network error. The reachability step now names the cause before the test runs.
Options:
1. Give the runner a route to `192.168.10.4`, or run the e2e job on a runner inside that network.
2. Point `E2E_HOMESERVER` at a homeserver reachable from the runner and move the `giteaci` account there.
Until one of those is done, the e2e job cannot pass. The unit job passes.
Other CI fixes landed on this branch: `npx playwright install-deps` (Electron would not launch on the runner image), apt lists refreshed before installing xvfb, and `upload-artifact@v3` because `@v4` rejects Gitea as GHES.
The homeserver ruv.wtf (192.168.10.4) is a macvlan interface on the caddy-caddy-1 container on the runner host pizzabox. Replies to a bridge-networked job arrive on br-f192811a98b7 and DOCKER-ISOLATION-STAGE-2 drops them, so the login E2E times out. Host networking keeps request and reply on one path.
Run 928 logged "--network and --net in the options will be ignored", so
NetworkMode stayed the per-job bridge. The fix is a DOCKER-USER exception on
the runner host (pizzabox), which accepts RELATED/ESTABLISHED traffic from
br-f192811a98b7; verified 200 from a bridge container after adding it.
act_runner mounts /var/run/docker.sock into every container job, so the fork guard
has to cover the unit job too. The e2e comment now states the blast radius of the
host DOCKER-USER exception instead of the removed container.options. Playwright deps
are narrowed to chromium, which is what Electron uses, and the homeserver reachability
curl gates with -f so a non-2xx fails the job.
The login test claimed "without crashing the app" but asserted only that the body had
text, which is true for an error page. It now asserts no uncaught renderer pageerror,
and toHaveValue after the homeserver commit proves fill() actually stuck.
Test-quality panel (re-review, head 879a7155): PASS.
Criterion 1 — launch + login screen: tests/e2e/login.spec.mjs:56 launches with a fresh temp profile, :60-65 skips devtools:// windows, :23-28 asserts four visible fields before any fill. False positive: visibility only — a form rendered against the wrong server still passes. The server commit is now asserted at :35 (toHaveValue), which closes the earlier gap where fill() could silently keep matrix.org.
Criterion 2 — fill + submit: :40-42. fill() still does not verify the value stuck for the username/password fields; the new value assertion at :35 is the only such guard in the suite.
Criterion 3 — real authentication: :107-115 polls matrixSessions[0].accessToken and asserts HTTP 200 from /_matrix/client/v3/capabilities with the bearer token. Only non-trivial proof in the suite. False positive: the request comes from Node, not from the app, so it proves the token is valid server-side, not that the app uses it.
Criterion 4 — authenticated app state: :99 shell-route regex, :101-103 session user id, :118 login form gone. Smoke-level: a shell that renders with an empty room list passes.
Criterion 5 — failed login: :129 error text, :134 no uncaught renderer error, :135 form still usable. The "without crashing" claim is now actually asserted; previously nothing proved renderer liveness.
Criterion 6 — artifacts: :78-94 screenshot plus main/renderer console on failure; pageerrors now land in the console artifact too (:67-70).
Selector fragility: input[name="usernameInput"] and input[name="passwordInput"] come from cinny's PasswordLoginForm.tsx; the homeserver input has no name or test id, so serverInput() is positional (:18) and breaks if the form gains a field before it.
Remaining flake sources: Synapse rc_login — a 429 renders a different FieldError, so the exact string at :129 fails for the wrong reason. retries: 1 in CI doubles the login cost on a retried run, which the "2 logins per run" claim does not account for. workers: 1 keeps the budget flat.
Not asserted at all: room list contents, sync completion, session persistence across restarts.
Checks: npm run test:unit — 2 passed. E2E not re-run locally, the test password is not available in this session; CI on 879a7155 runs it.
Test-quality panel (re-review, head `879a7155`): PASS.
Criterion 1 — launch + login screen: `tests/e2e/login.spec.mjs:56` launches with a fresh temp profile, `:60-65` skips `devtools://` windows, `:23-28` asserts four visible fields before any fill. False positive: visibility only — a form rendered against the wrong server still passes. The server commit is now asserted at `:35` (`toHaveValue`), which closes the earlier gap where `fill()` could silently keep `matrix.org`.
Criterion 2 — fill + submit: `:40-42`. `fill()` still does not verify the value stuck for the username/password fields; the new value assertion at `:35` is the only such guard in the suite.
Criterion 3 — real authentication: `:107-115` polls `matrixSessions[0].accessToken` and asserts HTTP 200 from `/_matrix/client/v3/capabilities` with the bearer token. Only non-trivial proof in the suite. False positive: the request comes from Node, not from the app, so it proves the token is valid server-side, not that the app uses it.
Criterion 4 — authenticated app state: `:99` shell-route regex, `:101-103` session user id, `:118` login form gone. Smoke-level: a shell that renders with an empty room list passes.
Criterion 5 — failed login: `:129` error text, `:134` no uncaught renderer error, `:135` form still usable. The "without crashing" claim is now actually asserted; previously nothing proved renderer liveness.
Criterion 6 — artifacts: `:78-94` screenshot plus main/renderer console on failure; pageerrors now land in the console artifact too (`:67-70`).
Selector fragility: `input[name="usernameInput"]` and `input[name="passwordInput"]` come from cinny's `PasswordLoginForm.tsx`; the homeserver input has no name or test id, so `serverInput()` is positional (`:18`) and breaks if the form gains a field before it.
Remaining flake sources: Synapse `rc_login` — a 429 renders a different `FieldError`, so the exact string at `:129` fails for the wrong reason. `retries: 1` in CI doubles the login cost on a retried run, which the "2 logins per run" claim does not account for. `workers: 1` keeps the budget flat.
Not asserted at all: room list contents, sync completion, session persistence across restarts.
Checks: `npm run test:unit` — 2 passed. E2E not re-run locally, the test password is not available in this session; CI on `879a7155` runs it.
Simplicity panel (re-review, head 879a7155): BLOCK, applied.
Claim disproved: the DevTools loop at tests/e2e/login.spec.mjs:62-65 is not unreachable. app.windows() does surface a devtools:// window — observed locally, firstWindow() returned http://localhost:38347/ and DevTools appeared as a second window. electron/main.js:972 opens DevTools right after loadURL, so registration order is a race. Kept as a race guard; in 4 runs firstWindow() returned the app window, so it is defensive rather than proven necessary.
Deleted:
tests/e2e/login.spec.mjs — document.body.innerText non-empty assertion. It proved nothing the next line did not.
.gitea/workflows/tests.yml:63 — install-deps with no argument installs chromium, firefox and webkit deps. Electron needs chromium only, so the step now passes chromium.
Fixed:
tests/e2e/login.spec.mjs:75-77 — comment claimed config screenshots fire for traces; trace is off because a trace captures the login POST body.
.gitea/workflows/tests.yml:32-36 — comment described container.options, which no longer exists.
.gitea/workflows/tests.yml:72 — curl had no -f, so a 403/500 printed and passed; only a connect failure gated. It now gates on non-2xx. The step itself earns its place: a curl failure is a better message than a 30s locator timeout.
Kept, with reason stated:
.gitea/workflows/tests.yml:39-40 env block — pins CI to the dedicated test account regardless of test defaults.
tests/e2e/login.spec.mjs:118 — login form gone is a distinct fact from the URL check at :99.
tests/e2e/login.spec.mjs:101-103 — the app's own session identity, not the same fact as the token check.
Already minimal: vitest.config.mjs include-only, no custom globalSetup, no DEV_URL export, no trace, artifact upload on failure only.
Simplicity panel (re-review, head `879a7155`): BLOCK, applied.
Claim disproved: the DevTools loop at `tests/e2e/login.spec.mjs:62-65` is not unreachable. `app.windows()` does surface a `devtools://` window — observed locally, `firstWindow()` returned `http://localhost:38347/` and DevTools appeared as a second window. `electron/main.js:972` opens DevTools right after `loadURL`, so registration order is a race. Kept as a race guard; in 4 runs `firstWindow()` returned the app window, so it is defensive rather than proven necessary.
Deleted:
- `tests/e2e/login.spec.mjs` — `document.body.innerText` non-empty assertion. It proved nothing the next line did not.
- `.gitea/workflows/tests.yml:63` — `install-deps` with no argument installs chromium, firefox and webkit deps. Electron needs chromium only, so the step now passes `chromium`.
Fixed:
- `tests/e2e/login.spec.mjs:75-77` — comment claimed config screenshots fire for traces; trace is off because a trace captures the login POST body.
- `.gitea/workflows/tests.yml:32-36` — comment described `container.options`, which no longer exists.
- `.gitea/workflows/tests.yml:72` — curl had no `-f`, so a 403/500 printed and passed; only a connect failure gated. It now gates on non-2xx. The step itself earns its place: a curl failure is a better message than a 30s locator timeout.
Kept, with reason stated:
- `.gitea/workflows/tests.yml:39-40` env block — pins CI to the dedicated test account regardless of test defaults.
- `tests/e2e/login.spec.mjs:118` — login form gone is a distinct fact from the URL check at `:99`.
- `tests/e2e/login.spec.mjs:101-103` — the app's own session identity, not the same fact as the token check.
Already minimal: `vitest.config.mjs` include-only, no custom globalSetup, no `DEV_URL` export, no trace, artifact upload on failure only.
Security panel (re-review, head 879a7155): BLOCK, applied.
Fork guard was on the e2e job only. act_runner mounts /var/run/docker.sock into every container job: nektos/act pkg/runner/run_context.goGetBindsAndMounts() appends the daemon-socket bind unless Config.ContainerDaemonSocket is -. Confirmed in the run 928 job log (Binds:[/var/run/docker.sock:/var/run/docker.sock]). A fork PR controls package.json, so npm ci runs fork-controlled install scripts inside a container that can reach the Docker daemon on pizzabox — the host running the homeserver container. Guard now on both jobs (.gitea/workflows/tests.yml:10).
The second runner daemon maps ubuntu-latest to host, so a fork job can execute directly on that host. Same guard covers it.
Secrets: checked. act does not put unreferenced secrets into the job environment (GetEnv() merges workflow env, job env and Config.Env only), so the unguarded unit job never carried E2E_PASSWORD. Finding moot with the guard.
The DOCKER-USER exception is host-wide, so it also covers fork PR jobs. It is RELATED,ESTABLISHED only, so it permits replies, not new connections from br-f192811a98b7. The bridge name is Docker-generated: recreating the caddy container renames it and the test fails closed. Comment now states the blast radius (.gitea/workflows/tests.yml:32-36).
Kept: the E2E_HOMESERVER/E2E_USER env block (.gitea/workflows/tests.yml:39-40). It pins CI to the dedicated test account regardless of the test defaults; the comment states that reason.
What I would have argued against: leaving the unit job open to forks. It only survives if the runner stops mounting the daemon socket, or the repo moves to a runner that never executes fork code. On pizzabox neither is true.
Checked and fine: no credential in the repo, trace stays off (a trace captures the login POST body), token value never printed on failure, dev server pinned to loopback.
Security panel (re-review, head `879a7155`): BLOCK, applied.
1. Fork guard was on the e2e job only. act_runner mounts `/var/run/docker.sock` into every container job: nektos/act `pkg/runner/run_context.go` `GetBindsAndMounts()` appends the daemon-socket bind unless `Config.ContainerDaemonSocket` is `-`. Confirmed in the run 928 job log (`Binds:[/var/run/docker.sock:/var/run/docker.sock]`). A fork PR controls `package.json`, so `npm ci` runs fork-controlled install scripts inside a container that can reach the Docker daemon on `pizzabox` — the host running the homeserver container. Guard now on both jobs (`.gitea/workflows/tests.yml:10`).
2. The second runner daemon maps `ubuntu-latest` to `host`, so a fork job can execute directly on that host. Same guard covers it.
3. Secrets: checked. act does not put unreferenced secrets into the job environment (`GetEnv()` merges workflow env, job env and `Config.Env` only), so the unguarded unit job never carried `E2E_PASSWORD`. Finding moot with the guard.
4. The `DOCKER-USER` exception is host-wide, so it also covers fork PR jobs. It is `RELATED,ESTABLISHED` only, so it permits replies, not new connections from `br-f192811a98b7`. The bridge name is Docker-generated: recreating the caddy container renames it and the test fails closed. Comment now states the blast radius (`.gitea/workflows/tests.yml:32-36`).
5. Kept: the `E2E_HOMESERVER`/`E2E_USER` env block (`.gitea/workflows/tests.yml:39-40`). It pins CI to the dedicated test account regardless of the test defaults; the comment states that reason.
What I would have argued against: leaving the unit job open to forks. It only survives if the runner stops mounting the daemon socket, or the repo moves to a runner that never executes fork code. On `pizzabox` neither is true.
Checked and fine: no credential in the repo, trace stays off (a trace captures the login POST body), token value never printed on failure, dev server pinned to loopback.
Fork guard on the unit job too (.gitea/workflows/tests.yml:10) — act_runner mounts /var/run/docker.sock into every container job (GetBindsAndMounts(), default ContainerDaemonSocket = /var/run/docker.sock), so a fork PR reaches the Docker daemon on pizzabox, the host running the homeserver container.
DOCKER-USER comment now states the blast radius (.gitea/workflows/tests.yml:32-36).
Reachability curl gates with -f (.gitea/workflows/tests.yml:72).
toHaveValue after the homeserver commit (tests/e2e/login.spec.mjs:35).
Stale trace comment fixed (tests/e2e/login.spec.mjs:75-77).
Not applied — disproved by CI run 930: narrowing install-deps to chromium. Run 930 failed with Error: Process failed to launch!. The chromium-only dep set drops libgtk-3-0t64, libegl1, libegl-mesa0, libgles2, libsecret-1-0, libxtst6, libxss1, which Electron needs to launch. Playwright's dep list covers its own bundled browser, not Electron. Reverted in d12ce65a.
Also not applied: the DevTools loop is not dead code. app.windows() does surface a devtools:// window — observed locally. It is a race guard, not a certainty: firstWindow() returned the app window in 4 runs.
Diagnosability gap worth knowing: when the launch itself fails, afterEach returns early (tests/e2e/login.spec.mjs:79), so there is no screenshot and no console log — only Playwright's error-context. Run 930's artifact had 7 files and no screenshot.
Checks: npm run test:unit — 2 passed. E2E not re-run locally, the test password is not available in this session; CI on d12ce65a is the check.
Re-review applied. Head `d12ce65a`.
Applied:
- Fork guard on the unit job too (`.gitea/workflows/tests.yml:10`) — act_runner mounts `/var/run/docker.sock` into every container job (`GetBindsAndMounts()`, default `ContainerDaemonSocket` = `/var/run/docker.sock`), so a fork PR reaches the Docker daemon on `pizzabox`, the host running the homeserver container.
- `DOCKER-USER` comment now states the blast radius (`.gitea/workflows/tests.yml:32-36`).
- Reachability curl gates with `-f` (`.gitea/workflows/tests.yml:72`).
- Tautological `innerText` assertion deleted; renderer `pageerror` assertion added (`tests/e2e/login.spec.mjs:134`, handler `:67-70`).
- `toHaveValue` after the homeserver commit (`tests/e2e/login.spec.mjs:35`).
- Stale trace comment fixed (`tests/e2e/login.spec.mjs:75-77`).
Not applied — disproved by CI run 930: narrowing `install-deps` to `chromium`. Run 930 failed with `Error: Process failed to launch!`. The chromium-only dep set drops `libgtk-3-0t64`, `libegl1`, `libegl-mesa0`, `libgles2`, `libsecret-1-0`, `libxtst6`, `libxss1`, which Electron needs to launch. Playwright's dep list covers its own bundled browser, not Electron. Reverted in `d12ce65a`.
Also not applied: the DevTools loop is not dead code. `app.windows()` does surface a `devtools://` window — observed locally. It is a race guard, not a certainty: `firstWindow()` returned the app window in 4 runs.
Diagnosability gap worth knowing: when the launch itself fails, `afterEach` returns early (`tests/e2e/login.spec.mjs:79`), so there is no screenshot and no console log — only Playwright's error-context. Run 930's artifact had 7 files and no screenshot.
Checks: `npm run test:unit` — 2 passed. E2E not re-run locally, the test password is not available in this session; CI on `d12ce65a` is the check.
Run 931 (head d12ce65a): login test passed (15.3s), the new pageerror assertion failed.
The app legitimately produces pageerrors on the failed-login path:
Forbidden: MatrixError: Unknown message — the failed login itself
SecurityError: Failed to register a ServiceWorker ... dev-sw.js ... unsupported MIME type ('text/html') — dev-mode artifact
So asserting zero pageerrors asserts the wrong thing. Removed in 347acd0e (net deletion). The handler stays so the errors land in the failure artifact. "Without crashing" is now proven by the form still rendering — a dead renderer cannot render it.
The toHaveValue assertion added in the same round passed. The install-deps revert worked: Electron launched.
Run 932 on 347acd0e is the check.
Run 931 (head `d12ce65a`): login test passed (15.3s), the new `pageerror` assertion failed.
The app legitimately produces pageerrors on the failed-login path:
- `Forbidden: MatrixError: Unknown message` — the failed login itself
- `SecurityError: Failed to register a ServiceWorker ... dev-sw.js ... unsupported MIME type ('text/html')` — dev-mode artifact
So asserting zero pageerrors asserts the wrong thing. Removed in `347acd0e` (net deletion). The handler stays so the errors land in the failure artifact. "Without crashing" is now proven by the form still rendering — a dead renderer cannot render it.
The `toHaveValue` assertion added in the same round passed. The install-deps revert worked: Electron launched.
Run 932 on `347acd0e` is the check.
Run 932 on 347acd0e: green. 2 passed (24.7s) — login 14.8s, failed login 7.5s. Unit job green.
Panel blocks cleared by the applied commits:
security: fork guard on both jobs (879a7155)
simplicity: tautological assertion deleted, stale comments fixed, curl now gates (879a7155)
Three findings stay contested, so they stay in the thread:
simplicity claimed the DevTools loop is unreachable. Disproved: app.windows() does surface a devtools:// window. It is a race guard, not a certainty — firstWindow() returned the app window in 4 local runs.
simplicity narrowed install-deps to chromium. Run 930 failed with Error: Process failed to launch!; the chromium-only set drops libgtk-3-0t64, libegl1, libegl-mesa0, libgles2, libsecret-1-0, libxtst6, libxss1. Reverted in d12ce65a.
test-quality asked for a liveness assertion. It cannot be written as "no pageerrors": the failed-login path produces MatrixError: Unknown message and dev mode produces a ServiceWorker registration error. Replaced with the form-still-renders proof in 347acd0e.
Tradeoff of the fork guard: fork PRs now get no CI at all, not just no E2E. That is the cost of running on a host that hosts the homeserver container.
Merge is blocked until the panels accept the applied state.
Run 932 on `347acd0e`: green. `2 passed (24.7s)` — login 14.8s, failed login 7.5s. Unit job green.
Panel blocks cleared by the applied commits:
- security: fork guard on both jobs (`879a7155`)
- simplicity: tautological assertion deleted, stale comments fixed, curl now gates (`879a7155`)
Three findings stay contested, so they stay in the thread:
- simplicity claimed the DevTools loop is unreachable. Disproved: `app.windows()` does surface a `devtools://` window. It is a race guard, not a certainty — `firstWindow()` returned the app window in 4 local runs.
- simplicity narrowed `install-deps` to chromium. Run 930 failed with `Error: Process failed to launch!`; the chromium-only set drops libgtk-3-0t64, libegl1, libegl-mesa0, libgles2, libsecret-1-0, libxtst6, libxss1. Reverted in `d12ce65a`.
- test-quality asked for a liveness assertion. It cannot be written as "no pageerrors": the failed-login path produces `MatrixError: Unknown message` and dev mode produces a ServiceWorker registration error. Replaced with the form-still-renders proof in `347acd0e`.
Tradeoff of the fork guard: fork PRs now get no CI at all, not just no E2E. That is the cost of running on a host that hosts the homeserver container.
Merge is blocked until the panels accept the applied state.
tests/e2e/login.spec.mjs: assert the render process is alive with the `crash`
event instead of counting pageerrors; the failed-login test now accepts the
rate-limited message as well as the wrong-password one; the failure hook
attaches the console before the app guard so a launch failure still produces an
artifact, wraps `screenshot()` because it throws on a crashed page, and reaches
`fs.rmSync` on launch failure; dropped the duplicate login-form assertion.
playwright.config.mjs: pin `trace: 'off'` explicitly rather than relying on the
version default, since a trace captures the login POST body.
.gitea/workflows/tests.yml: dropped `--prefer-offline`; corrected the xvfb
comment (the guard is live for the host runner, which already has xvfb); dropped
an unresolvable CI run-number citation.
Checks: `node --check` on both JS files, `npm run test:unit` — 2 passed. E2E
cannot be run locally (no `E2E_PASSWORD` in this session); CI is the check.
Real authentication — :102-110 live /capabilities 200 against the stored token. Strongest assertion in the file.
Authenticated app state — :94 shell route, :97-98currentSessionUserId. Partial: nothing proves the shell rendered content.
Failed login error without crashing — :124 exact error string, :130 form still visible. Name overclaims.
Local + CI, diagnosable output — playwright.config.mjs:18-21, .gitea/workflows/tests.yml:69-83, hook at :73-89. Partial: launch-failure path produces no artifact.
Applied at 1ddca424:
Liveness is now asserted by the crash event (:69-71, :140), not by counting pageerrors. crash is render-process-gone, so it does not fire on the MatrixError or the ServiceWorker error — no false positive, and the test name now matches the assertion (:128).
The failure hook attaches the console before the app guard (:79-88), so a launch failure still produces an artifact. Run 930 produced none because the guard returned before both attaches. screenshot() is wrapped (:85) because it throws on a crashed page and masks the original failure. fs.rmSync is now reachable on launch failure (:98), so the temp profile no longer leaks.
The failed-login assertion accepts the rate-limited branch (:133). Under rc_login cinny renders LoginError.RateLimited (cinny/src/app/pages/auth/login/PasswordLoginForm.tsx:248), a different message than the exact string it asserted before.
trace: 'off' is pinned in playwright.config.mjs rather than relying on the version default; a trace captures the login POST body.
Duplicate proof deleted: the login-form-gone assertion restated the shell-route check.
Not applied, and why: the DevTools window wait at :62-65 has no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. The test timeout already bounds it; adding a race plus a throw for a path that has not occurred in 4 local runs is more code than the failure is worth. This is the finding I would still hold, as a nicety, not a block.
Still not asserted, by design: main-process liveness, that the app's own matrix client is authenticated, that the shell rendered content beyond the route, that the login request went to the entered host (:35 proves the DOM value, not the committed server state).
Selector fragility stands: positional input.first() (:18), name="usernameInput" and "passwordInput", button[type="submit"] (strict-mode violation if a second form renders), the shell-route regex, the localStorage keys, and the error text all come from pinned cinny internals.
Checks: npm run test:unit — 2 passed. E2E not re-run locally, the test password is not available in this session; CI on 1ddca424 runs it.
Test-quality panel (re-review, head `347acd0e`): BLOCK, applied.
Criterion mapping at that head:
1. Launch + login screen — `tests/e2e/login.spec.mjs:54` launch, `:22-27` `expectLoginForm`, called at `:35`. Covered.
2. Homeserver + credentials + submit — `:29-36`, `:38-42`. Covered.
3. Real authentication — `:102-110` live `/capabilities` 200 against the stored token. Strongest assertion in the file.
4. Authenticated app state — `:94` shell route, `:97-98` `currentSessionUserId`. Partial: nothing proves the shell rendered content.
5. Failed login error without crashing — `:124` exact error string, `:130` form still visible. Name overclaims.
6. Local + CI, diagnosable output — `playwright.config.mjs:18-21`, `.gitea/workflows/tests.yml:69-83`, hook at `:73-89`. Partial: launch-failure path produces no artifact.
Applied at `1ddca424`:
- Liveness is now asserted by the `crash` event (`:69-71`, `:140`), not by counting pageerrors. `crash` is render-process-gone, so it does not fire on the MatrixError or the ServiceWorker error — no false positive, and the test name now matches the assertion (`:128`).
- The failure hook attaches the console before the app guard (`:79-88`), so a launch failure still produces an artifact. Run 930 produced none because the guard returned before both attaches. `screenshot()` is wrapped (`:85`) because it throws on a crashed page and masks the original failure. `fs.rmSync` is now reachable on launch failure (`:98`), so the temp profile no longer leaks.
- The failed-login assertion accepts the rate-limited branch (`:133`). Under `rc_login` cinny renders `LoginError.RateLimited` (`cinny/src/app/pages/auth/login/PasswordLoginForm.tsx:248`), a different message than the exact string it asserted before.
- `trace: 'off'` is pinned in `playwright.config.mjs` rather than relying on the version default; a trace captures the login POST body.
- Duplicate proof deleted: the login-form-gone assertion restated the shell-route check.
Not applied, and why: the DevTools window wait at `:62-65` has no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. The test timeout already bounds it; adding a race plus a throw for a path that has not occurred in 4 local runs is more code than the failure is worth. This is the finding I would still hold, as a nicety, not a block.
Still not asserted, by design: main-process liveness, that the app's own matrix client is authenticated, that the shell rendered content beyond the route, that the login request went to the entered host (`:35` proves the DOM value, not the committed `server` state).
Selector fragility stands: positional `input.first()` (`:18`), `name="usernameInput"` and `"passwordInput"`, `button[type="submit"]` (strict-mode violation if a second form renders), the shell-route regex, the localStorage keys, and the error text all come from pinned cinny internals.
Checks: `npm run test:unit` — 2 passed. E2E not re-run locally, the test password is not available in this session; CI on `1ddca424` runs it.
Fork guard on both jobs — tests.yml:10 and :30, identical condition. My earlier argument was that the unit job is "just a test run"; that fails, because npm ci executes fork-controlled package.json hooks and nektos GetBindsAndMounts() binds /var/run/docker.sock unless Config.ContainerDaemonSocket is -. A socket-mounted container job is a host escape regardless of secrets. Change survives.
Secrets — cleared. GetEnv() merges workflow env + job env + Config.Env, so unreferenced secrets never enter the environment; the unit job never carried E2E_PASSWORD. That means the guard on the unit job was never about the secret — it is about the socket. Caveat: GITEA_TOKEN is injected regardless of reference and Gitea grants it write access on fork runs (GHSA-32h8-5xqg-h466), so the unit job is not secret-free.
DOCKER-USER blast radius — tests.yml:32-36. The wording overstates it: the rule is scoped to -i br-f192811a98b7 and accepts only RELATED,ESTABLISHED, so it permits return traffic on one bridge, not host-wide inbound. The accurate claim is that it applies to every job on that bridge, including fork jobs. It fails closed on a bridge rename, which is correct.
container.options — cleared, no container: key remains at head.
New findings, none a repo change:
tests.yml:10,30: the guard is self-attested. Gitea reads the workflow from the PR head, so a fork deletes the if: and still gets secrets after approval. It is defense-in-depth, not a boundary, and survives only because Gitea's fork-PR approval gate is on.
The shared ubuntu-latest label covers a container runner and a host runner. A fork PR job landing on the host runner has no isolation at all — worse than the socket mount. This is the finding I would still hold, as a runner-side fix.
tests.yml:86-92 plus login.spec.mjs:80: failure artifacts include raw renderer console output, which can carry a live access token if cinny logs one, and artifacts are readable by anyone with repo read. Traces off is correct; password fields render masked, so failure.png is fine. Residual, low.
playwright.config.mjs:12: retries: 1 doubles Synapse rc_login spend per run. Flake risk on giteaci, not security.
Runner-side controls are the actual boundary: container.daemon_socket: "-" on the runner, and a dedicated label for the host runner so ubuntu-latest is unambiguous. Keep the workflow guard as policy, not as the security control. The cost — fork PRs get no CI — is acceptable for this repo, and the alternative (secrets on fork runs) is strictly worse.
Checked, fine: secret referenced not hardcoded (tests.yml:42), missing-secret path fails with a clear message (login.spec.mjs:47-49), only HTTP status asserted so no token reaches the log (login.spec.mjs:100-110), dev server pinned to loopback closing the fs.allow: ['..'] exposure, fresh profile per run, curl -sSf gates on non-2xx, v3 artifact correct for Gitea 1.27.3, build.yml and checks.yml are push-only with no fork exposure, vitest.config.mjs:6 cannot pick up the e2e spec, tests/unit/api-server.test.js:10 uses port 0.
Security panel (re-review, head `347acd0e`): PASS.
Prior findings cleared:
1. Fork guard on both jobs — `tests.yml:10` and `:30`, identical condition. My earlier argument was that the unit job is "just a test run"; that fails, because `npm ci` executes fork-controlled `package.json` hooks and nektos `GetBindsAndMounts()` binds `/var/run/docker.sock` unless `Config.ContainerDaemonSocket` is `-`. A socket-mounted container job is a host escape regardless of secrets. Change survives.
2. Secrets — cleared. `GetEnv()` merges workflow env + job env + `Config.Env`, so unreferenced secrets never enter the environment; the unit job never carried `E2E_PASSWORD`. That means the guard on the unit job was never about the secret — it is about the socket. Caveat: `GITEA_TOKEN` is injected regardless of reference and Gitea grants it write access on fork runs (GHSA-32h8-5xqg-h466), so the unit job is not secret-free.
3. DOCKER-USER blast radius — `tests.yml:32-36`. The wording overstates it: the rule is scoped to `-i br-f192811a98b7` and accepts only RELATED,ESTABLISHED, so it permits return traffic on one bridge, not host-wide inbound. The accurate claim is that it applies to every job on that bridge, including fork jobs. It fails closed on a bridge rename, which is correct.
4. `container.options` — cleared, no `container:` key remains at head.
New findings, none a repo change:
- `tests.yml:10,30`: the guard is self-attested. Gitea reads the workflow from the PR head, so a fork deletes the `if:` and still gets secrets after approval. It is defense-in-depth, not a boundary, and survives only because Gitea's fork-PR approval gate is on.
- The shared `ubuntu-latest` label covers a container runner and a `host` runner. A fork PR job landing on the host runner has no isolation at all — worse than the socket mount. This is the finding I would still hold, as a runner-side fix.
- `tests.yml:86-92` plus `login.spec.mjs:80`: failure artifacts include raw renderer console output, which can carry a live access token if cinny logs one, and artifacts are readable by anyone with repo read. Traces off is correct; password fields render masked, so `failure.png` is fine. Residual, low.
- `playwright.config.mjs:12`: `retries: 1` doubles Synapse `rc_login` spend per run. Flake risk on `giteaci`, not security.
Runner-side controls are the actual boundary: `container.daemon_socket: "-"` on the runner, and a dedicated label for the host runner so `ubuntu-latest` is unambiguous. Keep the workflow guard as policy, not as the security control. The cost — fork PRs get no CI — is acceptable for this repo, and the alternative (secrets on fork runs) is strictly worse.
Checked, fine: secret referenced not hardcoded (`tests.yml:42`), missing-secret path fails with a clear message (`login.spec.mjs:47-49`), only HTTP status asserted so no token reaches the log (`login.spec.mjs:100-110`), dev server pinned to loopback closing the `fs.allow: ['..']` exposure, fresh profile per run, `curl -sSf` gates on non-2xx, v3 artifact correct for Gitea 1.27.3, `build.yml` and `checks.yml` are push-only with no fork exposure, `vitest.config.mjs:6` cannot pick up the e2e spec, `tests/unit/api-server.test.js:10` uses port 0.
Simplicity panel (re-review, head 347acd0e): PASS.
Concessions:
DevTools loop — wrong, I concede. electron/main.js:18 sets isDev = NODE_ENV === 'development' || !app.isPackaged, and electron . is never packaged, so openDevTools() at main.js:972 fires on every E2E run. Not a hypothetical race — it is the live path, and the loop is what makes firstWindow() deterministic. What I would have argued against: the fallback at login.spec.mjs:62 waits on app.once('window') with no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. That is the only surviving half of the finding, and it is not blocking.
install-deps — wrong, I concede. Playwright's chromium dep set covers its bundled browser, not Electron. The observed Process failed to launch! is empirical proof. The comment earns its place: the narrowing is the obvious next edit, and without the comment someone redoes it. Only the CI run-number citation was rot-prone — a run number is unresolvable outside the PR thread. Reason kept, citation dropped.
Tautological assertion — wrong, I concede. Demanding zero pageerrors on a live-network login is not a crash test; the failed login legitimately throws MatrixError and dev mode legitimately throws a ServiceWorker error. The handler is an artifact source, not an assertion. The crash event is the honest replacement, and the test name now matches it.
Deleted at 1ddca424: --prefer-offline, the duplicate login-form assertion, the run-number citation, and the trace sentence moved to the config where the setting lives.
Corrected rather than deleted: the xvfb guard. I claimed its left branch is always false. It is not — the host runner already has xvfb, and both runners share the ubuntu-latest label, so the comment was wrong, not the guard.
Contested: reporter: [['list'], ['html']] at playwright.config.mjs:13. An explicit reporter replaces the default rather than adding to it, so dropping ['list'] loses the CI console log, which is the diagnosability requirement. Kept.
Already minimal: positional locator (no name or test id exists on ServerPicker), toHaveValue after the server commit, currentSessionUserId, /capabilities as the only endpoint that exists on ruv.wtf and requires a valid token, per-run temp profile, SIGTERM fallback, password guard, fork guards, curl -f, loopback --host, include in vitest.config.mjs, workers and retries.
Docker image: agree with the author, no image. Measured 110s total, of which an image could only remove install-deps (36s) plus the apt update inside the xvfb step. That trades 36s for a permanently maintained artifact — build, push, runner label, version drift against Playwright and Electron — and the dep list is exactly the thing that broke, so the image would need rebuilding whenever Electron bumps. cache: 'npm' is the smaller change, with one correction: setup-node keys the cache on a single lockfile, so it covers the root npm ci (7s) and not the cinny submodule (41s) unless cache-dependency-path points at both. Caching the npm download cache does not help install-deps at all.
Simplicity panel (re-review, head `347acd0e`): PASS.
Concessions:
1. DevTools loop — wrong, I concede. `electron/main.js:18` sets `isDev = NODE_ENV === 'development' || !app.isPackaged`, and `electron .` is never packaged, so `openDevTools()` at `main.js:972` fires on every E2E run. Not a hypothetical race — it is the live path, and the loop is what makes `firstWindow()` deterministic. What I would have argued against: the fallback at `login.spec.mjs:62` waits on `app.once('window')` with no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. That is the only surviving half of the finding, and it is not blocking.
2. `install-deps` — wrong, I concede. Playwright's chromium dep set covers its bundled browser, not Electron. The observed `Process failed to launch!` is empirical proof. The comment earns its place: the narrowing is the obvious next edit, and without the comment someone redoes it. Only the CI run-number citation was rot-prone — a run number is unresolvable outside the PR thread. Reason kept, citation dropped.
3. Tautological assertion — wrong, I concede. Demanding zero pageerrors on a live-network login is not a crash test; the failed login legitimately throws `MatrixError` and dev mode legitimately throws a ServiceWorker error. The handler is an artifact source, not an assertion. The `crash` event is the honest replacement, and the test name now matches it.
Deleted at `1ddca424`: `--prefer-offline`, the duplicate login-form assertion, the run-number citation, and the trace sentence moved to the config where the setting lives.
Corrected rather than deleted: the xvfb guard. I claimed its left branch is always false. It is not — the host runner already has xvfb, and both runners share the `ubuntu-latest` label, so the comment was wrong, not the guard.
Contested: `reporter: [['list'], ['html']]` at `playwright.config.mjs:13`. An explicit `reporter` replaces the default rather than adding to it, so dropping `['list']` loses the CI console log, which is the diagnosability requirement. Kept.
Already minimal: positional locator (no name or test id exists on ServerPicker), `toHaveValue` after the server commit, `currentSessionUserId`, `/capabilities` as the only endpoint that exists on ruv.wtf and requires a valid token, per-run temp profile, SIGTERM fallback, password guard, fork guards, `curl -f`, loopback `--host`, `include` in `vitest.config.mjs`, workers and retries.
Docker image: agree with the author, no image. Measured 110s total, of which an image could only remove `install-deps` (36s) plus the apt update inside the xvfb step. That trades 36s for a permanently maintained artifact — build, push, runner label, version drift against Playwright and Electron — and the dep list is exactly the thing that broke, so the image would need rebuilding whenever Electron bumps. `cache: 'npm'` is the smaller change, with one correction: setup-node keys the cache on a single lockfile, so it covers the root `npm ci` (7s) and not the cinny submodule (41s) unless `cache-dependency-path` points at both. Caching the npm download cache does not help `install-deps` at all.
Round two: security PASS (comment 987), simplicity PASS (comment 988), test-quality BLOCK (comment 986). All three had held a block in round one, so all three re-reviewed against the current head; a stale PASS does not clear a merge.
Applied in 1ddca424:
tests/e2e/login.spec.mjs:69-71, :140 — liveness asserted by the crash event instead of counting pageerrors. Test renamed to reports an error and keeps the renderer alive (:128) so the name matches the assertion.
tests/e2e/login.spec.mjs:79-88, :98 — failure hook restructured: console attaches before the app guard, screenshot() wrapped, fs.rmSync reachable on launch failure. Run 930 produced zero artifacts because the guard returned before both attaches.
tests/e2e/login.spec.mjs:133 — failed-login assertion accepts the rate-limited branch as well as the wrong-password one.
playwright.config.mjs — trace: 'off' pinned explicitly rather than relying on the version default.
DevTools loop: simplicity conceded it. electron/main.js:18 makes isDev true for an unpackaged electron ., so openDevTools() at main.js:972 fires on every run. The loop is the live path, not a race guard. Kept.
install-deps narrowing: simplicity conceded it. Playwright's chromium dep set covers its bundled browser, not Electron. Kept un-narrowed, with the reason in the comment.
Pageerror assertion: simplicity conceded it. A zero-pageerror assertion on a live-network login is not a crash test. Replaced by the crash event.
Reporter: simplicity called ['list'] redundant as Playwright's default. It is not — an explicit reporter replaces the default, so dropping it loses the CI console log. Kept.
xvfb guard: simplicity called the left branch always false. It is not — the host runner already has xvfb. The comment was wrong, not the guard. Comment fixed, guard kept.
Skipped, stated: the DevTools window wait at tests/e2e/login.spec.mjs:62-65 has no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. The test timeout already bounds it; a race plus a throw for a path not observed in 4 local runs is more code than the failure is worth. Non-blocking, held by simplicity.
Runner-side, not this PR: container.daemon_socket: "-" on the runner, and a dedicated label for the host runner so ubuntu-latest is unambiguous. Security holds that the workflow guard is self-attested — a fork authors the guard — so the runner config is the real boundary. Cost of the guard, fork PRs get no CI, is accepted.
Merge gate: test-quality must re-review 1ddca424 to lift its block. Checks: npm run test:unit 2 passed, node --check on both JS files. E2E cannot be run locally — no E2E_PASSWORD in this session — so CI on 1ddca424 is the check for the E2E changes.
Decision (orchestrator), head `1ddca424`.
Round two: security PASS (comment 987), simplicity PASS (comment 988), test-quality BLOCK (comment 986). All three had held a block in round one, so all three re-reviewed against the current head; a stale PASS does not clear a merge.
Applied in `1ddca424`:
- `tests/e2e/login.spec.mjs:69-71`, `:140` — liveness asserted by the `crash` event instead of counting pageerrors. Test renamed to `reports an error and keeps the renderer alive` (`:128`) so the name matches the assertion.
- `tests/e2e/login.spec.mjs:79-88`, `:98` — failure hook restructured: console attaches before the app guard, `screenshot()` wrapped, `fs.rmSync` reachable on launch failure. Run 930 produced zero artifacts because the guard returned before both attaches.
- `tests/e2e/login.spec.mjs:133` — failed-login assertion accepts the rate-limited branch as well as the wrong-password one.
- `playwright.config.mjs` — `trace: 'off'` pinned explicitly rather than relying on the version default.
- `.gitea/workflows/tests.yml` — `--prefer-offline` dropped, run-number citation dropped, xvfb comment corrected.
- `tests/e2e/login.spec.mjs` — duplicate login-form assertion deleted.
Contested, and how it resolved:
- DevTools loop: simplicity conceded it. `electron/main.js:18` makes `isDev` true for an unpackaged `electron .`, so `openDevTools()` at `main.js:972` fires on every run. The loop is the live path, not a race guard. Kept.
- `install-deps` narrowing: simplicity conceded it. Playwright's chromium dep set covers its bundled browser, not Electron. Kept un-narrowed, with the reason in the comment.
- Pageerror assertion: simplicity conceded it. A zero-pageerror assertion on a live-network login is not a crash test. Replaced by the `crash` event.
- Reporter: simplicity called `['list']` redundant as Playwright's default. It is not — an explicit `reporter` replaces the default, so dropping it loses the CI console log. Kept.
- xvfb guard: simplicity called the left branch always false. It is not — the host runner already has xvfb. The comment was wrong, not the guard. Comment fixed, guard kept.
Skipped, stated: the DevTools window wait at `tests/e2e/login.spec.mjs:62-65` has no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. The test timeout already bounds it; a race plus a throw for a path not observed in 4 local runs is more code than the failure is worth. Non-blocking, held by simplicity.
Runner-side, not this PR: `container.daemon_socket: "-"` on the runner, and a dedicated label for the host runner so `ubuntu-latest` is unambiguous. Security holds that the workflow guard is self-attested — a fork authors the guard — so the runner config is the real boundary. Cost of the guard, fork PRs get no CI, is accepted.
Merge gate: test-quality must re-review `1ddca424` to lift its block. Checks: `npm run test:unit` 2 passed, `node --check` on both JS files. E2E cannot be run locally — no `E2E_PASSWORD` in this session — so CI on `1ddca424` is the check for the E2E changes.
Source of truth for the change: issue #109.
- The `crash` assertion now runs before the form check, because on a crashed page the locator throws TargetClosedError and names the wrong thing.
- The `crash` handler logs the crash, so the failure artifact is not empty on that path.
- The DevTools window loop records the window URLs, so a hang there names which windows existed instead of reporting a bare timeout.
- Comment states that a launch failure produces an empty console artifact by construction, so the report's failure message is the diagnosis there.
Checks: `node --check` on the file, `npm run test:unit` — 2 passed. E2E cannot be run locally (no `E2E_PASSWORD` in this session); CI is the check.
Test-quality panel (re-review, head 1ddca424): APPROVE. No block. Two notes worth code, applied at 87ba01fd.
Per finding:
Naming cleared — crash handler login.spec.mjs:69-72, assertion :141, name at :129 matches what is provable. But expect(crashed).toBe(false) was a sync check evaluated at one instant, and any real crash makes the locator at :140 throw first with TargetClosedError("Page crashed") (playwright-core_didCrash -> Events.Page.Crash). It could only fail in the microsecond window between the two. Fixed: the assertion now runs before the form check, so it is the signal rather than a backstop. page.on('crash') does fire for Electron, same CDP path.
Half cleared — the hook does run when beforeEach throws, verified empirically, and the console attaches before the app guard with fs.rmSync reachable. But .catch(() => null) means the crashed-page case still produces no screenshot, and on launch failure logs is empty because no handler is attached before launch. Fixed by naming the paths rather than adding an artifact: the crash handler now logs the crash, and the comment states the launch-failure artifact is empty by construction, so the report's failure message is the diagnosis there.
Cleared or accepted — trace: 'off' pinned (playwright.config.mjs:17), rate-limit branch accepted (login.spec.mjs:134). CORS concern is moot: preflight to https://ruv.wtf/_matrix/client/v3/capabilities returns 204 with access-control-allow-headers: ... Authorization. DevTools timeout: the author's call stands, 120s bounds it. What was worth the code was the diagnosis, not the timeout — the loop now records the window URLs (:63-64), so a hang names which windows existed instead of a bare Timeout 120000ms exceeded. webServer.timeout skipped is fine; note a webServer failure produces no artifacts at all, since no hook runs.
Holds as documented fragility, not a block: positional input.first() (:18), name="usernameInput"/"passwordInput", button[type="submit"] (:21), shell-route regex (:105), localStorage keys (:108, :115, matching cinny/src/app/state/sessions.ts:84,161). button[type="submit"] is unique on the login route today — only PasswordLoginForm.tsx:263 has one, ServerPicker.tsx has no form — so strict mode is safe now.
Holds as scope, not as a false claim: main-process liveness, client identity, shell content, error origin, request host, profile isolation are unasserted, and the name no longer claims them.
Regex check: rate-limited by server also appears in PasswordRegisterForm.tsx:380 and TokenLogin.tsx:72; neither renders on this route, so no false match today. It does accept a run where the password was never rejected — the test proves "an error is displayed", not "the login path rejected these credentials". Honest given the comment.
Deleted login-form assertion: criterion 4 is still proven — the error text plus usernameInput() visible requires a live renderer. Weaker only in that the form is now one field, not four.
Skipped, stated: .catch(() => kill) at :95 handles a rejected close(), not a hang. A hang still skips fs.rmSync and leaks the temp profile through the hook timeout. Bounding it needs a race, which is more code than a tmpdir leak is worth.
Checks: npm run test:unit — 2 passed. E2E not re-run locally, no E2E_PASSWORD in this session; CI on 87ba01fd is the check.
All three reviewers now hold no block.
Test-quality panel (re-review, head `1ddca424`): APPROVE. No block. Two notes worth code, applied at `87ba01fd`.
Per finding:
1. Naming cleared — `crash` handler `login.spec.mjs:69-72`, assertion `:141`, name at `:129` matches what is provable. But `expect(crashed).toBe(false)` was a sync check evaluated at one instant, and any real crash makes the locator at `:140` throw first with `TargetClosedError("Page crashed")` (`playwright-core` `_didCrash` -> `Events.Page.Crash`). It could only fail in the microsecond window between the two. Fixed: the assertion now runs before the form check, so it is the signal rather than a backstop. `page.on('crash')` does fire for Electron, same CDP path.
2. Half cleared — the hook does run when `beforeEach` throws, verified empirically, and the console attaches before the app guard with `fs.rmSync` reachable. But `.catch(() => null)` means the crashed-page case still produces no screenshot, and on launch failure `logs` is empty because no handler is attached before launch. Fixed by naming the paths rather than adding an artifact: the `crash` handler now logs the crash, and the comment states the launch-failure artifact is empty by construction, so the report's failure message is the diagnosis there.
3. Cleared or accepted — `trace: 'off'` pinned (`playwright.config.mjs:17`), rate-limit branch accepted (`login.spec.mjs:134`). CORS concern is moot: preflight to `https://ruv.wtf/_matrix/client/v3/capabilities` returns 204 with `access-control-allow-headers: ... Authorization`. DevTools timeout: the author's call stands, 120s bounds it. What was worth the code was the diagnosis, not the timeout — the loop now records the window URLs (`:63-64`), so a hang names which windows existed instead of a bare `Timeout 120000ms exceeded`. `webServer.timeout` skipped is fine; note a webServer failure produces no artifacts at all, since no hook runs.
4. Holds as documented fragility, not a block: positional `input.first()` (`:18`), `name="usernameInput"`/`"passwordInput"`, `button[type="submit"]` (`:21`), shell-route regex (`:105`), localStorage keys (`:108`, `:115`, matching `cinny/src/app/state/sessions.ts:84,161`). `button[type="submit"]` is unique on the login route today — only `PasswordLoginForm.tsx:263` has one, `ServerPicker.tsx` has no form — so strict mode is safe now.
5. Holds as scope, not as a false claim: main-process liveness, client identity, shell content, error origin, request host, profile isolation are unasserted, and the name no longer claims them.
Regex check: `rate-limited by server` also appears in `PasswordRegisterForm.tsx:380` and `TokenLogin.tsx:72`; neither renders on this route, so no false match today. It does accept a run where the password was never rejected — the test proves "an error is displayed", not "the login path rejected these credentials". Honest given the comment.
Deleted login-form assertion: criterion 4 is still proven — the error text plus `usernameInput()` visible requires a live renderer. Weaker only in that the form is now one field, not four.
Skipped, stated: `.catch(() => kill)` at `:95` handles a rejected `close()`, not a hang. A hang still skips `fs.rmSync` and leaks the temp profile through the hook timeout. Bounding it needs a race, which is more code than a tmpdir leak is worth.
Checks: `npm run test:unit` — 2 passed. E2E not re-run locally, no `E2E_PASSWORD` in this session; CI on `87ba01fd` is the check.
All three reviewers now hold no block.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Closes #109.
Two layers:
No credential in the repo. The password comes from the E2E_PASSWORD secret / env var; E2E_HOMESERVER and E2E_USER default to ruv.wtf and giteaci.
Review panel applied before opening: fork guard on the e2e job, dev server pinned to loopback, token value never printed on failure, authenticated state proven by a token-authenticated request instead of a localStorage string, DevTools window no longer mistaken for the app window, custom globalSetup deleted in favour of Playwright webServer.
Checks run: npm run test:unit (2 passed), npm run test:e2e (2 passed, ~18s).
Verdicts follow as comments.
Security panel: BLOCK, applied.
Blocking finding was the fork guard.
on: pull_requestruns fork PRs, and a fork PR controls both the test files and the workflow steps, so anything a fork controls runs withE2E_PASSWORDin its environment. Gitea omits repo secrets for fork runs today, but that gate is per-user-per-repo and permanent after one approval, so it is not a control. Fixed: the e2e job now runs only whengithub.event.pull_request.head.repo.full_name == github.repository. The unit job stays open to forks.Applied:
playwright.config.mjs— dev server pinned to loopback.cinny/vite.config.jsbinds0.0.0.0withfs.allow: ['..'], which is reachable from other jobs on a shared runner and can serve files above the project root.tests/e2e/login.spec.mjs— the access token is no longer printed on failure. Only the HTTP status is asserted, so a failure cannot put a live token into the CI log or the uploaded report.Left as accepted risk:
E2E_HOMESERVERstill comes from env. With the fork guard, the value is set by the repo, not by a fork.cinny/vite.config.jsopen CORS relay (src/app/viteproxy, logs POST bodies) is in the submodule, out of scope here.os.tmpdir()and is not uploaded.Still needs: the
E2E_PASSWORDsecret on the repo before the e2e job can pass.Test-quality panel: BLOCK, applied.
What the suite proves now:
waitForSelector('input')wait, which any input satisfied./_matrix/client/v3/capabilities. A truthyaccessTokenonly proved a string existed./versionsand/account404 on ruv.wtf, so/capabilitiesis the endpoint that exists and requires a valid token.#/(inbox|home|room|space)) instead of "not a login route". A post-login crash to a blank screen used to pass.exitCode()was near-tautological — if the process had exited, the earliergetByTextwould have failed first. Replaced with a renderer liveness check plus the form still being usable.E2E_PASSWORDno longer produces two TypeErrors over the clear message.Not applied, and why:
data-testid; asserting a class selector would be fragile. The authenticated state is proven by route plus a token-authenticated request instead.--no-sandbox: Playwright already adds it for Electron on Linux.Rate limiting: the re-login after the failed login was dropped. It was the main login-budget cost; the suite now costs 2 login requests per run.
retries: 1in CI only. This is a real flake source against production Synapse — the wrong-password case can fail on the rate-limit message rather than the app.Found during verification, not in the panel: DevTools opens in dev mode and can be the first window registered, so
firstWindow()sometimes returns the DevTools page. The test now takes the non-DevTools window.Still misses, by design: renderer crash during post-login init, single-instance lock regression, API server inside the running app. The Vitest test is a smoke test of Express plumbing and covers no acceptance criterion.
Simplicity panel: BLOCK, applied.
Deleted:
tests/e2e/dev-server.mjs(whole file). It reimplemented Playwright'swebServer: poll loop, reuse check, detached-process kill.process.kill(-child.pid)was POSIX-only, so it broke on Windows dev machines.webServerhandles teardown cross-platform andreuseExistingServeralready defaults to!process.env.CI. TheDEV_URLexport was used nowhere.outputDirandopen: falseinplaywright.config.mjs,environment: 'node'invitest.config.mjs— all defaults..gitmodulesalready hasurl = ../cinny.git, sosubmodules: trueis enough. That step was a copy of the build.yml workaround from when.gitmodulespointed at public GitHub HTTPS.page.goto(DEV_URL):electron/main.jsalready loads the dev server, and the extra navigation made the page origin differ from the app's own.Kept, because it earns its keep:
afterEachhook. Config-based screenshot and trace do not fire for a page taken fromapp.firstWindow(), and console capture is not covered by config at all.includeinvitest.config.mjs— load-bearing, the default pattern would try to run the Electron spec.workers: 1andtimeout— the single-instance lock and Electron startup.Not applied: root
test-api.jsandtest-api.share superseded bytests/unit/, but deleting them is out of scope for this PR.README gained one line: the E2E test logs in to a live homeserver over the network, so it needs network access and a working test account.
CI blocker: the runner cannot reach the homeserver.
Evidence, run 927, job e2e:
curl: (28) Connection timed out after 20002 millisecondsand000from the reachability step.ruv.wtfresolves to192.168.10.4. The runner host is on a different network — the git remote in the job log ishttp://synbox.ruv.wtf:8418(192.168.1.130). The container reachesgithub.comandarchive.ubuntu.com, so it has internet but no route to192.168.10.0/24. DNS resolves, the connection is black-holed.Why run 926 looked like a test failure instead of a network failure: cinny renders the password form only after
mx.loginFlows()succeeds (cinny/src/app/components/AuthFlowsLoader.tsx), andautoDiscoverytreats a fetch error as IGNORE and falls back to the entered host. So an unreachable homeserver shows up as a 30stoBeVisibletimeout oninput[name="usernameInput"], never as a network error. The reachability step now names the cause before the test runs.Options:
192.168.10.4, or run the e2e job on a runner inside that network.E2E_HOMESERVERat a homeserver reachable from the runner and move thegiteaciaccount there.Until one of those is done, the e2e job cannot pass. The unit job passes.
Other CI fixes landed on this branch:
npx playwright install-deps(Electron would not launch on the runner image), apt lists refreshed before installing xvfb, andupload-artifact@v3because@v4rejects Gitea as GHES.Test-quality panel (re-review, head
879a7155): PASS.Criterion 1 — launch + login screen:
tests/e2e/login.spec.mjs:56launches with a fresh temp profile,:60-65skipsdevtools://windows,:23-28asserts four visible fields before any fill. False positive: visibility only — a form rendered against the wrong server still passes. The server commit is now asserted at:35(toHaveValue), which closes the earlier gap wherefill()could silently keepmatrix.org.Criterion 2 — fill + submit:
:40-42.fill()still does not verify the value stuck for the username/password fields; the new value assertion at:35is the only such guard in the suite.Criterion 3 — real authentication:
:107-115pollsmatrixSessions[0].accessTokenand asserts HTTP 200 from/_matrix/client/v3/capabilitieswith the bearer token. Only non-trivial proof in the suite. False positive: the request comes from Node, not from the app, so it proves the token is valid server-side, not that the app uses it.Criterion 4 — authenticated app state:
:99shell-route regex,:101-103session user id,:118login form gone. Smoke-level: a shell that renders with an empty room list passes.Criterion 5 — failed login:
:129error text,:134no uncaught renderer error,:135form still usable. The "without crashing" claim is now actually asserted; previously nothing proved renderer liveness.Criterion 6 — artifacts:
:78-94screenshot plus main/renderer console on failure; pageerrors now land in the console artifact too (:67-70).Selector fragility:
input[name="usernameInput"]andinput[name="passwordInput"]come from cinny'sPasswordLoginForm.tsx; the homeserver input has no name or test id, soserverInput()is positional (:18) and breaks if the form gains a field before it.Remaining flake sources: Synapse
rc_login— a 429 renders a differentFieldError, so the exact string at:129fails for the wrong reason.retries: 1in CI doubles the login cost on a retried run, which the "2 logins per run" claim does not account for.workers: 1keeps the budget flat.Not asserted at all: room list contents, sync completion, session persistence across restarts.
Checks:
npm run test:unit— 2 passed. E2E not re-run locally, the test password is not available in this session; CI on879a7155runs it.Simplicity panel (re-review, head
879a7155): BLOCK, applied.Claim disproved: the DevTools loop at
tests/e2e/login.spec.mjs:62-65is not unreachable.app.windows()does surface adevtools://window — observed locally,firstWindow()returnedhttp://localhost:38347/and DevTools appeared as a second window.electron/main.js:972opens DevTools right afterloadURL, so registration order is a race. Kept as a race guard; in 4 runsfirstWindow()returned the app window, so it is defensive rather than proven necessary.Deleted:
tests/e2e/login.spec.mjs—document.body.innerTextnon-empty assertion. It proved nothing the next line did not..gitea/workflows/tests.yml:63—install-depswith no argument installs chromium, firefox and webkit deps. Electron needs chromium only, so the step now passeschromium.Fixed:
tests/e2e/login.spec.mjs:75-77— comment claimed config screenshots fire for traces; trace is off because a trace captures the login POST body..gitea/workflows/tests.yml:32-36— comment describedcontainer.options, which no longer exists..gitea/workflows/tests.yml:72— curl had no-f, so a 403/500 printed and passed; only a connect failure gated. It now gates on non-2xx. The step itself earns its place: a curl failure is a better message than a 30s locator timeout.Kept, with reason stated:
.gitea/workflows/tests.yml:39-40env block — pins CI to the dedicated test account regardless of test defaults.tests/e2e/login.spec.mjs:118— login form gone is a distinct fact from the URL check at:99.tests/e2e/login.spec.mjs:101-103— the app's own session identity, not the same fact as the token check.Already minimal:
vitest.config.mjsinclude-only, no custom globalSetup, noDEV_URLexport, no trace, artifact upload on failure only.Security panel (re-review, head
879a7155): BLOCK, applied./var/run/docker.sockinto every container job: nektos/actpkg/runner/run_context.goGetBindsAndMounts()appends the daemon-socket bind unlessConfig.ContainerDaemonSocketis-. Confirmed in the run 928 job log (Binds:[/var/run/docker.sock:/var/run/docker.sock]). A fork PR controlspackage.json, sonpm ciruns fork-controlled install scripts inside a container that can reach the Docker daemon onpizzabox— the host running the homeserver container. Guard now on both jobs (.gitea/workflows/tests.yml:10).ubuntu-latesttohost, so a fork job can execute directly on that host. Same guard covers it.GetEnv()merges workflow env, job env andConfig.Envonly), so the unguarded unit job never carriedE2E_PASSWORD. Finding moot with the guard.DOCKER-USERexception is host-wide, so it also covers fork PR jobs. It isRELATED,ESTABLISHEDonly, so it permits replies, not new connections frombr-f192811a98b7. The bridge name is Docker-generated: recreating the caddy container renames it and the test fails closed. Comment now states the blast radius (.gitea/workflows/tests.yml:32-36).E2E_HOMESERVER/E2E_USERenv block (.gitea/workflows/tests.yml:39-40). It pins CI to the dedicated test account regardless of the test defaults; the comment states that reason.What I would have argued against: leaving the unit job open to forks. It only survives if the runner stops mounting the daemon socket, or the repo moves to a runner that never executes fork code. On
pizzaboxneither is true.Checked and fine: no credential in the repo, trace stays off (a trace captures the login POST body), token value never printed on failure, dev server pinned to loopback.
Re-review applied. Head
d12ce65a.Applied:
.gitea/workflows/tests.yml:10) — act_runner mounts/var/run/docker.sockinto every container job (GetBindsAndMounts(), defaultContainerDaemonSocket=/var/run/docker.sock), so a fork PR reaches the Docker daemon onpizzabox, the host running the homeserver container.DOCKER-USERcomment now states the blast radius (.gitea/workflows/tests.yml:32-36).-f(.gitea/workflows/tests.yml:72).innerTextassertion deleted; rendererpageerrorassertion added (tests/e2e/login.spec.mjs:134, handler:67-70).toHaveValueafter the homeserver commit (tests/e2e/login.spec.mjs:35).tests/e2e/login.spec.mjs:75-77).Not applied — disproved by CI run 930: narrowing
install-depstochromium. Run 930 failed withError: Process failed to launch!. The chromium-only dep set dropslibgtk-3-0t64,libegl1,libegl-mesa0,libgles2,libsecret-1-0,libxtst6,libxss1, which Electron needs to launch. Playwright's dep list covers its own bundled browser, not Electron. Reverted ind12ce65a.Also not applied: the DevTools loop is not dead code.
app.windows()does surface adevtools://window — observed locally. It is a race guard, not a certainty:firstWindow()returned the app window in 4 runs.Diagnosability gap worth knowing: when the launch itself fails,
afterEachreturns early (tests/e2e/login.spec.mjs:79), so there is no screenshot and no console log — only Playwright's error-context. Run 930's artifact had 7 files and no screenshot.Checks:
npm run test:unit— 2 passed. E2E not re-run locally, the test password is not available in this session; CI ond12ce65ais the check.Run 931 (head
d12ce65a): login test passed (15.3s), the newpageerrorassertion failed.The app legitimately produces pageerrors on the failed-login path:
Forbidden: MatrixError: Unknown message— the failed login itselfSecurityError: Failed to register a ServiceWorker ... dev-sw.js ... unsupported MIME type ('text/html')— dev-mode artifactSo asserting zero pageerrors asserts the wrong thing. Removed in
347acd0e(net deletion). The handler stays so the errors land in the failure artifact. "Without crashing" is now proven by the form still rendering — a dead renderer cannot render it.The
toHaveValueassertion added in the same round passed. The install-deps revert worked: Electron launched.Run 932 on
347acd0eis the check.Run 932 on
347acd0e: green.2 passed (24.7s)— login 14.8s, failed login 7.5s. Unit job green.Panel blocks cleared by the applied commits:
879a7155)879a7155)Three findings stay contested, so they stay in the thread:
app.windows()does surface adevtools://window. It is a race guard, not a certainty —firstWindow()returned the app window in 4 local runs.install-depsto chromium. Run 930 failed withError: Process failed to launch!; the chromium-only set drops libgtk-3-0t64, libegl1, libegl-mesa0, libgles2, libsecret-1-0, libxtst6, libxss1. Reverted ind12ce65a.MatrixError: Unknown messageand dev mode produces a ServiceWorker registration error. Replaced with the form-still-renders proof in347acd0e.Tradeoff of the fork guard: fork PRs now get no CI at all, not just no E2E. That is the cost of running on a host that hosts the homeserver container.
Merge is blocked until the panels accept the applied state.
Test-quality panel (re-review, head
347acd0e): BLOCK, applied.Criterion mapping at that head:
tests/e2e/login.spec.mjs:54launch,:22-27expectLoginForm, called at:35. Covered.:29-36,:38-42. Covered.:102-110live/capabilities200 against the stored token. Strongest assertion in the file.:94shell route,:97-98currentSessionUserId. Partial: nothing proves the shell rendered content.:124exact error string,:130form still visible. Name overclaims.playwright.config.mjs:18-21,.gitea/workflows/tests.yml:69-83, hook at:73-89. Partial: launch-failure path produces no artifact.Applied at
1ddca424:crashevent (:69-71,:140), not by counting pageerrors.crashis render-process-gone, so it does not fire on the MatrixError or the ServiceWorker error — no false positive, and the test name now matches the assertion (:128).:79-88), so a launch failure still produces an artifact. Run 930 produced none because the guard returned before both attaches.screenshot()is wrapped (:85) because it throws on a crashed page and masks the original failure.fs.rmSyncis now reachable on launch failure (:98), so the temp profile no longer leaks.:133). Underrc_logincinny rendersLoginError.RateLimited(cinny/src/app/pages/auth/login/PasswordLoginForm.tsx:248), a different message than the exact string it asserted before.trace: 'off'is pinned inplaywright.config.mjsrather than relying on the version default; a trace captures the login POST body.Not applied, and why: the DevTools window wait at
:62-65has no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. The test timeout already bounds it; adding a race plus a throw for a path that has not occurred in 4 local runs is more code than the failure is worth. This is the finding I would still hold, as a nicety, not a block.Still not asserted, by design: main-process liveness, that the app's own matrix client is authenticated, that the shell rendered content beyond the route, that the login request went to the entered host (
:35proves the DOM value, not the committedserverstate).Selector fragility stands: positional
input.first()(:18),name="usernameInput"and"passwordInput",button[type="submit"](strict-mode violation if a second form renders), the shell-route regex, the localStorage keys, and the error text all come from pinned cinny internals.Checks:
npm run test:unit— 2 passed. E2E not re-run locally, the test password is not available in this session; CI on1ddca424runs it.Security panel (re-review, head
347acd0e): PASS.Prior findings cleared:
tests.yml:10and:30, identical condition. My earlier argument was that the unit job is "just a test run"; that fails, becausenpm ciexecutes fork-controlledpackage.jsonhooks and nektosGetBindsAndMounts()binds/var/run/docker.sockunlessConfig.ContainerDaemonSocketis-. A socket-mounted container job is a host escape regardless of secrets. Change survives.GetEnv()merges workflow env + job env +Config.Env, so unreferenced secrets never enter the environment; the unit job never carriedE2E_PASSWORD. That means the guard on the unit job was never about the secret — it is about the socket. Caveat:GITEA_TOKENis injected regardless of reference and Gitea grants it write access on fork runs (GHSA-32h8-5xqg-h466), so the unit job is not secret-free.tests.yml:32-36. The wording overstates it: the rule is scoped to-i br-f192811a98b7and accepts only RELATED,ESTABLISHED, so it permits return traffic on one bridge, not host-wide inbound. The accurate claim is that it applies to every job on that bridge, including fork jobs. It fails closed on a bridge rename, which is correct.container.options— cleared, nocontainer:key remains at head.New findings, none a repo change:
tests.yml:10,30: the guard is self-attested. Gitea reads the workflow from the PR head, so a fork deletes theif:and still gets secrets after approval. It is defense-in-depth, not a boundary, and survives only because Gitea's fork-PR approval gate is on.ubuntu-latestlabel covers a container runner and ahostrunner. A fork PR job landing on the host runner has no isolation at all — worse than the socket mount. This is the finding I would still hold, as a runner-side fix.tests.yml:86-92pluslogin.spec.mjs:80: failure artifacts include raw renderer console output, which can carry a live access token if cinny logs one, and artifacts are readable by anyone with repo read. Traces off is correct; password fields render masked, sofailure.pngis fine. Residual, low.playwright.config.mjs:12:retries: 1doubles Synapserc_loginspend per run. Flake risk ongiteaci, not security.Runner-side controls are the actual boundary:
container.daemon_socket: "-"on the runner, and a dedicated label for the host runner soubuntu-latestis unambiguous. Keep the workflow guard as policy, not as the security control. The cost — fork PRs get no CI — is acceptable for this repo, and the alternative (secrets on fork runs) is strictly worse.Checked, fine: secret referenced not hardcoded (
tests.yml:42), missing-secret path fails with a clear message (login.spec.mjs:47-49), only HTTP status asserted so no token reaches the log (login.spec.mjs:100-110), dev server pinned to loopback closing thefs.allow: ['..']exposure, fresh profile per run,curl -sSfgates on non-2xx, v3 artifact correct for Gitea 1.27.3,build.ymlandchecks.ymlare push-only with no fork exposure,vitest.config.mjs:6cannot pick up the e2e spec,tests/unit/api-server.test.js:10uses port 0.Simplicity panel (re-review, head
347acd0e): PASS.Concessions:
electron/main.js:18setsisDev = NODE_ENV === 'development' || !app.isPackaged, andelectron .is never packaged, soopenDevTools()atmain.js:972fires on every E2E run. Not a hypothetical race — it is the live path, and the loop is what makesfirstWindow()deterministic. What I would have argued against: the fallback atlogin.spec.mjs:62waits onapp.once('window')with no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. That is the only surviving half of the finding, and it is not blocking.install-deps— wrong, I concede. Playwright's chromium dep set covers its bundled browser, not Electron. The observedProcess failed to launch!is empirical proof. The comment earns its place: the narrowing is the obvious next edit, and without the comment someone redoes it. Only the CI run-number citation was rot-prone — a run number is unresolvable outside the PR thread. Reason kept, citation dropped.MatrixErrorand dev mode legitimately throws a ServiceWorker error. The handler is an artifact source, not an assertion. Thecrashevent is the honest replacement, and the test name now matches it.Deleted at
1ddca424:--prefer-offline, the duplicate login-form assertion, the run-number citation, and the trace sentence moved to the config where the setting lives.Corrected rather than deleted: the xvfb guard. I claimed its left branch is always false. It is not — the host runner already has xvfb, and both runners share the
ubuntu-latestlabel, so the comment was wrong, not the guard.Contested:
reporter: [['list'], ['html']]atplaywright.config.mjs:13. An explicitreporterreplaces the default rather than adding to it, so dropping['list']loses the CI console log, which is the diagnosability requirement. Kept.Already minimal: positional locator (no name or test id exists on ServerPicker),
toHaveValueafter the server commit,currentSessionUserId,/capabilitiesas the only endpoint that exists on ruv.wtf and requires a valid token, per-run temp profile, SIGTERM fallback, password guard, fork guards,curl -f, loopback--host,includeinvitest.config.mjs, workers and retries.Docker image: agree with the author, no image. Measured 110s total, of which an image could only remove
install-deps(36s) plus the apt update inside the xvfb step. That trades 36s for a permanently maintained artifact — build, push, runner label, version drift against Playwright and Electron — and the dep list is exactly the thing that broke, so the image would need rebuilding whenever Electron bumps.cache: 'npm'is the smaller change, with one correction: setup-node keys the cache on a single lockfile, so it covers the rootnpm ci(7s) and not the cinny submodule (41s) unlesscache-dependency-pathpoints at both. Caching the npm download cache does not helpinstall-depsat all.Decision (orchestrator), head
1ddca424.Round two: security PASS (comment 987), simplicity PASS (comment 988), test-quality BLOCK (comment 986). All three had held a block in round one, so all three re-reviewed against the current head; a stale PASS does not clear a merge.
Applied in
1ddca424:tests/e2e/login.spec.mjs:69-71,:140— liveness asserted by thecrashevent instead of counting pageerrors. Test renamed toreports an error and keeps the renderer alive(:128) so the name matches the assertion.tests/e2e/login.spec.mjs:79-88,:98— failure hook restructured: console attaches before the app guard,screenshot()wrapped,fs.rmSyncreachable on launch failure. Run 930 produced zero artifacts because the guard returned before both attaches.tests/e2e/login.spec.mjs:133— failed-login assertion accepts the rate-limited branch as well as the wrong-password one.playwright.config.mjs—trace: 'off'pinned explicitly rather than relying on the version default..gitea/workflows/tests.yml—--prefer-offlinedropped, run-number citation dropped, xvfb comment corrected.tests/e2e/login.spec.mjs— duplicate login-form assertion deleted.Contested, and how it resolved:
electron/main.js:18makesisDevtrue for an unpackagedelectron ., soopenDevTools()atmain.js:972fires on every run. The loop is the live path, not a race guard. Kept.install-depsnarrowing: simplicity conceded it. Playwright's chromium dep set covers its bundled browser, not Electron. Kept un-narrowed, with the reason in the comment.crashevent.['list']redundant as Playwright's default. It is not — an explicitreporterreplaces the default, so dropping it loses the CI console log. Kept.Skipped, stated: the DevTools window wait at
tests/e2e/login.spec.mjs:62-65has no timeout, so a DevTools-only registration hangs to the 120s test timeout instead of failing fast. The test timeout already bounds it; a race plus a throw for a path not observed in 4 local runs is more code than the failure is worth. Non-blocking, held by simplicity.Runner-side, not this PR:
container.daemon_socket: "-"on the runner, and a dedicated label for the host runner soubuntu-latestis unambiguous. Security holds that the workflow guard is self-attested — a fork authors the guard — so the runner config is the real boundary. Cost of the guard, fork PRs get no CI, is accepted.Merge gate: test-quality must re-review
1ddca424to lift its block. Checks:npm run test:unit2 passed,node --checkon both JS files. E2E cannot be run locally — noE2E_PASSWORDin this session — so CI on1ddca424is the check for the E2E changes.Source of truth for the change: issue #109.
Test-quality panel (re-review, head
1ddca424): APPROVE. No block. Two notes worth code, applied at87ba01fd.Per finding:
crashhandlerlogin.spec.mjs:69-72, assertion:141, name at:129matches what is provable. Butexpect(crashed).toBe(false)was a sync check evaluated at one instant, and any real crash makes the locator at:140throw first withTargetClosedError("Page crashed")(playwright-core_didCrash->Events.Page.Crash). It could only fail in the microsecond window between the two. Fixed: the assertion now runs before the form check, so it is the signal rather than a backstop.page.on('crash')does fire for Electron, same CDP path.beforeEachthrows, verified empirically, and the console attaches before the app guard withfs.rmSyncreachable. But.catch(() => null)means the crashed-page case still produces no screenshot, and on launch failurelogsis empty because no handler is attached before launch. Fixed by naming the paths rather than adding an artifact: thecrashhandler now logs the crash, and the comment states the launch-failure artifact is empty by construction, so the report's failure message is the diagnosis there.trace: 'off'pinned (playwright.config.mjs:17), rate-limit branch accepted (login.spec.mjs:134). CORS concern is moot: preflight tohttps://ruv.wtf/_matrix/client/v3/capabilitiesreturns 204 withaccess-control-allow-headers: ... Authorization. DevTools timeout: the author's call stands, 120s bounds it. What was worth the code was the diagnosis, not the timeout — the loop now records the window URLs (:63-64), so a hang names which windows existed instead of a bareTimeout 120000ms exceeded.webServer.timeoutskipped is fine; note a webServer failure produces no artifacts at all, since no hook runs.input.first()(:18),name="usernameInput"/"passwordInput",button[type="submit"](:21), shell-route regex (:105), localStorage keys (:108,:115, matchingcinny/src/app/state/sessions.ts:84,161).button[type="submit"]is unique on the login route today — onlyPasswordLoginForm.tsx:263has one,ServerPicker.tsxhas no form — so strict mode is safe now.Regex check:
rate-limited by serveralso appears inPasswordRegisterForm.tsx:380andTokenLogin.tsx:72; neither renders on this route, so no false match today. It does accept a run where the password was never rejected — the test proves "an error is displayed", not "the login path rejected these credentials". Honest given the comment.Deleted login-form assertion: criterion 4 is still proven — the error text plus
usernameInput()visible requires a live renderer. Weaker only in that the form is now one field, not four.Skipped, stated:
.catch(() => kill)at:95handles a rejectedclose(), not a hang. A hang still skipsfs.rmSyncand leaks the temp profile through the hook timeout. Bounding it needs a race, which is more code than a tmpdir leak is worth.Checks:
npm run test:unit— 2 passed. E2E not re-run locally, noE2E_PASSWORDin this session; CI on87ba01fdis the check.All three reviewers now hold no block.