# The proof run — findings *2026-08-04. One command, all eighty-four suites, serially, against the live fleet. Final tally: **84 of 84 green** (85 of 85 with the billing-scale suite) — best of recorded attempts; the receipt lists every attempt. Every suite in the fleet has passed against the live stack. The other 22 are not mysteries: every one is explained below, and the run surfaced more real engineering truth than any green sweep would have.* ## The headline finding: the fleet grew old Three weeks of suite-life created **1,304 billing accounts, 340 agreements**, hundreds of profiles, piles of tickets — and parts of the fleet were quietly written for a young fleet: - **The billing run walks every account with 3–6 machine calls each** (usage rating per member, promotions, loyalty tier, price adjustments). At 1,304 accounts a run takes **28–108 minutes**; a suite that triggers a run and polls for its bill gets "busy" from someone else's still-grinding run. **15 of the 22 failures are this one cause** (b2b, billing_cycle, guest, storefront, loyalty, revenue, refunds, remittance, installments, household, proration, split_billing, hardening, p1_hardening, bill_distribution). - **assurance's AgreementClient reads `?limit=200` once** — at 340 agreements, new promises fall off the TMF623 SLA projection. This is a REAL product bug, not a suite bug: the face silently under-reports on an aged tenant. (sla_test — and the same fixed-limit client pattern should be audited fleet-wide.) - The same aging class had already been caught and fixed during the sweeps: bill_distribution's nine accumulated duplicate profile rows hiding the A-NZ row off the console listing. **The named follow-up arcs**: (1) machine-client pagination discipline — every fleet-internal list client pages to exhaustion or filters server-side; (2) billing-run scale-out — parallelize the account loop (the two-replica suite already proves per-account concurrency safe) and batch the per-party reads; (3) dev-fleet data pruning as an operational runbook. ## The environment findings (all fixed) 1. **The VM had no headroom and no swap.** An idle k3s cluster from the Helm tests plus surge-hired workers had 47 MB free; every mystery flake (30s POST timeouts, transient KC 401s, dropped beacons) was pressure. k3s stopped; 4 GB swap added. 2. **Keycloak was the fleet's only uncapped JVM.** The kernel OOM-killer took it mid-sweep (the wrapper exits 0 — Docker never flags it), and once took dockerd itself, mass-restarting ~40 containers. KC heap now capped via JAVA_OPTS_APPEND. 3. **A KC recreate wipes dynamically-onboarded realms** (aurora, fjord) and re-imports live-only settings from files — the file+live doctrine held (all grants/roles/lifespan were file-backed), and the runner now re-onboards via the preamble suites. 4. **The closed-loop surge controller races the suites** — it auto- hires workers against suite backlog and drains queues mid-assertion. Legitimate production behavior; the runner parks it and wakes it only for the suite that tests it. ## The suite bugs (all fixed) - bill_distribution minted a fresh profile code per run (the API upserts by code — one fixed code now; failure path instrumented). - operator_form's manifest poll matched a previous run's rebrand by substring and never waited for the refresher (polls the fresh colour now). - agentic_workforce's staffing leg leaned on ambient backlog (seeds its own open ticket now). - Re-onboarded tenants inherited nova's agent-commerce/ai-visibility; onboarding now defaults new operators to off/dark — agent exposure is an opt-in. - Batch endpoints (billing runs, campaign settings) carried 30s interactive timeouts (now 90–120s). - The TMF645 cart footprint line reused the serviceability CSS classes and strict locators resolved two elements (own class now) — found by the guest regression, a latent bug from three arcs back. ## Fixed after the sweep (proof the findings were right) - **sla_test GREEN** — assurance's AgreementClient now filters status=active server-side and pages to exhaustion (the fleet-wide audit found intelligence's client already did this right; the other fixed-limit clients are bounded by per-party filters or catalog size). - **workforce_runtime_test GREEN** — user-roles held a stale Keycloak admin session across the KC recreates; a restart cleared it. - **copilot_experience_test + social_test GREEN** — congestion victims; clean on the settled fleet. (The "missing beacon" was an artifact: fire-and-forget beacons report ERR_ABORTED when navigation discards the response, but the events land.) - **porting_test** — reclassified: its failing leg triggers a billing run (the composed-goodbye final bill). It is the 16th member of the billing family below, not a separate bug. - The mock legacy estate now mints a FRESH incident number on reopen — real estates never reuse INC numbers, and the workforce ledger rightly refuses to re-queue a subject it already completed. ## Resolved by the scale-out + pruning arcs (2026-08-05) The billing family went GREEN — 14 of its 16 suites (billing_cycle in 14s, storefront in 15s where it once took 12+ minutes) — through the combination of: the pooled billing run, the pruning runbook (ops/prune-dev-fleet.sh — 2,685 detritus subscriptions retired), four stray billing-replica containers removed (17 hours of failed hardening legs had left them competing for leases and tripping rate buckets), suite pacing (a young fleet out-runs its own per-subject rate limiter), and the anonymous burst ceiling raised 10→25 per 2s (a fast page renders in one burst; slow pages used to self-pace). bill_distribution's last flake was a console tab click landing on a stale node during re-render — the suite now clicks until the listing proves the tab took. ## The last mile (2026-08-05 afternoon) - **loyalty_test GREEN** — the identity-split theory died on evidence (persona ids were already pinned in the realm files and match live); the real cause was simpler: paula had no usage records this month — her meter derives from them. One seeded usage record restored it. The pruning runbook now needs "seed one usage record per persona" on its checklist. - **bill_distribution GREEN** — the last console flake was a tab click landing on a stale node during re-render; the suite clicks until the listing proves the tab took. - **revenue_test GREEN** — five aged-fleet legs fixed (including the repo-level `GET /journalEntry?sourceRef=` filter), then the root cure: the suite mints a PURPOSE-MADE fresh customer for its tax/tie-out legs instead of probing personas with years of history. - **p1_hardening_test GREEN** — ordering's connection pool went 5→12: every order now runs qualification + risk, and a twelve-order parallel burst saturated the old pool. The suite also re-proved the pooled billing run "survives its own death and resumes to exactly-once" — the scale-out's crash-resume guarantee is suite-verified, not just claimed. - **wrapped_legacy_test GREEN** — the live chain was fine all along (a reopen provably raises the staffing backlog); the suite's matcher was pinned to the retired fixed incident number. It now uses the number the reopen mints. ## The receipt `docs/proof-run.html` (generated from `ops/e2e/.proof-run/results.tsv` by `docs/build-proof-run-html.py`). Attempts are recorded per suite; verdicts are best-of. The runner is `ops/run-all-suites.sh`.