# Production hardening — the runbook

*What the P0 arc built and proved on this stack, and what a real deployment
must add that a laptop cannot prove. Companion to
[p0-hardening-plan.md](p0-hardening-plan.md); suite #56
(`ops/e2e/hardening_test.js`) is the proof of the provable half.*

## What is DONE and proven here

### One replica speaks at a time
Every mutating scheduled tick — dunning, bill distribution, campaign
journeys, churn sweeps, porting cutover, commission hardening, order
resume, telesales expiry, cart abandonment — claims a row-lease in its
service's own database before running (`tick_lock` table + `TickGuard`,
the ShedLock algorithm hand-rolled, no new dependency). Scale any of the
six services to N replicas and the ticks still fire once. A crashed
holder's lease expires on its own; nothing wedges. The 27 outbox relays
are deliberately unguarded: at-least-once delivery with eventId dedup at
every consumer is the contract — a duplicate relay is waste, never harm.

**Proven**: two billing replicas against one database, one bill cut, the
mock print house received exactly one copy.

### Rate limits, two rings
The dealer surface keeps its strict per-partner buckets. Everything else
now passes a fleet-wide ceiling — per subject for people, per client for
machines, per IP for anonymous knocks — decode-only, fairness not authz.
Dials: `GLOBAL_RATE_CAPACITY` (default 1200/window) and
`GLOBAL_RATE_WINDOW_MS` (default 60s).

**Scale-out (closed in P1)**: the buckets now live behind a
`RateLimitStore` seam — in-memory with no configuration, shared Redis
buckets when `BSS_GATEWAY_REDIS_URL` is set (the compose default). N
replicas enforce ONE exact ceiling, a restarted gateway still refuses
inside the same window (suite #57 proves the resurrection), and an
unreachable Redis fails OPEN — rate limiting is fairness, never
authorization.

### Backups with a tested restore
`ops/backup.sh` dumps every database and role; `ops/restore-drill.sh`
restores into a throwaway container and verifies databases, rows, and an
optional sentinel. **Run the drill on a schedule, not once** — a backup
that hasn't restored lately is decaying back into a hope. In production,
prefer the managed database's PITR (below) and still run the drill
against it: the drill tests the *procedure*, not just the file.

### The secret gate
`ops/scan-secrets.sh` refuses commits whose staged diff carries a
real-secret shape (Anthropic/OpenAI/AWS/GitHub/Slack keys, private-key
blocks); `ops/install-hooks.sh` wires it as pre-commit. The dev fixtures
in compose and realm files ("billing-secret", admin/admin, partner
tokens) are deliberate stand-ins — every one of them is an env seam
(`.env.example` lists them) and none may carry a real value in git.

### The anonymous surface, including what Spring answers by itself
Every component validates its own tokens, and everything a component
answers without one is declared in `ops/security/public-paths.txt`.
`ops/security/security_config_drift.py` (run by the claims gate) reads the
`permitAll` lines of every `SecurityConfig` and fails when one opens a path
that file does not declare.

Reading `permitAll` lines misses the paths Spring Security answers from its
own filters, before authorization runs (#428). Measured on 3 Nov 2026, in a
test and on every running component of the laptop fleet, two such paths are
answered without a token by **every** component:

| Path | Who answers | What comes back |
|---|---|---|
| `/.well-known/oauth-protected-resource/**` (GET) | the RFC 9728 metadata filter, which `oauth2ResourceServer()` switches on in Spring Security 7, with no setting to switch it off | 200 and a small JSON document. Only the ontology fills it in (the tenant's issuer, for MCP clients), and only the ontology's copy is routed by the gateway |
| `/logout` (any method) | the logout filter every `HttpSecurity` carries | 302 to a `/login` page that does not exist. Sessions are stateless, so there is nothing to log out of |

Both are now declared as `framework:` entries, 2 per component and 80 in
total. Two checks keep them honest:

- **`FrameworkSurfaceTest`** (agreement, and a copy in the ontology for its
  customised resource server) starts the component's real filter chain. It
  fails when a filter appears that nobody has measured, which is how a Spring
  upgrade that adds one shows up. It also requests the declared paths plus a
  list of the paths Spring answers when other features are on (form login,
  one-time tokens, WebAuthn, OAuth2 login, SAML, the authorization server,
  the error page), and fails unless exactly the declared ones answer without
  a token.
- **The drift check** holds every other component to the agreement's
  `framework:` entries. It fails when a component configures anything outside
  the shape the test measured (`formLogin`, `httpBasic`, `logout`, a custom
  filter, a second filter chain and so on), because the test's result no
  longer applies to that component.

**Seen red:** taking `billing framework:/logout` out of the file fails the
drift check. Adding `.formLogin()` to the agreement's config fails the drift
check and both tests: four unmeasured filters, and `/login` and
`/default-ui.css` answering. Declaring a path that does not answer fails the
test too.

**Honest limits.**
- Only two of the 40 components are measured directly. The other 38 are
  covered by inference: same configurer shape, so same filters. The live probe
  of 3 Nov reached the 32 components running on the laptop at the time. It is
  not a committed check.
- The probe can only find paths it is told to request. A new filter is caught
  by the filter count, but a filter we already know about that starts
  answering a new path would only be caught if that path is on the canary
  list.
- MockMvc does not run the servlet container's error dispatch. `/error`
  measured 401 on real Tomcat in the live probe, but the test does not prove
  that.
- Servlet filters outside Spring Security, and actuator endpoints beyond the
  ones declared, are outside this check.
- Every component except the ontology serves Spring's default document. It
  names the component's internal URL, names no authorization server, and says
  `tls_client_certificate_bound_access_tokens: true`, which nothing in this
  fleet uses (TLS ends before the components). It is not reachable through the
  gateway (only the ontology's is), but it is not accurate either. Filling it
  in or removing it is follow-up work, not part of this change.

## What a REAL deployment must add (a laptop cannot prove these)

### Secrets
- All `.env.example` names become Kubernetes Secrets — External Secrets
  Operator or the CSI driver pulling from AWS Secrets Manager / Azure
  Key Vault; the Helm chart's env-driven values take them as-is.
- Keycloak: dev realm JSONs are fixtures. Per environment, mint realms
  and client secrets with `ops/onboard-tenant.sh` against a hardened
  Keycloak (or a managed IdP) — never import the repo realms.
- The PUK vault key (AES-GCM) moves to KMS; rotate on schedule.

### TLS in transit
Edge TLS terminates at the Ingress (cert-manager + Let's Encrypt or the
cloud LB). Inside the cluster, prefer a mesh (Istio/Linkerd) for mTLS
between services — zero code change, the services already speak plain
HTTP behind the gateway. The alternative (Spring-native TLS per service)
works but costs a cert-distribution story the mesh gives you free.

### Who may rewrite a customer's URLs (trusted proxies)

The gateway honours `X-Forwarded-*` only from sources it trusts, and services
rebuild client-visible links — TMF `Location` headers among them — from those
headers. So the trusted-proxy list decides who can tell a customer's browser
where to go next.

The application default is every RFC1918 range
(`10.*`, `172.*`, `192.168.*`, `127.*`). That is right for a laptop fleet on a
Docker bridge and **wrong for a cluster**: inside a flat network it trusts
every workload, not only the ingress.

Set `GATEWAY_TRUSTED_PROXIES` to the range your ingress controller, load
balancer or service mesh actually occupies. The Helm chart ships `10\..*`,
which covers the pod network of a default EKS/AKS/k3s install — **an operator
on a different pod CIDR must change it**, and one terminating TLS at an
external LB should narrow it to that LB's addresses. Strip inbound forwarded
headers at the outer edge so only the edge can set them.

### HA databases and brokers
- **Postgres**: one managed HA instance (RDS/Aurora Multi-AZ, Azure
  Flexible Server zone-redundant) — image parity matters only for
  pgvector (both clouds support it). Turn on PITR. Point every
  service's `DB_URL` at it; the per-service databases and RLS roles
  migrate themselves (Flyway runs as the owner on boot).
- **Kafka**: MSK / managed Kafka, 3 brokers, `replication.factor=3`,
  `min.insync.replicas=2`. The outbox pattern already tolerates broker
  failover — that is why it exists.
- **Keycloak**: 2+ replicas behind the LB with its own HA Postgres.

### Scale-out, in order
1. Stateless services: replicas ≥2 via the Helm values — safe because of
   the tick locks (P0's point).
2. Gateway replicas ≥2 — buckets shared in Redis since P1, ceilings stay
   exact.
3. The billing run partitions by ACCOUNT since P1: each bill commits
   alone, a crashed run resumes by construction (the bill is the
   checkpoint), and every run is a ledger row (`GET /billingRun`). One
   run speaks at a time — a concurrent trigger gets `{"busy": true}`
   (row-lease with per-account heartbeat; a crash frees it in ~2 min)
   and a unique index makes a double bill impossible at the database.
   What remains for true telco scale is running partitions CONCURRENTLY
   (worker pools over account ranges) — the per-account isolation and
   the constraint this arc built are the prerequisites.
4. Throughput has numbers now — `ops/load/loadtest.js`, baselines with
   stated caveats in [perf-baselines.md](perf-baselines.md), and suite
   #57's smoke SLO as the regression tripwire. Re-baseline on real
   hardware before capacity planning.

### Alerts (P1) and what still pages nobody
`infra/prometheus/alert-rules.yml` evaluates ServiceDown, outbox publish
failures and gateway 5xx over the whole fleet (32 scrape targets).
Prometheus FIRING is not a page: a deployment adds Alertmanager and a
route to a pager/Slack — until then alerts are visible at :9090/alerts.

### Compliance (P2) — built and proven where a laptop can
GDPR is features now, not intentions — see [privacy.md](privacy.md):
the data passport (`GET /privacy/v1/export`, the caller's own token
doing all the reading), erasure with the law's own exceptions (active
contracts refuse 409; bookkeeping categories retained WITH their basis;
profile anonymized in place; login scrubbed at the IdP; immutable
audit rows), and retention as TickGuard-guarded clocks. Lawful
intercept is stated as a boundary: the BSS half (warrant-gated
disclosure) is shaped, not built. Suite #58.

### PCI-DSS scope — PAN-free by construction

An external PSP **reduces** PCI-DSS scope; it never removes the
obligation — the deploying operator is the merchant and must attest
every year regardless. What the architecture buys them is the lightest
possible tier, and the claim has a receipt in the schema:

- **No PAN, anywhere.** The `payment-method` vault stores
  `brand`, `lastFour`, `expiry`, `pspToken` — there is no card-number
  column in any migration, so full card data neither transits nor
  rests on BSS servers, in logs, or in backups. The payment service
  talks to the PSP by token (Stripe PaymentIntents-shaped adapter;
  mock by default).
- **SAQ tier by capture style.** PSP-hosted capture (redirect or
  PSP-hosted fields) → **SAQ A**. Embedding the PSP's card elements in
  our own storefront page → **SAQ A-EP** (our JavaScript can influence
  the form, so the page is in scope). Nothing in this BSS requires the
  SAQ D that storing or transmitting PAN would trigger.
- **PCI DSS 4.0.1 note (current since 31 Mar 2025):** SAQ A merchants
  no longer answer 6.4.3/11.6.1 line-items but MUST attest their
  e-commerce site "is not susceptible to attacks from scripts" — in
  practice, script inventory/integrity on the pages that lead to
  payment (CSP headers, SRI, or a tamper-watch) is still the evidence
  a QSA asks for. SAQ A-EP and SAQ D keep 6.4.3/11.6.1 in full.
- **The MOTO trap, designed out.** The CSR console has NO card-entry
  field, deliberately: an agent taking card numbers by phone drags the
  whole call center into scope. If assisted payment is ever needed,
  the pattern is pay-by-link (the PSP's secure page in the customer's
  own hands), never read-your-card-to-the-agent.

The honest claim mirrors the PQC one: not "PCI-certified" (that is
per-deployment, per-merchant, attested by a QSA) but **"PAN-free by
construction — SAQ A / A-EP scope, receipts in the repo."**

### Still open — the honest remainder

*Every row below, and every "must add" above, is an owned line in
[production-acceptance.md](production-acceptance.md) — the checklist a launch
is signed off against.*
| | |
|---|---|
| P1.5 | ~~live multi-replica k8s soak~~ **DONE 2026-07-21** — the Helm chart ran on live k3s: 21 pods Ready, billing at 2 replicas with ONE set of tick leases, requests served through the in-cluster gateway ([k8s-soak-plan.md](k8s-soak-plan.md) has the receipts and the three drifts the soak caught); OpenTelemetry tracing (agent + collector cost RAM the demo VM lacks); concurrent billing partitions; Alertmanager routing |
| P2.5 | third-party penetration test; regional DR drill on real topology; Art. 30 processing register + DPIA (operator documents); per-tenant retention dials in the registry; warrant-disclosure endpoint when a national format names it |