Operational Runbooks

Step-by-step procedures for operating the deployed surfaces. Companion to the Release Process (how to ship) — this page covers verifying, recovering, and responding when something goes wrong. Deployment authority and the edge-function operational model are defined in docs/adr/ADR-005 and docs/adr/ADR-008.

Account isolation: every command here targets the Newmatik org only (Cloudflare, Supabase, Sentry). Confirm the account before acting.

Post-deploy verification

Run immediately after a web deploy to main:

pnpm run release:verify

One command (scripts/release-verify.mjs), fails closed, one line per check (OK, FAIL, SKIP). It runs, in order:

1. Synthetic probe — the curated public routes return 2xx/3xx (the same check as pnpm run synthetic:check).
2. Deployed build — the Nuxt app manifest (/_nuxt/builds/latest.json and /_nuxt/builds/meta/<id>.json, the files stale tabs poll to detect a new deploy) is reachable and well-formed. The SPA is client-rendered, so there is no server-rendered footer to read a version from; the build id is the deploy identity. Workers Builds sets WORKERS_CI_COMMIT_SHA and nuxt.config.ts uses it as the build id, so set RELEASE_BUILD_ID=<commit sha> (short SHAs work) and the check fails unless that commit is live. Unset, the comparison is reported as SKIP.
3. Docs (D1) — /docs and one deep page (default /docs/development/runbooks, override with RELEASE_VERIFY_DOCS_PATH) return 200 with the SPA shell, and the Nuxt Content query route (/__nuxt_content/docs/query, backed by D1 in production) returns each page's row with its title. The first query after a deploy is what imports dump.docs.sql into D1, so a FAIL here means the docs migration did not run or the D1 binding is wrong.
4. Sentry release — the same check as pnpm run sentry:verify; SKIP with a notice unless SENTRY_AUTH_TOKEN and SENTRY_ORG are set.
5. Backend — with SUPABASE_ACCESS_TOKEN, verify Newmatik ownership, migration versions, function/column privileges, required secret metadata, gateway JWT settings, and downloaded function sources including shared dependencies. This also detects retired endpoints and probes invitation CORS without sending email. Set RELEASE_REQUIRE_BACKEND=1 for a production gate that fails if credentials are missing. Run pnpm run backend:verify for the backend gate alone.

Target another host with RELEASE_VERIFY_BASE_URL=https://staging.gerbtrace.com pnpm run release:verify.

If the build check still reports the previous build id after a deploy, purge cache (Cloudflare → Caching → Configuration → Purge Everything) and re-run.

Web app rollback (Cloudflare Workers)

The web app deploys via Cloudflare Workers Builds (ADR-005). To roll back:

1. Open Cloudflare → Workers & Pages → gerbtrace → Deployments.
2. Identify the last known-good deployment (match the commit SHA / version).
3. Use Rollback to promote it, or re-run the build for the good commit.
4. Re-run pnpm run release:verify with RELEASE_BUILD_ID set to the known-good commit so the build check proves the rollback is live.
5. If the bad build shipped a D1 docs migration, the docs check in release:verify covers it; the docs table is rebuilt from dump.docs.sql on each deploy, so rolling the Worker back also restores the previous docs snapshot.

Never hand-edit production Worker code outside the build pipeline.

Supabase incident / migration recovery

Migrations are applied to Supabase Cloud (ADR-001/003). If a migration causes an incident:

1. Stop further deploys.
2. Diagnose with get_logs / get_advisors (Supabase MCP) before changing anything.
3. Because migrations are idempotent and additive, prefer a forward fix: a new idempotent migration that corrects the schema, rather than a destructive rollback.
4. For a bad column default or policy, ship a corrective CREATE OR REPLACE / ALTER ... IF EXISTS migration and re-run pnpm run migrations:check.
5. Treat production data as off-limits for destructive statements without explicit approval.

Edge Function rollback

Edge Functions share auth and timeout helpers (supabase/functions/_shared/, ADR-008) and return the real Supabase error message.

1. Redeploy the previous known-good function from the last good commit (Supabase dashboard or CLI).
2. Confirm the function returns the expected error surface (not a hardcoded string) via a probe request.
3. Check get_logs for the function to confirm the error rate returns to baseline.

Incident response (quick path)

1. Confirm scope — is it web, desktop, API, or docs? Use the synthetic check and Sentry to localize.
2. Filter noise — in Sentry, split production (www.gerbtrace.com) from localhost/feature-branch noise via the url tag; ignore the known bot/old-browser patterns.
3. Mitigate — roll back the affected surface (sections above).
4. Verify — re-run pnpm run release:verify (or synthetic:check for a quick route probe); watch Sentry error rate.
5. Follow up — add a Vitest regression first when possible, then a browser smoke check only if the failure needs the browser (ADR-007).

Synthetic checks

• Manual: pnpm run synthetic:check.
• Custom target: SYNTHETIC_BASE_URL=https://staging.gerbtrace.com pnpm run synthetic:check.
• Scheduled: .github/workflows/synthetic.yml runs the check on a cron and can be triggered manually from the Actions tab.

Sentry release verification

• pnpm run release:verify runs this check as its last step; run pnpm run sentry:verify on its own with SENTRY_AUTH_TOKEN and SENTRY_ORG set (optionally SENTRY_RELEASE, else the package.json version).
• The build plugin uploads debug-ID artifact bundles, not legacy release files. The check loads the deployed site (RELEASE_VERIFY_BASE_URL, default production), reads the Sentry debug ID injected into each entry chunk, and asks Sentry's artifact lookup for it (SENTRY_PROJECT, default gerbtrace). It then reads the matching bundle's manifest, because a bundle can hold the minified file without its source map.
• It fails if the release is missing, a checked chunk carries no debug ID, or any checked chunk has no uploaded source map, meaning production stack traces would not symbolicate. The upload runs only in Workers Builds (CI set) and needs the SENTRY_AUTH_TOKEN build secret on the Deploy default branch trigger (Workers & Pages → gerbtrace → Settings → Build → Variables). Use a Newmatik org auth token (org:ci scope), then redeploy. The build log line No auth token provided. Will not upload source maps means it is missing.

Elexess integration

BOM pricing searches go through main/elexess-search to the Elexess REST API (https://api.elexess.com/functions/v1, override with the ELEXESS_URL Edge Function secret). It authenticates with an Elexess team API token (el_live_...). The platform-wide token is set in Admin → Platform Settings, and a team-specific override in Admin → Teams → Integrations. Both are stored service-role only (platform_config, team_integration_secrets) and shown masked to their first 12 characters.

• The production platform token is gerbtrace-production-donotdelete on the Newmatik Elexess team (Elexess → Dashboard → API Tokens). Revoking it stops all Gerbtrace pricing searches.
• Test Connection calls the quota-free /suppliers endpoint. Each search spends one credit of the token's Elexess team.
• The Edge Function maps REST results (stock, lead_time, prices) onto the current_stock / current_leadtime / pricebreaks shape stored in project pricing caches, so existing caches stay readable.
• Errors: a rejected token returns 503 ("Contact your administrator"), an exhausted Elexess quota returns 429, and an outage across all suppliers returns 503.
• The retired Basic-auth API (api.dev.elexess.com) no longer resolves. The one team's migrated username/password were deleted on 2026-09-30. Setting or clearing a team token in the admin UI also removes such values.
• Rollout order for 20260928150000_team_integration_secrets.sql: apply the migration first, then immediately deploy main, admin-team-action and admin-usage. Between the two steps, older functions see no team keys: Spark runs fall back to the platform key and admin pages show "Global". New functions deployed before the migration fail closed.

Security and billing remediation rollout

These changes require a coordinated backend deployment before the frontend. Do not roll the database permissions back to restore an older app. Use forward migrations for database fixes.

1. In an isolated local Supabase project, apply all migrations and run supabase test db and pnpm run test:account:local. The account integration runner accepts only local API/database URLs, creates disposable users/team/file fixtures, verifies preserved bytes, and cleans up its fixtures. It supplies the local service-role table grants missing from recent CLI bootstrap images. Replay the three 20260928 remediation migrations in timestamp order and rerun the tests. Run typecheck, unit tests, Deno checks/tests, browser smoke, and a Worker build.
2. Prepare a dedicated Gerbtrace portal configuration in Stripe test mode first, then in production during the approved rollout. Enable subscription updates with exactly the Pro and Team prices, proration_behavior=always_invoice, and billing_cycle_anchor=unchanged. Enable cancellation with mode=at_period_end. Enable payment-method updates, invoice history, and customer name/email/address/tax-ID updates. Ensure downgrade scheduling matches the published billing terms. Set its ID as the Edge Function secret STRIPE_PORTAL_CONFIGURATION_ID. Never change the shared default portal for the other products on this Stripe account.
3. Apply 20260928130000_harden_team_and_account_access.sql, 20260928131000_atomic_billing_reconciliation.sql, and 20260928132000_fix_checkout_reservations.sql in timestamp order. ACL/ownership failures abort the migration: have the function owner apply the ACL changes rather than skipping them. Regenerate database types from the updated schema. The first migration seeds no memberships for unverified addresses; the billing migration preserves existing granted tiers as a baseline. Review previously suspicious memberships, customer associations, and duplicate subscriptions separately before authorizing any cleanup.
4. Deploy stripe-checkout, stripe-portal, stripe-webhook, send-invitation, delete-account, and admin-team-action from this commit, using the gateway settings in supabase/config.toml. Deploy main to remove its old duplicate handlers. Redeploy any other function whose downloaded dependency differs from git. The retired handle-team-join endpoint was deleted on 2026-09-28 after 30 days of edge logs showed no callers (ADR-001). supabase/config.toml declares verify_jwt for every function; the backend gate fails on an undeclared or differing gateway setting, and on any deployed function without source in git. These operations require the separately approved production rollout.
5. Merge/push the frontend changes for Cloudflare Workers Builds. Desktop builds use the same plan enum requests and no longer depend on public Stripe price IDs.
6. Run RELEASE_REQUIRE_BACKEND=1 RELEASE_BUILD_ID=<commit> pnpm run release:verify with Newmatik credentials. Inspect Stripe webhook retries, function errors, and Sentry after deployment. Do not use a live card payment, send a real invitation/feedback message, or delete a production account as a verification probe.

Stripe reconciliation reads the latest customer subscription list, filters to Gerbtrace prices, and commits subscriptions, the granted plan, billing details, and the event receipt in one transaction. A generation check rejects overlapping stale snapshots so Stripe retries. Failed renewals retain the last granted tier; unpaid upgrades cannot grant a higher tier. Foreign prices are never persisted as Gerbtrace prices. A team deletion records its intent and blocks new checkout reservations before canceling all associated Gerbtrace subscriptions; a pending checkout lease delays deletion for up to 45 minutes. Failed cancellations or team deletes preserve the team and release the deletion marker for retry. Successful deletions retain their tombstone. Only new Checkout sessions acquire a lease. A fresh lease gives session creation five minutes of headroom above Stripe's minimum lifetime; late retries recover an existing session by its request metadata or return a conflict until the lease expires without changing the idempotent payload.

For support verification, use mocked SDK transport acceptance/rejection and Help Scout identity/consent tests locally. In a separately authorized staging check, confirm an actual feedback item and support conversation reach the Newmatik destinations. A Sentry event ID alone is not a delivery acknowledgement. Bug report drafts survive unconfirmed delivery; the Contact Support menu falls back to software@newmatik.com if Beacon fails to load.

Project access and enrolled-MFA rollout (2026-10-01)

This is a release procedure, not an automatic migration step. Confirm the Newmatik Supabase organization/project and Cloudflare account before invoking external services. The implementation has only been exercised against an isolated local database; production remains unchanged.

1. Check existing authenticator role configuration for a pgrst.db_pre_request hook. The MFA migration deliberately refuses to replace an unrelated hook. Compose the existing hook with public.require_mfa_assurance in a reviewed migration if necessary.
2. Apply 20261001090000_align_space_and_bom_access.sql, then 20261001091000_enforce_enrolled_mfa.sql. Replay both on staging and run the database tests. Subsequent migrations that add RLS tables must also add the restrictive enrolled_mfa policy; the database regression detects omissions.
3. Redeploy all Edge Functions from the approved commit, including mcp, main, send-invitation, delete-account, the admin functions and Stripe functions, because their shared authorization dependency changed. Use the gateway configuration in supabase/config.toml; MCP remains verify_jwt = false with caller-token authorization inside the handler.
4. Verify unenrolled AAL1 sign-in still works. For an enrolled test account, verify AAL1 cannot read/update account data or invoke privileged functions, then challenge to AAL2 and verify access resumes. Exercise recovery, factor removal and an OAuth/MCP return path. Do not alter production users for testing. An enrolled caller whose OAuth token is AAL1 must establish an AAL2 session before MCP tools work.
5. Verify editor/viewer/guest storage and BOM access, including assigned and unassigned spaces. Inactive space members must lose access; storage access uses the actual team/project path and project authorization.
6. Deploy the Worker and run the post-deploy gate for the exact commit, docs D1 delivery and Sentry source maps. Desktop releases run the complete shared CI gate before signing. Keep deployment rollback tied to the approved previous frontend/function release; removing the MFA gate is a separate security decision, not an automatic incident workaround.

Persistence upgrade

The outbox database gains account/project keys. Legacy rows stay in the original table as an unclaimed quarantine; they are never replayed by the next account. Do not delete or assign these rows automatically. If a user reports legacy unsaved work, recover it only after verifying ownership. New storage failures remain visible and keep unsaved edits in memory; users should keep the project open while retrying.

Public HTML and image delivery

pnpm run build prerenders the public inventory and retains the Worker/D1 architecture. Run node scripts/check-public-html.mjs after the build. Verify an actual public document without JavaScript, and confirm application routes still require sign-in. pnpm run generate sets NUXT_DESKTOP=1 and produces the desktop SPA. Keep the /dump.docs.sql redirect in both configurations. Regenerate documentation WebP variants with pnpm run docs:images after changing source screenshots; the full PNG remains the enlarged view.

Machine-program acceptance

The deterministic JPSys tests and reference snapshot are necessary release checks. Before shipping the machine-program changes, validate the supplied reference export in JPSys on the target machine profile and perform the normal simulation/dry-run procedure. Record the software/machine profile and expected orientation, fiducials, volume and dot path. This local implementation has not run a physical machine or a production printing job.