Keeper, operations, and recovery runbook

The foreground supervisor admits Source A before importing repository code, seals the exact Python and Node production closure, pins Git, Python, Node, Cast and the executable tree, and gives each child shared public runtime configuration plus only that role’s declared credentials. The upstream RPC credential remains in an in-process relay; children receive a short-lived loopback capability and never the raw URL. It strips operator credentials and every other role’s private keys. A process being alive is not proof that its sealed runtime, lease, database, external infrastructure, outbox or finalized cursor is healthy.

Use this gate only with a fresh fork-pinned Anvil on loopback, chain ID 46630, and clean committed Source A. The launcher validates Source-A-approved Forge and Cast before constructing its RPC client, filters every child to catalog-declared protocol variables plus fixed HOME/PATH/TMPDIR/LANG/LC_ALL, and has no public-Testnet authorization flag.

Source A fixes the rehearsal origin to Robinhood Testnet block 93015525 and its reviewed block/genesis hashes. Every new Source A re-proves that exact historical identity and finalized-height coverage through its own canonical Tenderly archive environment; it never copies an older release’s env or fork-origin artifact.

FLIP_V3_SOURCE_A_COMMIT="$(/usr/bin/git rev-parse HEAD)"
FLIP_V3_RELEASE_ID="$(/usr/bin/printf '%.12s' "$FLIP_V3_SOURCE_A_COMMIT")"
REHEARSAL_ARCHIVE_ENV="$HOME/.config/flip/46630/$FLIP_V3_RELEASE_ID/rehearsal-archive.env"

/bin/sh infra/generate-testnet-rehearsal-archive-env.sh \
  --source "$HOME/.config/flip/46630/$FLIP_V3_RELEASE_ID/release.env" \
  --output "$REHEARSAL_ARCHIVE_ENV"

Use two terminals. The isolated operator reads the archive endpoint as inert data only from the canonical, release-render-sealed derivative, keeps it behind an in-process loopback relay, and gives Anvil only the rbh_testnet alias. The raw URL is never placed in Cast or Anvil argv, printed, or copied into an artifact.

FLIP_V3_SOURCE_A_COMMIT="$(/usr/bin/git rev-parse HEAD)"
FLIP_V3_RELEASE_ID="$(/usr/bin/printf '%.12s' "$FLIP_V3_SOURCE_A_COMMIT")"
REHEARSAL_ARCHIVE_ENV="$HOME/.config/flip/46630/$FLIP_V3_RELEASE_ID/rehearsal-archive.env"

/usr/bin/python3 -I -S infra/protocol_v3_operator.py rehearsal-anvil \
  --repository-root "$PWD" \
  --operator-env "$REHEARSAL_ARCHIVE_ENV" \
  --port 8547

FLIP_V3_SOURCE_A_COMMIT="$(/usr/bin/git rev-parse HEAD)"
FLIP_V3_RELEASE_ID="$(/usr/bin/printf '%.12s' "$FLIP_V3_SOURCE_A_COMMIT")"
REHEARSAL_ARCHIVE_ENV="$HOME/.config/flip/46630/$FLIP_V3_RELEASE_ID/rehearsal-archive.env"

/usr/bin/python3 -I -S infra/protocol_v3_operator.py rehearsal-run \
  --repository-root "$PWD" \
  --operator-env "$REHEARSAL_ARCHIVE_ENV" \
  --rpc-url http://127.0.0.1:8547 \
  --forge-bin "$HOME/.foundry/bin/forge" \
  --cast-bin "$HOME/.foundry/bin/cast"

• Expected outputs are fresh under deployments/46630/<release-id>/rehearsal/: RPC and toolchain bindings, inherited Provider 4/BHS/Permit2 snapshots, signer setup, dry-run/mined journals, manifest, rehearsal policy, fork trace canary, acceptance config and plan.
• The private operator and role inputs must be ordinary non-symlink mode-0600 files outside the repository. The tracked .env.v3.46630.example is a blank template only.
• Keep Terminal A dedicated to this one Anvil. Do not mine, mutate, or send a transaction before Terminal B starts the launcher; the launcher requires the local head to remain Source A’s pinned historical block. Stop Anvil after the run and restart only from the same re-proved Source-A origin.
• Never enable shell tracing, echo RBH_TESTNET_RPC_URL, replace rbh_testnet with the expanded environment value, or attach the private environment to support evidence. Process argv and artifacts must contain only the alias or loopback URL.
• Rehearsal artifacts are local-fault-injection evidence. They do not authorize a public Bet, VRF request, funding transfer, accepted report or Evidence B claim.

Every public mutation is a Source-A catalog action. The isolated Operator checks the clean Git identity and external mode-0600 release environment, then delegates only to the fixed action runner. That runner seals the exact environment and toolchain, executes the reviewed dry-run through a credential-safe relay, creates an immutable intent, claims one broadcast attempt, and verifies canonical receipts. Private keys and the upstream RPC URL are never command-line arguments.

SOURCE_A=$(git rev-parse HEAD)
RELEASE_DIRECTORY="$PWD/deployments/46630/${SOURCE_A:0:12}"

/usr/bin/python3 -I -S infra/protocol_v3_operator.py release-action \
  --repository-root "$PWD" execute \
  --release-directory "$RELEASE_DIRECTORY" \
  --action <catalog-action-id> \
  --stage-overlay <verified-prior-stage-overlay.json> \
  --allow-public-testnet

/usr/bin/python3 -I -S infra/protocol_v3_operator.py release-action \
  --repository-root "$PWD" prepare \
  --release-directory "$RELEASE_DIRECTORY" \
  --action <catalog-action-id> \
  --stage-overlay <verified-prior-stage-overlay.json>

# After independent review of the sealed dry-run and intent:
/usr/bin/python3 -I -S infra/protocol_v3_operator.py release-action \
  --repository-root "$PWD" broadcast \
  --release-directory "$RELEASE_DIRECTORY" \
  --action <catalog-action-id> \
  --allow-public-testnet

Release-scoped FLIP application PostgreSQL
  ├─ indexer / API
  ├─ coin-settler
  ├─ jit + jit-quoter
  ├─ holder
  ├─ parity + holding-reporter
  └─ attestor

Independent Chainlink PostgreSQL
  └─ Chainlink Node only

The application database uses a release-owned PostgreSQL 16 volume, loopback port, generated 96-hex-character password and canonical external runtime at ~/.config/flip/46630/<SourceA12>/<release|rehearsal>/app-postgres. The directory is exact mode 0700; its three private files are exact mode 0600, owner-owned, regular, non-symlink and single-link. The generator, stable-descriptor reads, container labels, pinned image, primary-state check and indexer_runtime_identity bind it to one Source A and manifest. Chainlink’s database, the legacy Beta Compose database, an operator’s default DATABASE_URL, and a Git-ignored repository directory are not substitutes.

1. 01

   Freeze Source A

   Require clean tracked state, exact HEAD/tree identity and the canonical deployments/46630/<release>/release/manifest.json. Do not start a release service from a mutable worktree.

2. 02

   Generate the application DB runtime once

   Run the generator against that manifest and the exact external ~/.config/flip/46630/<SourceA12>/<scope>/app-postgres directory. It refuses repository-local or noncanonical paths, an existing credential set, unsafe permissions/links, dirty source, wrong scope, wrong chain or wrong commit.

3. 03

   Start and claim PostgreSQL

   Start the pinned application container, wait for health, build the exact Source-A Indexer, run idempotent migrations and bind indexer_runtime_identity. Then run the independent database check.

4. 04

   Let the isolated Operator load configuration

   Never source either private file. protocol_v3_operator.py parses the external release-scoped release.env and canonical generated application.env as inert data, lets the release database replace any legacy/default DATABASE_URL, verifies every direct owner-controlled ancestor and gives each child only its declared values. Both files must be ordinary exact-mode-0600 files; repository dotenv and Git-ignored repository credentials are prohibited.

5. 05

   Check before run

   Run the supervisor with /usr/bin/python3 -I -S. Before any repository import it verifies protocol-v3 mode, chain 46630, mainnet-ineligible manifest, Source A/tree, clean worktree, external-health policy and each selected role credential; isolated mode blocks user site code and PYTHONPATH injection.

6. 06

   Seal and start

   The supervisor creates a release-scoped content-addressed runtime containing the built Indexer, its production dependency closure and fixed Python zipapps, then pins every tool/file inode, link count, mode and SHA. It starts children directly from that seal—never through npm or a mutable worktree.

7. 07

   Run and wait for readiness

   Keep the foreground supervisor, its capability-scoped RPC relay and external-health sidecar alive. Do not admit users or an acceptance action until readiness is green with the same release, manifest, runtime-seal, lease and finalized identity.

8. 08

   Stop without deleting state

   Stop the supervisor/service manager first. Then use the scoped app-postgres stop command. Do not stop unrelated Chainlink infrastructure and do not delete either volume.

SOURCE_A=$(git rev-parse HEAD)
RELEASE_ID=${SOURCE_A:0:12}
MANIFEST="$PWD/deployments/46630/$RELEASE_ID/release/manifest.json"
APP_DB_RUNTIME="$HOME/.config/flip/46630/$RELEASE_ID/release/app-postgres"
APP_ENV="$APP_DB_RUNTIME/application.env"
SUPERVISOR_STATUS="$PWD/deployments/46630/$RELEASE_ID/release/runtime/live/protocol-supervisor-status.json"

python3 infra/protocol_v3_app_postgres.py generate --manifest "$MANIFEST" --runtime-dir "$APP_DB_RUNTIME"
python3 infra/protocol_v3_app_postgres.py start    --manifest "$MANIFEST" --runtime-dir "$APP_DB_RUNTIME"
python3 infra/protocol_v3_app_postgres.py check    --manifest "$MANIFEST" --runtime-dir "$APP_DB_RUNTIME"

/usr/bin/python3 -I -S infra/protocol_v3_operator.py supervisor check \
  --repository-root "$PWD" \
  --manifest "$MANIFEST" \
  --application-env "$APP_ENV"
/usr/bin/python3 -I -S infra/protocol_v3_operator.py supervisor run \
  --repository-root "$PWD" \
  --manifest "$MANIFEST" \
  --application-env "$APP_ENV" \
  --status-file "$SUPERVISOR_STATUS"

# After the supervisor is stopped:
python3 infra/protocol_v3_app_postgres.py stop --manifest "$MANIFEST" --runtime-dir "$APP_DB_RUNTIME"

external-health cadence

Every 15 seconds from the fixed Source-A producer

maximum accepted age

60 seconds; stopping the sidecar makes every V3 readiness gate fail closed

GET /health

Liveness and component state; HTTP 200 does not authorize a transaction

GET /ready

Manual operational gate

GET /ready/mystery

Manual gate plus genuine seven-day Mystery readiness

GET /ready/full-acceptance

Complete release-acceptance gate

The API and Attestor independently verify the artifact path, SHA/pointer, source producer, runtime identity, finalized block and all nested checks. Missing Docker/Node/PostgreSQL health, stale evidence, Provider/subscription drift, legacy Provider 3 reactivation, pending VRF, insufficient balances, BHS Keeper coverage/funding/process/supervisor drift, runtime mismatch or worker failure returns 503; no old wrapper, tracked mutable heartbeat, or later supervisor PID may override it.

The supervisor runs in the foreground under launchd, systemd or a hosted service manager, restarts isolated sealed children with bounded backoff, terminates process groups cleanly and writes a secret-free public status artifact. A release-scoped owner-private lease stays locked across live probes and restart decisions; inode replacement or seal/tool drift stops the service. Restart one role without changing its runtime identity or key; leave PostgreSQL, Chainlink Node and BHS Keeper independent unless their own recovery plan requires action.

/usr/bin/python3 -I -S infra/protocol_v3_operator.py supervisor run \
  --repository-root "$PWD" \
  --only jit

• A holding-reporter replica must acquire the session-scoped holding-reporter:<player> advisory lock on one checked-out PostgreSQL client and explicitly unlock it before reuse.
• A busy player lock skips that player; it does not serialize other players. A work/query/unlock error destroys the client rather than returning a leaked lock to the pool.
• The external-health sidecar is not a heartbeat substitute. Chainlink Node, its PostgreSQL, BHS, RPC and onchain runtime identities remain independently checked.

runtime key + service role
stable business intent key + generation
sender + chain ID 46630 + fixed nonce
destination + value + calldata hash
complete signed raw transaction
expected tx hash = keccak256(raw transaction)

1. 01

   Rebuild the expected intent

   Bind manifest runtime, role, immutable business identity, target and calldata; reject any sender/target/calldata/runtime mismatch or ambiguous hash.

2. 02

   Reconcile the exact expected hash

   Query receipt and transaction presence. Pending means wait and retain the same raw bytes—never sign a replacement merely because an RPC timed out.

3. 03

   Rebroadcast only the same envelope

   If the hash is absent and both latest and pending sender nonces still equal the reserved nonce, rebroadcast the exact stored raw transaction.

4. 04

   Stop on nonce conflict

   If either sender nonce advanced without the expected hash, mark the outbox row conflict and stop. Never guess which transaction consumed it.

5. 05

   Confirm from canonical evidence

   Mark confirmed only after the configured depth and required event. Holding requires its exact accepted-report event; JIT is a delivery only when the receipt contains StockDelivered.

A receipt-proven revert may create a new immutable generation because it had no business effect. Missing or ambiguous receipts do not. A confirmed JIT attempt without StockDelivered is one completed attempt, not a delivery; a later retry gets a new logical attempt only after that receipt is reconciled.

SELECT service_role,status,count(*)
FROM signed_transaction_outbox
GROUP BY service_role,status
ORDER BY service_role,status;

SELECT service_role,intent_key,sender,nonce,transaction_hash,status,last_error
FROM signed_transaction_outbox
WHERE status IN ('prepared','broadcast','mined','conflict')
ORDER BY updated_at;

Recovery plans are release-local, scope-bound and fixed to a tracked producer catalog. Public-Testnet recovery may prove backup failover and deterministic journal replay without mutating the chain, but only after the outer Operator verifies distinct public control-plane declarations. The current primary and Explorer endpoints both declare robinhood-official, so different URLs do not release this gate. RPC outage and reorg fault injection belong only to the local-fork scope.

1. 01

   Rebuild the verifier from Source A

   Create strict verification receipts with the tracked producer reconstructed from the exact Source-A Git archive and an offline dependency install. A mutable worktree verifier cannot sign its own acceptance.

2. 02

   Bind the Testnet release authority

   Use the Source-A-pinned operational executor. Its key is read only from a repository-external owner-owned 0600 single-link file; it never enters argv, stdout, stderr or an Evidence artifact and is explicitly Mainnet-ineligible.

3. 03

   Exclude private runtime state

   Evidence discovery rejects or excludes release-sealed.env, execution-environment.private.json, Foundry cache/out/tmp, sealed execution/workspace trees, database credentials, RPC credentials, private keys and raw signed transactions. Public evidence that unexpectedly resembles a secret still fails closed.

4. 04

   Verify all three receipts

   The final Evidence B registry must bind source, online and live strict-verifier receipts to the same Source A, manifest, authority, finalized blocks and accepted artifacts. Missing, stale or independently unverifiable receipts block publication.

• Do not use --allow-dirty output as release evidence; it is local debugging only.
• Do not expose commands, environment values, signatures, DATABASE_URL values or private keys in a status artifact or support packet.
• Do not run a state-changing acceptance stage on stale external health, a false readiness layer, a dirty tree or a manifest/runtime mismatch.
• Do not predict or hand-edit receipt-derived IDs, replace an unknown broadcast, reuse a quote/nonce, or advance a task after an unexplained conflict.
• Do not stop/delete unrelated Docker projects or volumes, and never use down -v for the release databases.
• Do not copy Testnet envelopes, nonces, task projections, database volumes, artifacts, RPC identities or private keys into Mainnet.