feat(bot): orchestrator daemon (bot/main.py) + Docker packaging #22

Merged

buddysan merged 4 commits from orchestrator-main-wp7931 into main 2026-09-16 16:54:53 +00:00

Conversation 3 Commits 4 Files changed 16 +1342 -82

buddysan commented 2026-09-16 16:32:06 +00:00

Owner

Copy link

Summary

• bot/main.py — the orchestrator: registry -> reserves -> cycles -> search -> bidding -> submit, wired into one continuously-running process (architecture.md §5), with a two-key dry-run safety gate, startup preflight (eth_getCode on the executor + every registry pool), graceful SIGTERM/SIGINT shutdown, and Prometheus textfile metrics (merges into the host's existing arc-mev-metrics.service, never binds :9109 itself).
• bot/tests/test_main.py — offline, fixture-based, end-to-end integration tests exercising the whole pipeline with a fake JSON-RPC transport (no network). This caught a real bug spanning bot/bidding.py + bot/submit.py: Opportunity.contention is a dict per architecture §2.3, but both modules treated it as a bare int — fixed both, updated their unit tests to the correct shape.
• Found and fixed the same negative-scientific-notation regex gap in test_metrics.py and test_ordering_regime_export.py (their exposition-format assertions rejected a real, valid Prometheus float like 8.9e-06).
• config/bot.toml was missing from .gitignore/secret-scan despite architecture §4.7 documenting it as machine-local like rpc-pool.toml — fixed for consistency (no secrets in it).
• Dockerfile + docker-compose.yml — independent of fortiblox-arc-arb's compose stack, python:3.12-slim, network_mode: host, non-root user, no key material mounted this wave, restart: unless-stopped, HEALTHCHECK.
• Safety default: dry_run = true, pinned further by --force-dry-run in the container's default command. This PR ships and verifies dry-run only.

Test plan

• make lint (py_compile + secret-scan) — clean
• make test — 283/283 offline unit tests pass (264 prior + 19 new in test_main.py)
• make forge-test — unchanged contracts/, runs in CI (no local forge on this host)
• Live dry-run verification on LXC 140 (docker build + docker compose up, ≥200 real blocks, confirm zero transactions sent) — in progress, results to follow in a PR comment

🤖 Generated with Claude Code

## Summary - `bot/main.py` — the orchestrator: registry -> reserves -> cycles -> search -> bidding -> submit, wired into one continuously-running process (architecture.md §5), with a two-key dry-run safety gate, startup preflight (`eth_getCode` on the executor + every registry pool), graceful SIGTERM/SIGINT shutdown, and Prometheus textfile metrics (merges into the host's existing `arc-mev-metrics.service`, never binds :9109 itself). - `bot/tests/test_main.py` — offline, fixture-based, end-to-end integration tests exercising the whole pipeline with a fake JSON-RPC transport (no network). This caught a real bug spanning `bot/bidding.py` + `bot/submit.py`: `Opportunity.contention` is a dict per architecture §2.3, but both modules treated it as a bare int — fixed both, updated their unit tests to the correct shape. - Found and fixed the same negative-scientific-notation regex gap in `test_metrics.py` and `test_ordering_regime_export.py` (their exposition-format assertions rejected a real, valid Prometheus float like `8.9e-06`). - `config/bot.toml` was missing from `.gitignore`/secret-scan despite architecture §4.7 documenting it as machine-local like `rpc-pool.toml` — fixed for consistency (no secrets in it). - `Dockerfile` + `docker-compose.yml` — independent of `fortiblox-arc-arb`'s compose stack, `python:3.12-slim`, `network_mode: host`, non-root user, no key material mounted this wave, `restart: unless-stopped`, `HEALTHCHECK`. - Safety default: `dry_run = true`, pinned further by `--force-dry-run` in the container's default command. This PR ships and verifies dry-run only. ## Test plan - [x] `make lint` (py_compile + secret-scan) — clean - [x] `make test` — 283/283 offline unit tests pass (264 prior + 19 new in `test_main.py`) - [ ] `make forge-test` — unchanged `contracts/`, runs in CI (no local `forge` on this host) - [ ] Live dry-run verification on LXC 140 (`docker build` + `docker compose up`, ≥200 real blocks, confirm zero transactions sent) — in progress, results to follow in a PR comment 🤖 Generated with [Claude Code](https://claude.com/claude-code)

buddysan added 1 commit 2026-09-16 16:32:06 +00:00

feat(bot): compose the orchestrator daemon (bot/main.py) + Docker packaging ... d9ba81f1a4

Assembles every prior Phase 0-4 module into one continuously-running process
per architecture.md §5 (registry -> reserves -> cycles -> search -> bidding ->
submit), with a two-key dry-run safety gate (config dry_run=false AND
--allow-live both required before any send), startup preflight (eth_getCode
on the executor and every registry pool, per contracts/README.md item 8),
graceful SIGTERM/SIGINT shutdown, and Prometheus textfile metrics that merge
into the host's existing arc-mev-metrics.service (never binds :9109 itself).

Offline integration tests (bot/tests/test_main.py) replay real recorded chain
state through the whole pipeline end-to-end with a fake JSON-RPC transport --
the class of test no single prior lane could write. That replay caught a real
bug spanning bot/bidding.py and bot/submit.py: search.py emits
Opportunity.contention as {touched_pools, swaps_head_block, distinct_senders}
(architecture §2.3), but validate_opportunity() required a bare int and
submit.py's max_contention gate compared it directly with `>` -- either would
have raised/misbehaved on every real Opportunity search.py ever produces.
Fixed both, updated their own unit tests to the correct shape, and found the
same negative-scientific-notation regex gap in two metrics-exposition tests
(test_metrics.py, test_ordering_regime_export.py) along the way.

Also: config/bot.toml was missing from .gitignore/secret-scan even though
architecture.md §4.7 documents it as machine-local like rpc-pool.toml (no
secrets in it, low severity, fixed for consistency).

Docker packaging: Dockerfile (python:3.12-slim, matches LXC 140's Python
3.12.3; bot/ verified pure-stdlib) + docker-compose.yml, independent of
fortiblox-arc-arb's compose stack, network_mode: host, non-root user, no key
material mounted this wave (dry_run=true / signer=none needs none), restart:
unless-stopped, HEALTHCHECK against the metrics-textfile writer's liveness.

WP 7931, 7932, 7933 (summary 7930).

Signed-off-by: BuddySan <[email protected]>

Co-Authored-By: Claude Sonnet 5 <[email protected]>

buddysan added the

auto-merge

label 2026-09-16 16:32:45 +00:00

buddysan added 1 commit 2026-09-16 16:37:08 +00:00

fix(bot): page the startup eth_getCode preflight against large registries ... 561b6654f1

Found live on LXC 140 during dry-run verification against the real 983-pool
discover_pools.py sweep: preflight_code_checks() sent every registry pool's
eth_getCode as ONE unbounded JSON-RPC batch array (rpc_pool.py's local
endpoint class has no batch_cap by config schema, since §4.5's -32005 rule is
public-only) -- the local node's HTTP response came back with a missing id,
rpc_pool.py correctly cooled that endpoint and failed over to public
endpoints, which then hit their own -32005 rate limit. "Unbounded" in the
pool config schema is a rate-limit-class statement, not a guarantee that any
batch size is safe against a real node's HTTP/JSON stack.

Fix: page the getCode batch into PREFLIGHT_BATCH_PAGE=200-request chunks
regardless of the endpoint's declared cap. This is a one-time startup cost,
not the hot path, so a conservative fixed page size is the right, boring
fix. Added a test proving the request count per pool.batch() call never
exceeds the page size, however large the registry is.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Signed-off-by: BuddySan <[email protected]>

buddysan added 1 commit 2026-09-16 16:44:38 +00:00

fix(config): backfill chain_id + fee_bps in the committed 8-pool sample registry ... 1d7f46c920

Found live on LXC 140 during Docker verification (WP 7931): with a real
container running bot/main.py's default config against the default
committed config/pools.testnet.json, the process crash-looped forever
(restart: unless-stopped) because registry.py's validator -- correctly --
refused the file: no top-level chain_id, and every v2 pool missing its
fee_bps key entirely (an older discover_pools.py schema, predating fee_bps
discovery). This wasn't containerization-specific: bot/search.py's own
dry-run CLI (`registry_mod.load(path, expect_chain_id=...)`) would hit the
identical RegistryError against this same file -- it just had never been
run end-to-end before.

Fixed by replacing these same 8 pools with their verified counterparts from
the real discover_pools.py full sweep (983 pools, data-director's Phase 1
output) -- same pair/token/factory/block, with chain_id, fee_bps (30 for
all 8, matching the four factories' on-chain-verified fee, see
config/bot.example.toml's [fees] comment) and reserves_nonzero_at_block
backfilled from that sweep, plus the matching `factories` entries and a
recomputed `registry_hash`. Every added value is real on-chain-verified
data pulled from the existing sweep output, not a guess.

`python3 -m bot.registry validate config/pools.testnet.json --chain-id
5042002` now passes; all 284 offline tests still pass (nothing depended on
the old broken values). Simplified test_main.py's ConfigBundleWiringTest to
use the real file directly now that it's self-consistent.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Signed-off-by: BuddySan <[email protected]>

fortiblox-dev commented 2026-09-16 16:47:18 +00:00

Owner

Copy link

Live dry-run verification (LXC 140, 2026-09-16) — complete

Docker-in-LXC: verified clean early (docker version/docker info, overlayfs + cgroup v2, no nested-virt issues). Not a blocker.

Build + host networking: docker build succeeds (no pip installs needed — bot/ verified pure-stdlib via an AST import scan before the Dockerfile was written). docker run --network host --entrypoint python3 arc-mev-bot:latest -c '<eth_blockNumber via 127.0.0.1:8545>' returned a real block number — network_mode: host reaches the local node exactly like the Arc node's own containers do.

Two real bugs found and fixed live (both already pushed to this branch):

1. preflight_code_checks() sent all-983-pools' eth_getCode as one unbounded JSON-RPC batch → the local node's HTTP response came back with a missing id. Fixed: paged into 200-request chunks (commit 561b665).
2. The committed config/pools.testnet.json (8-pool sample) has no top-level chain_id and every pool is missing fee_bps — registry.py's validator correctly refused it, which crash-looped the container against the default config. This wasn't Docker-specific: bot/search.py's own dry-run CLI would hit the identical RegistryError — it had just never been run end-to-end. Fixed by replacing the 8 pools with their verified counterparts from the real discover_pools.py full sweep (983 pools, data-director's output) — same pools, backfilled chain_id/fee_bps/reserves_nonzero_at_block from that verified data, recomputed registry_hash (commit 1d7f46c).

Permission gap found and fixed on the box (not committed — host/operational, not repo state): /var/lib/arc-mev-bot/textfile was root:root 0755; the container's non-root arcmev user (UID 999) couldn't write its .prom file there. chmod 0777 (multi-writer drop zone for non-sensitive Prometheus text — the same directory the host's arc-mev-ordering-regime timer, running as root, also writes into). Confirmed merging into http://127.0.0.1:9109/metrics afterward.

220-block live run (real chain state, block 62431931→62432154, real 983-pool registry via a local override, dry-run only): 138 opportunities found (all real 2-3 hop v2 cycles), all ultimately dropped at the bidding stage (30 regime_cold during warm-up, 108 unprofitable_at_base — none cleared the real 1.00 USDC bidding margin against these thin testnet pools), so submit.Submitter.execute()'s sim/dry-run path itself wasn't exercised by this live run (it is exhaustively covered offline in bot/tests/test_main.py against real recorded opportunities). truncated_blocks: 0. search_ms p50 5.7ms/p90 8.6ms (matches revenue-director's own isolated benchmark). Whole-pipeline wall_ms p50 76.7ms/p90 126ms but p99 563ms / max 1877ms — over the ~500ms block budget under real RPC contention/failover (public endpoints hitting -32005, occasional local-node "response missing id"); flagged as an open item below, not hidden.

Resource usage: arc-mev-bot container 1.4-6.7% CPU, 33-49MiB RAM vs. arc-execution's 7GB/13%+ on the same 4-vCPU box — no node contention.

Zero transactions sent, confirmed on-chain across the entire session (baseline → 220-block run → restart → second run): executor 0x3b32…BD6c nonce stayed 0x3, balance stayed 40716030781864998014722 wei; owner 0x4a35…F290 nonce stayed 0x0, balance stayed 1000000000000000000 wei; executor contract's own USDC inventory stayed 2000000 (2.000000 USDC). No new tx from either address.

Container survives a restart: proven twice — (1) the --max-blocks 220 verification run exited 0 cleanly and Docker's restart: unless-stopped brought it back up automatically; (2) an explicit docker compose restart bot afterward came back healthy within the HEALTHCHECK window. RestartCount: 0 in its current steady state (production defaults, committed 8-pool registry).

Metrics merge — real finding, NOT fixed in this PR (flagging for devops-director): the host's standalone arc-mev-metrics.service (python3 -m bot.metrics) and this container both import the same bot/metrics.py shared metric set. Metrics only ever .inc()'d by the container (e.g. arcmev_blocks_seen_total) merge fine (the host's own copy has no children, so only a duplicate # TYPE/# HELP header appears — cosmetic). But arcmev_up_since_seconds and arcmev_build_info are .set() at import time in both processes, so the merged /metrics output has two genuine duplicate-name samples for arcmev_up_since_seconds (different values) — a real Prometheus exposition-format violation once anything actually scrapes this endpoint (wave-3, per docs/monitoring.md, not yet registered). Recommend: once a real bot container is the source of truth, stop arc-mev-metrics.service's own in-process REGISTRY.render() (textfile-passthrough only), matching that unit's own comment ("When the bot itself calls metrics.serve() later, stop this unit — one listener per port"). Out of this PR's scope (shared bot/metrics.py contract + arc-mev-metrics.service ownership).

Open items (not blocking, all honestly reported):

• wall_ms p99/max over the ~500ms block budget under contended/rate-limited conditions — no per-block deadline enforcement at the orchestrator level yet (search.py has its own internal cycle-enumeration deadline; the whole-pipeline loop does not). Worth a follow-up wave.
• No registry hot-reload — registry/cycles loaded once at startup for the process lifetime (documented limitation).
• Live send path (submit.Submitter.execute() past simulate()) wasn't exercised by the live 220-block run itself (every real opportunity found was economically below the real bidding margin) — it IS exhaustively covered offline in bot/tests/test_main.py with real recorded opportunities and a fake transport.
• Duplicate-metric-name exposition issue above.

make lint / make test: 284/284 pass. make forge-test: not run locally (no forge on this host, unchanged contracts/, runs in CI).

## Live dry-run verification (LXC 140, 2026-09-16) — complete **Docker-in-LXC**: verified clean early (`docker version`/`docker info`, overlayfs + cgroup v2, no nested-virt issues). Not a blocker. **Build + host networking**: `docker build` succeeds (no pip installs needed — `bot/` verified pure-stdlib via an AST import scan before the Dockerfile was written). `docker run --network host --entrypoint python3 arc-mev-bot:latest -c '<eth_blockNumber via 127.0.0.1:8545>'` returned a real block number — `network_mode: host` reaches the local node exactly like the Arc node's own containers do. **Two real bugs found and fixed live** (both already pushed to this branch): 1. `preflight_code_checks()` sent all-983-pools' `eth_getCode` as one unbounded JSON-RPC batch → the local node's HTTP response came back with a missing id. Fixed: paged into 200-request chunks (commit `561b665`). 2. The committed `config/pools.testnet.json` (8-pool sample) has no top-level `chain_id` and every pool is missing `fee_bps` — `registry.py`'s validator correctly refused it, which crash-looped the container against the *default* config. This wasn't Docker-specific: `bot/search.py`'s own dry-run CLI would hit the identical `RegistryError` — it had just never been run end-to-end. Fixed by replacing the 8 pools with their verified counterparts from the real `discover_pools.py` full sweep (983 pools, data-director's output) — same pools, backfilled `chain_id`/`fee_bps`/`reserves_nonzero_at_block` from that verified data, recomputed `registry_hash` (commit `1d7f46c`). **Permission gap found and fixed on the box** (not committed — host/operational, not repo state): `/var/lib/arc-mev-bot/textfile` was root:root 0755; the container's non-root `arcmev` user (UID 999) couldn't write its `.prom` file there. `chmod 0777` (multi-writer drop zone for non-sensitive Prometheus text — the same directory the host's `arc-mev-ordering-regime` timer, running as root, also writes into). Confirmed merging into `http://127.0.0.1:9109/metrics` afterward. **220-block live run** (real chain state, block 62431931→62432154, real 983-pool registry via a local override, dry-run only): 138 opportunities found (all real 2-3 hop v2 cycles), all ultimately dropped at the bidding stage (30 `regime_cold` during warm-up, 108 `unprofitable_at_base` — none cleared the real 1.00 USDC bidding margin against these thin testnet pools), so `submit.Submitter.execute()`'s sim/dry-run path itself wasn't exercised by *this* live run (it is exhaustively covered offline in `bot/tests/test_main.py` against real recorded opportunities). `truncated_blocks: 0`. `search_ms` p50 5.7ms/p90 8.6ms (matches revenue-director's own isolated benchmark). Whole-pipeline `wall_ms` p50 76.7ms/p90 126ms but **p99 563ms / max 1877ms — over the ~500ms block budget** under real RPC contention/failover (public endpoints hitting `-32005`, occasional local-node "response missing id"); flagged as an open item below, not hidden. **Resource usage**: `arc-mev-bot` container 1.4-6.7% CPU, 33-49MiB RAM vs. `arc-execution`'s 7GB/13%+ on the same 4-vCPU box — no node contention. **Zero transactions sent, confirmed on-chain** across the entire session (baseline → 220-block run → restart → second run): executor `0x3b32…BD6c` nonce stayed `0x3`, balance stayed `40716030781864998014722` wei; owner `0x4a35…F290` nonce stayed `0x0`, balance stayed `1000000000000000000` wei; executor contract's own USDC inventory stayed `2000000` (2.000000 USDC). No new tx from either address. **Container survives a restart**: proven twice — (1) the `--max-blocks 220` verification run exited 0 cleanly and Docker's `restart: unless-stopped` brought it back up automatically; (2) an explicit `docker compose restart bot` afterward came back `healthy` within the `HEALTHCHECK` window. `RestartCount: 0` in its current steady state (production defaults, committed 8-pool registry). **Metrics merge — real finding, NOT fixed in this PR (flagging for devops-director)**: the host's standalone `arc-mev-metrics.service` (`python3 -m bot.metrics`) and this container both import the same `bot/metrics.py` shared metric set. Metrics only ever `.inc()`'d by the container (e.g. `arcmev_blocks_seen_total`) merge fine (the host's own copy has no children, so only a duplicate `# TYPE`/`# HELP` header appears — cosmetic). But `arcmev_up_since_seconds` and `arcmev_build_info` are `.set()` at **import time** in *both* processes, so the merged `/metrics` output has two genuine duplicate-name samples for `arcmev_up_since_seconds` (different values) — a real Prometheus exposition-format violation once anything actually scrapes this endpoint (wave-3, per `docs/monitoring.md`, not yet registered). Recommend: once a real bot container is the source of truth, stop `arc-mev-metrics.service`'s own in-process `REGISTRY.render()` (textfile-passthrough only), matching that unit's own comment ("When the bot itself calls `metrics.serve()` later, stop this unit — one listener per port"). Out of this PR's scope (shared `bot/metrics.py` contract + `arc-mev-metrics.service` ownership). **Open items** (not blocking, all honestly reported): - `wall_ms` p99/max over the ~500ms block budget under contended/rate-limited conditions — no per-block deadline enforcement at the orchestrator level yet (search.py has its own internal cycle-enumeration deadline; the whole-pipeline loop does not). Worth a follow-up wave. - No registry hot-reload — registry/cycles loaded once at startup for the process lifetime (documented limitation). - Live send path (`submit.Submitter.execute()` past `simulate()`) wasn't exercised by the live 220-block run itself (every real opportunity found was economically below the real bidding margin) — it IS exhaustively covered offline in `bot/tests/test_main.py` with real recorded opportunities and a fake transport. - Duplicate-metric-name exposition issue above. make lint / make test: 284/284 pass. make forge-test: not run locally (no `forge` on this host, unchanged `contracts/`, runs in CI).

buddysan added 1 commit 2026-09-16 16:49:33 +00:00

Merge branch 'main' into orchestrator-main-wp7931 28e54a3302

fortiblox-dev approved these changes 2026-09-16 16:54:44 +00:00

fortiblox-dev left a comment

Copy link

automerger v2 auto-approved: all 7 gates pass. Approver: fortiblox-dev (opposite-identity rule). Gates: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker.

automerger v2 auto-approved: all 7 gates pass. Approver: fortiblox-dev (opposite-identity rule). Gates: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker.

buddysan merged commit 28e54a3302 into main 2026-09-16 16:54:53 +00:00

buddysan deleted branch orchestrator-main-wp7931 2026-09-16 16:54:55 +00:00

buddysan commented 2026-09-16 16:54:56 +00:00

Author

Owner

Copy link

Auto-merged by forgejo-automerger at 2026-09-16 16:54:56 UTC. Approver: fortiblox-dev (opposite-identity rule, Option C upgrade 2026-05-13). Merger: buddysan via fast-forward-only (NOT admin force_merge). Criteria passed: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker. Branch 28e54a3302 merged onto main and feature branch deleted. See project_forgejo_automerger memory for rationale.

Auto-merged by **forgejo-automerger** at 2026-09-16 16:54:56 UTC. **Approver:** `fortiblox-dev` (opposite-identity rule, Option C upgrade 2026-05-13). **Merger:** `buddysan` via `fast-forward-only` (NOT admin `force_merge`). **Criteria passed:** `mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker`. Branch `28e54a3302` merged onto `main` and feature branch deleted. See `project_forgejo_automerger` memory for rationale.

Sign in to join this conversation.

Reviewers

No reviewers

fortiblox-dev

Labels

Clear labels   

auto-merge

Opt-in: automerger will land this PR after CI green + all gates pass

  

auto-merge

Automerge when CI passes (set by webhook bridge)

No labels

auto-merge

auto-merge

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

fortiblox/arc-mev-bot!22

No description provided.