fortiblox/arc-mev-bot
Watch
3
0
Fork
You've already forked arc-mev-bot
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(devops): rpc_probe + measured rpc-pool, log/metrics libs, node runbook (WP 7898, 7916) #8

Merged
buddysan merged 1 commit from devops/rpc-probe-observability into main 2026-09-16 13:54:41 +00:00
Conversation 2 Commits 1 Files changed 13 +3845 -4
buddysan commented 2026-09-16 13:51:03 +00:00
Owner
Copy link

WP 7898 — real RPC latency (Phase 0 close-out) + WP 7916 — structured logs & Prometheus (library part)

What

  • bot/rpc_probe.py (stdlib): per [[endpoint]] — N×eth_blockNumber p50/p95/max, one eth_getLogs, block-height lag in a tight final round, real RFC 6455 websocket round-trip (eth_chainId over a frame). --write stamps measured_latency_ms in place with when/where; --check exits 3 on 0 placeholders; --max-p50/--max-lag exit 4 (health-check mode); resolve = "<ip>" keeps hostname for SNI/Host (needed for arc-node-lxc139, MagicDNS gap on LXC 140). load_pool(require_measured=True) is the runtime guard for the pool loader.
  • config/rpc-pool.example.toml: 9 verified endpoints (local node, LXC 139 tailnet node, arc.io + 3 official provider mirrors, dRPC/Tenderly/arc.network), measured on LXC 140 2026-09-16 13:43 UTC, provenance comment per line; excluded/keyed endpoints listed in the header. The real gitignored config/rpc-pool.toml is installed on the box.
  • bot/log.py: JSON lines, fixed ts/level/event/block first, ints > 2^53 as strings, any 64-hex redacted (architecture §4.9), single-write thread safety, bound context loggers.
  • bot/metrics.py: Counter/Gauge/Histogram + text exposition 0.0.4, shared arcmev_* metric set (documented in the module docstring), serve() on 0.0.0.0:9109 (/metrics, /healthz → 503 on unhealthy), env overrides.
  • monitoring/prometheus/arc-mev-bot.scrape.yml + monitoring/grafana/arc-mev-bot-dashboard.json: drop-ins for the fleet stack on the validator (registration = separate wave-3 PR to fortiblox/x1-trading-bot, documented in docs/monitoring.md).
  • docs/runbook-node.md: start/stop/health-check the node, PATH-under-pct-exec fix, probe procedure, measured table, observed node backlog on probe day.
  • docs/architecture.md: one row only (bot/metrics.py → bind 0.0.0.0:9109, overridable) per that file's change-control rule; needed so the validator Prometheus can scrape over Tailscale.
  • Makefile untouched: make probe already runs python3 bot/rpc_probe.py, which works as a script.

Tests

make lint (py-compile + secret-scan) and make test green locally and on LXC 140 (Python 3.12.3): 63 offline cases incl. a recorded probe fixture bot/tests/fixtures/rpc_probe/probe-lxc140-62411028.json.

🤖 Generated with Claude Code

## WP 7898 — real RPC latency (Phase 0 close-out) + WP 7916 — structured logs & Prometheus (library part) ### What - `bot/rpc_probe.py` (stdlib): per `[[endpoint]]` — N×`eth_blockNumber` p50/p95/max, one `eth_getLogs`, block-height lag in a tight final round, real RFC 6455 websocket round-trip (`eth_chainId` over a frame). `--write` stamps `measured_latency_ms` in place with when/where; `--check` exits 3 on `0` placeholders; `--max-p50/--max-lag` exit 4 (health-check mode); `resolve = "<ip>"` keeps hostname for SNI/Host (needed for `arc-node-lxc139`, MagicDNS gap on LXC 140). `load_pool(require_measured=True)` is the runtime guard for the pool loader. - `config/rpc-pool.example.toml`: 9 verified endpoints (local node, LXC 139 tailnet node, arc.io + 3 official provider mirrors, dRPC/Tenderly/arc.network), **measured on LXC 140 2026-09-16 13:43 UTC**, provenance comment per line; excluded/keyed endpoints listed in the header. The real gitignored `config/rpc-pool.toml` is installed on the box. - `bot/log.py`: JSON lines, fixed `ts/level/event/block` first, ints > 2^53 as strings, **any 64-hex redacted** (architecture §4.9), single-write thread safety, bound context loggers. - `bot/metrics.py`: Counter/Gauge/Histogram + text exposition 0.0.4, shared `arcmev_*` metric set (documented in the module docstring), `serve()` on `0.0.0.0:9109` (`/metrics`, `/healthz` → 503 on unhealthy), env overrides. - `monitoring/prometheus/arc-mev-bot.scrape.yml` + `monitoring/grafana/arc-mev-bot-dashboard.json`: drop-ins for the fleet stack on the validator (registration = separate wave-3 PR to `fortiblox/x1-trading-bot`, documented in `docs/monitoring.md`). - `docs/runbook-node.md`: start/stop/health-check the node, PATH-under-pct-exec fix, probe procedure, measured table, observed node backlog on probe day. - **`docs/architecture.md`: one row only** (`bot/metrics.py` → bind `0.0.0.0:9109`, overridable) per that file's change-control rule; needed so the validator Prometheus can scrape over Tailscale. - Makefile untouched: `make probe` already runs `python3 bot/rpc_probe.py`, which works as a script. ### Tests `make lint` (py-compile + secret-scan) and `make test` green locally and on LXC 140 (Python 3.12.3): 63 offline cases incl. a recorded probe fixture `bot/tests/fixtures/rpc_probe/probe-lxc140-62411028.json`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(devops): rpc_probe + measured rpc-pool, structured log + Prometheus metrics, node runbook
All checks were successful
ci / forge-tests (pull_request) Successful in 1s
Details
ci / python-tests (pull_request) Successful in 5s
Details
dfc03c4268
 
WP 7898 (Phase 0 close-out) + WP 7916 (Phase 5 foundation, library part).

- bot/rpc_probe.py: per-endpoint eth_blockNumber p50/p95/max, eth_getLogs,
  height lag in a tight final round, real RFC 6455 ws round-trip; --write
  stamps measured_latency_ms with when/where; --check refuses 0 placeholders;
  --max-p50/--max-lag exit codes make it a health check; resolve=<ip> for
  tailnet hosts without MagicDNS (arc-node-lxc139). load_pool(require_measured)
  is the runtime guard.
- config/rpc-pool.example.toml: 9 verified endpoints, numbers measured on
  LXC 140 2026-09-16 (provenance comment per line); excluded/keyed endpoints
  listed in the header.
- bot/log.py: JSON lines, fixed ts/level/event/block first, big ints as
  strings, any 64-hex redacted (architecture §4.9), thread-safe single write.
- bot/metrics.py: stdlib Prometheus text exposition (Counter/Gauge/Histogram),
  shared arcmev_* metric set, serve() on 0.0.0.0:9109 (/metrics, /healthz).
- monitoring/: drop-in scrape job + Grafana 10.2 provisioning dashboard for
  the fleet stack (registration is a wave-3 PR to x1-trading-bot).
- docs/runbook-node.md, docs/monitoring.md; docs/architecture.md: only the
  bot/metrics.py row (bind 0.0.0.0:9109 so the validator Prometheus can scrape
  over Tailscale) per the file's change-control rule.
- tests: 63 offline unittest cases incl. recorded probe fixture from LXC 140.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: BuddySan <[email protected]>
buddysan force-pushed devops/rpc-probe-observability from dfc03c4268
All checks were successful
ci / forge-tests (pull_request) Successful in 1s
Details
ci / python-tests (pull_request) Successful in 5s
Details
to 304cc148be
Some checks failed
ci / forge-tests (pull_request) Successful in 2s
Details
ci / python-tests (pull_request) Successful in 6s
Details
ci / python-tests (push) Failing after 1s
Details
ci / forge-tests (push) Successful in 1s
Details
2026-09-16 13:52:58 +00:00 Compare
fortiblox-dev approved these changes 2026-09-16 13:54:40 +00:00
fortiblox-dev left a comment
Copy link

automerger v2 auto-approved: all 7 gates pass. Approver: fortiblox-dev (opposite-identity rule). Gates: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker.

automerger v2 auto-approved: all 7 gates pass. Approver: fortiblox-dev (opposite-identity rule). Gates: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker.
buddysan deleted branch devops/rpc-probe-observability 2026-09-16 13:54:41 +00:00
buddysan commented 2026-09-16 13:54:41 +00:00
Author
Owner
Copy link

Auto-merged by forgejo-automerger at 2026-09-16 13:54:41 UTC. Approver: fortiblox-dev (opposite-identity rule, Option C upgrade 2026-05-13). Merger: buddysan via fast-forward-only (NOT admin force_merge). Criteria passed: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker. Branch 304cc148be merged onto main and feature branch deleted. See project_forgejo_automerger memory for rationale.

Auto-merged by **forgejo-automerger** at 2026-09-16 13:54:41 UTC. **Approver:** `fortiblox-dev` (opposite-identity rule, Option C upgrade 2026-05-13). **Merger:** `buddysan` via `fast-forward-only` (NOT admin `force_merge`). **Criteria passed:** `mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker`. Branch `304cc148be` merged onto `main` and feature branch deleted. See `project_forgejo_automerger` memory for rationale.
Sign in to join this conversation.
Reviewers
No reviewers
fortiblox-dev
Labels
Clear labels   
auto-merge
Opt-in: automerger will land this PR after CI green + all gates pass

  
auto-merge
Automerge when CI passes (set by webhook bridge)

No labels
auto-merge
auto-merge
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
fortiblox/arc-mev-bot!8
No description provided.