fortiblox/arc-mev-bot
Watch
3
0
Fork
You've already forked arc-mev-bot
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(devops): Phase 5 alert metrics, ordering-regime timer units, alert rules (WP 7916) #13

Merged
buddysan merged 1 commit from devops/phase5-alerts into main 2026-09-16 14:02:26 +00:00
Conversation 2 Commits 1 Files changed 13 +1870 -142
buddysan commented 2026-09-16 14:00:11 +00:00
Owner
Copy link

WP 7916 follow-up — Phase 5 alert metrics, ordering-regime timer, alert rules (from research §4.5)

  • bot/metrics.py: adds arcmev_stage_seconds{stage}, arcmev_head_skipped_total{reason}, arcmev_refresh_failed_total{endpoint}, arcmev_executor_paused, arcmev_adapter_code_missing, arcmev_base_fee_wei, arcmev_inclusion_delay_blocks; documents the ordering-regime gauges (arcmev_ordering_inversions_unexplained_pct, arcmev_ordering_extra_data_mismatch, arcmev_ordering_probe_ts_seconds); merges *.prom textfiles from ARCMEV_TEXTFILE_DIR into /metrics; python3 -m bot.metrics runs standalone so 100.97.18.104:9109 is scrapeable before the bot process exists.
  • deploy/systemd/ (NOT installed — follow-up, install + rollback in its README): arc-mev-metrics.service, hourly arc-mev-ordering-regime.timer/.service running docs/research/probes/ordering_regime_probe.py against the local node, ordering_regime_export.py (JSON → textfile, exit 2 on inversions_unexplained > 1 % or extra_data mismatch > 0).
  • monitoring/prometheus/arc-mev-bot.rules.yml: 14 alerts (heads skipped > 5 %, revert ratio > 30 %, refresh failing 3 min, sending-while-paused, adapter code missing, base fee > 5× floor, loop over budget, RPC errors, ordering rule / fee model changed, probe stale/missing, down, head stale). Dashboard regenerated (29 panels) from the checked-in gen_dashboard.py.
  • .gitignore: __pycache__/ + *.pyc at any depth (PR #6/#7 class).
  • docs/monitoring.md: metric table + alert table + Phase 5 timer runbook.

Verified on LXC 140: make test OK (Python 3.12.3); probe over 40 local blocks → 0 unexplained inversions, 0 extra_data mismatch; exporter → textfile → served through /metrics; /healthz 200.

🤖 Generated with Claude Code

## WP 7916 follow-up — Phase 5 alert metrics, ordering-regime timer, alert rules (from research §4.5) - **`bot/metrics.py`**: adds `arcmev_stage_seconds{stage}`, `arcmev_head_skipped_total{reason}`, `arcmev_refresh_failed_total{endpoint}`, `arcmev_executor_paused`, `arcmev_adapter_code_missing`, `arcmev_base_fee_wei`, `arcmev_inclusion_delay_blocks`; documents the ordering-regime gauges (`arcmev_ordering_inversions_unexplained_pct`, `arcmev_ordering_extra_data_mismatch`, `arcmev_ordering_probe_ts_seconds`); merges `*.prom` textfiles from `ARCMEV_TEXTFILE_DIR` into `/metrics`; `python3 -m bot.metrics` runs standalone so `100.97.18.104:9109` is scrapeable before the bot process exists. - **`deploy/systemd/`** (NOT installed — follow-up, install + rollback in its README): `arc-mev-metrics.service`, hourly `arc-mev-ordering-regime.timer/.service` running `docs/research/probes/ordering_regime_probe.py` against the local node, `ordering_regime_export.py` (JSON → textfile, exit 2 on `inversions_unexplained > 1 %` or `extra_data mismatch > 0`). - **`monitoring/prometheus/arc-mev-bot.rules.yml`**: 14 alerts (heads skipped > 5 %, revert ratio > 30 %, refresh failing 3 min, sending-while-paused, adapter code missing, base fee > 5× floor, loop over budget, RPC errors, ordering rule / fee model changed, probe stale/missing, down, head stale). Dashboard regenerated (29 panels) from the checked-in `gen_dashboard.py`. - **`.gitignore`**: `__pycache__/` + `*.pyc` at any depth (PR #6/#7 class). - `docs/monitoring.md`: metric table + alert table + Phase 5 timer runbook. Verified on LXC 140: `make test` OK (Python 3.12.3); probe over 40 local blocks → 0 unexplained inversions, 0 extra_data mismatch; exporter → textfile → served through `/metrics`; `/healthz` 200. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
feat(devops): Phase 5 alert metrics, ordering-regime timer units, alert rules, gitignore pycache
All checks were successful
ci / forge-tests (pull_request) Successful in 2s
Details
ci / python-tests (pull_request) Successful in 6s
Details
ci / forge-tests (push) Successful in 1s
Details
ci / python-tests (push) Successful in 9s
Details
b82c89490b
 
Follow-up to #8 from docs/research/arc-arb-learnings.md §4.5.

- bot/metrics.py: arcmev_stage_seconds{stage}, head_skipped_total{reason},
  refresh_failed_total{endpoint}, executor_paused, adapter_code_missing,
  base_fee_wei, inclusion_delay_blocks; ordering-regime gauges documented;
  *.prom textfile merge (ARCMEV_TEXTFILE_DIR) and a standalone
  `python3 -m bot.metrics` main so :9109 is scrapeable before the bot exists.
- deploy/systemd/: arc-mev-metrics.service, arc-mev-ordering-regime.service
  + .timer (hourly, local node, read-only), ordering_regime_export.py
  (probe JSON -> textfile, exit 2 on inversions_unexplained > 1 % or
  extra_data mismatch). NOT installed yet; README has install + rollback.
- monitoring/prometheus/arc-mev-bot.rules.yml: 14 alerts mapped to
  arc-arb incidents; dashboard regenerated (29 panels) from
  monitoring/grafana/gen_dashboard.py.
- .gitignore: __pycache__/ and *.pyc at any depth (PR #6/#7 class).
- Verified on LXC 140: make test OK, probe 40 blocks -> 0 unexplained
  inversions, 0 extra_data mismatch, textfile served via /metrics.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: BuddySan <[email protected]>
fortiblox-dev approved these changes 2026-09-16 14:02:25 +00:00
fortiblox-dev left a comment
Copy link

automerger v2 auto-approved: all 7 gates pass. Approver: fortiblox-dev (opposite-identity rule). Gates: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker.

automerger v2 auto-approved: all 7 gates pass. Approver: fortiblox-dev (opposite-identity rule). Gates: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker.
buddysan deleted branch devops/phase5-alerts 2026-09-16 14:02:26 +00:00
buddysan commented 2026-09-16 14:02:26 +00:00
Author
Owner
Copy link

Auto-merged by forgejo-automerger at 2026-09-16 14:02:26 UTC. Approver: fortiblox-dev (opposite-identity rule, Option C upgrade 2026-05-13). Merger: buddysan via fast-forward-only (NOT admin force_merge). Criteria passed: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker. Branch b82c89490b merged onto main and feature branch deleted. See project_forgejo_automerger memory for rationale.

Auto-merged by **forgejo-automerger** at 2026-09-16 14:02:26 UTC. **Approver:** `fortiblox-dev` (opposite-identity rule, Option C upgrade 2026-05-13). **Merger:** `buddysan` via `fast-forward-only` (NOT admin `force_merge`). **Criteria passed:** `mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker`. Branch `b82c89490b` merged onto `main` and feature branch deleted. See `project_forgejo_automerger` memory for rationale.
Sign in to join this conversation.
Reviewers
No reviewers
fortiblox-dev
Labels
Clear labels   
auto-merge
Opt-in: automerger will land this PR after CI green + all gates pass

  
auto-merge
Automerge when CI passes (set by webhook bridge)

No labels
auto-merge
auto-merge
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
fortiblox/arc-mev-bot!13
No description provided.