feat(bot): extend discover_pools.py to v3/Curve, verify v2 fee_bps per factory #18

Merged

buddysan merged 6 commits from data/discover-v3-curve into main 2026-09-16 16:02:01 +00:00

Conversation 4 Commits 6 Files changed 8 +5804 -88

buddysan commented 2026-09-16 14:35:18 +00:00

Owner

Copy link

Discovery extension: v3/Curve + per-factory fee verification (data lane, WP 7901)

Extends bot/discover_pools.py from v2-only to a full sweep:

• v3 PoolCreated decoded and cross-checked against real slot0()/liquidity()/tickSpacing() reads on chain.
• Curve factory deploy events (6 topic shapes) — found zero on testnet in the swept window (evidence-backed: the tool scanned for them and can prove it). All 17 real Curve pools found instead through a TokenExchange-emitter pass + on-chain probe (coins/A/fee/ABI shape — one is a uint256-ABI Solidity stableswap distinct from the 16 int128-ABI NG-style pools).
• Aerodrome/Solidly/Algebra/Balancer/Uniswap-v4 events counted (other_venues), not registry kinds.

Per-factory v2 fee_bps verified two independent ways that must agree: swap-event reconstruction (exact integer match to the UniswapV2 formula) and eth_call simulation (pair.swap() with a state override on token0, binary-searching the accepted fee). Result: every UniswapV2-interface factory is 30 bps except one at 50 bps. Two PairCreated-emitting factories turned out to have no swap(uint256,uint256,address,bytes) selector in their pair bytecode — not UniswapV2 pairs regardless of the event — excluded from pools[] into excluded_factories (one is a 4,069-pool launchpad).

Sweep is resumable (checkpoint: next_block, per-source segments, topic counts) via an endpoint pool that prefers the local node and pages paced public providers for the pruned prefix (local node: logs only from block 62,000,000, eth_getLogs ≤ 100,000 blocks; public providers: full history but ≤ 10,000 blocks and a handful of req/s — all measured, see docs/discovery.md). No range is ever skipped.

config/pools.testnet.json regenerated from a live run over the local node full servable window (62,300,000–62,413,526): 229 pools (129 v2, 83 v3, 17 curve; 4,074 PairCreated pools correctly excluded), reserves_nonzero_at_block from a final Multicall3 pass. config/tokens.testnet.json follows (215 addresses). This is still a sample window, not full history — WP 7902 (full genesis sweep, running now on LXC 140) lands the scanned_from=0 file in the next PR.

docs/architecture.md updated in this PR (the one exception its own change-control rule allows) — §2.1 envelope/fields, §2.2 v3/curve verified, §7 fee_bps and v3/Curve-exists rows corrected. Conflict-resolved against revenue-director's and security-director's facts that landed on main meanwhile; both preserved.

Tests: bot/tests/test_discover_pools.py, 26 new cases — real recorded-log decoding for every event family, v2 fee math against a reference value + a 120-swap real pair, every provider error class seen during grounding, kill/resume idempotency.

🤖 Generated with Claude Code

## Discovery extension: v3/Curve + per-factory fee verification (data lane, WP 7901) Extends `bot/discover_pools.py` from v2-only to a full sweep: - **v3** `PoolCreated` decoded and cross-checked against real `slot0()`/`liquidity()`/`tickSpacing()` reads on chain. - **Curve** factory deploy events (6 topic shapes) — found **zero** on testnet in the swept window (evidence-backed: the tool scanned for them and can prove it). All 17 real Curve pools found instead through a `TokenExchange`-emitter pass + on-chain probe (`coins`/`A`/`fee`/ABI shape — one is a uint256-ABI Solidity stableswap distinct from the 16 int128-ABI NG-style pools). - Aerodrome/Solidly/Algebra/Balancer/Uniswap-v4 events counted (`other_venues`), not registry kinds. **Per-factory v2 `fee_bps`** verified two independent ways that must agree: swap-event reconstruction (exact integer match to the UniswapV2 formula) and `eth_call` simulation (`pair.swap()` with a state override on `token0`, binary-searching the accepted fee). Result: every UniswapV2-interface factory is 30 bps **except one at 50 bps**. Two `PairCreated`-emitting factories turned out to have **no `swap(uint256,uint256,address,bytes)` selector** in their pair bytecode — not UniswapV2 pairs regardless of the event — excluded from `pools[]` into `excluded_factories` (one is a 4,069-pool launchpad). Sweep is resumable (checkpoint: `next_block`, per-source segments, topic counts) via an endpoint pool that prefers the local node and pages paced public providers for the pruned prefix (local node: logs only from block 62,000,000, `eth_getLogs` ≤ 100,000 blocks; public providers: full history but ≤ 10,000 blocks and a handful of req/s — all measured, see `docs/discovery.md`). No range is ever skipped. `config/pools.testnet.json` regenerated from a live run over the local node full servable window (62,300,000–62,413,526): **229 pools** (129 v2, 83 v3, 17 curve; 4,074 `PairCreated` pools correctly excluded), `reserves_nonzero_at_block` from a final Multicall3 pass. `config/tokens.testnet.json` follows (215 addresses). This is still a **sample window**, not full history — WP 7902 (full genesis sweep, running now on LXC 140) lands the `scanned_from=0` file in the next PR. `docs/architecture.md` updated in this PR (the one exception its own change-control rule allows) — §2.1 envelope/fields, §2.2 v3/curve verified, §7 fee_bps and v3/Curve-exists rows corrected. Conflict-resolved against revenue-director's and security-director's facts that landed on `main` meanwhile; both preserved. Tests: `bot/tests/test_discover_pools.py`, 26 new cases — real recorded-log decoding for every event family, v2 fee math against a reference value + a 120-swap real pair, every provider error class seen during grounding, kill/resume idempotency. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

buddysan added 1 commit 2026-09-16 14:35:18 +00:00

feat(bot): extend discover_pools.py to v3/Curve, verify v2 fee_bps per factory (WP 7901) ... fa2d682b82

Extends the v2-only scanner to a full pool-creation sweep:
- v3 PoolCreated(address,address,uint24,int24,address), decoded and
  cross-checked against real slot0()/liquidity()/tickSpacing() reads.
- Curve factory deploy events (Plain/Meta/Crypto/Tricrypto/Twocrypto
  PoolDeployed) plus a TokenExchange-emitter pass for factory-less
  Curve pools (all 17 testnet Curve pools were emitter-discovered,
  not factory-deployed — the six factory topics found zero events in
  the swept window, which is itself the evidence-backed answer).
- Aerodrome/Solidly/Algebra/Balancer/Uniswap-v4 creation events are
  counted (other_venues) but not registry kinds, per §2.1.

Per-factory v2 fee_bps verification (two independent methods that
must agree): swap-event reconstruction against the exact integer
UniswapV2 formula, and eth_call simulation of pair.swap() with a
state override replacing token0's code by a constant balanceOf,
binary-searching the accepted fee. A factory with neither method
inherits the fee of a factory whose pair runtime bytecode is
identical post-metadata-strip. Result on testnet: every UniswapV2-
interface factory is 30 bps except one at 50 bps. Two factories that
emit PairCreated do NOT have a swap(uint256,uint256,address,bytes)
selector in their pair bytecode -- not UniswapV2 pairs regardless of
the event they emit -- excluded from pools[] and reported under
excluded_factories (a launchpad factory alone accounts for 4,069
excluded PairCreated pools).

The sweep is now resumable (checkpoint file: next_block, per-source
segments, topic counts, curve emitter table) and served by an
endpoint pool that prefers the local node and pages paced public
providers for the pruned prefix, adapting per-endpoint range caps
and topic-count caps at runtime from real provider errors (measured
limits and provider quirks recorded in docs/discovery.md). No range
is ever skipped; a range no endpoint can serve is a hard error, not
a silent gap.

config/pools.testnet.json is regenerated from a live run against the
LXC 140 local node over its full servable window (blocks
62,300,000-62,413,526): 229 pools (129 v2, 83 v3, 17 curve; 4,074
PairCreated pools correctly excluded), reserves_nonzero_at_block from
a final Multicall3 pass, registry_hash recomputed. config/tokens.testnet.json
follows (215 addresses, 6 colliding symbols confirming the key-by-address
rule). docs/architecture.md updated in this PR per its own change-control
rule (envelope fields, §2.1/§2.2/§7 corrections -- conflict-resolved
against revenue-director's and security-director's meanwhile-landed
facts, both preserved).

This PR's registry is still a local-node-window sample, not full
history -- WP 7902 (full genesis sweep, running now on LXC 140 via
docs/discovery.md's runbook) lands config/pools.testnet.json's final
scanned_from=0 version in a follow-up PR.

Tests: bot/tests/test_discover_pools.py, 26 new cases -- topic
decoding against real recorded logs for every event family, the v2
fee math against a reference UniswapV2 value and a 120-swap real
pair, endpoint-pool error classification (every provider message
seen during grounding: range caps, prune, 429, drpc free-plan,
Multicall3 result-count cap), and kill/resume idempotency (a
KeyboardInterrupt mid-chunk still produces a checkpoint a fresh
process resumes from byte-identically).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: BuddySan <[email protected]>

buddysan added the

auto-merge

label 2026-09-16 14:35:18 +00:00

buddysan added 1 commit 2026-09-16 14:40:42 +00:00

fix(bot): honor 'public RPC must not be used for the sweep'; drop cross-lane registry regression ... c299663dba

Two corrections to the previous commit, both found before merge:

1. --rpc previously defaulted to the three public testnet endpoints,
   so a bare 'sweep --from 0 --to head' (the exact WP 7902 full-sweep
   invocation) would quietly page the public RPC for the entire
   0-62,000,000 prefix -- directly contradicting the brief ('the
   public RPC is rate-limited and must not be used for the sweep').
   --rpc now has no default; the sweep is local-node-only unless a
   public endpoint is passed explicitly, and logs plainly when that
   leaves a prune-boundary gap unscanned. A regression test pins the
   no-default behavior. docs/discovery.md §8 documents the rationale
   and the two narrower cases --rpc remains useful for (a bounded
   dev/sample run below the prune boundary; the enrichment pass's
   getCode-based creation-block lookup, which is O(dozens) of calls
   per pool, not a log sweep).

2. Replacing config/pools.testnet.json with the 229-pool sample broke
   revenue-director's pinned bot/tests/fixtures/cycles/testnet_registry_usdc.json
   (Tier A CI is require-all, so this blocked merge regardless of
   which lane caused it) -- reverted config/pools.testnet.json to
   its committed state and dropped config/tokens.testnet.json from
   this PR. The full registry replacement + a coordinated fixture
   regeneration lands together in the WP 7902 PR. docs/discovery.md
   §9 records the sample run's real numbers as evidence without
   committing the file here.

make test: 217/217 (was 213/216 before this fix -- 3 cycles.py
failures caused by point 2).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: BuddySan <[email protected]>

buddysan commented 2026-09-16 14:40:50 +00:00

Author

Owner

Copy link

Fix-forward pushed (no rebase): (1) --rpc no longer defaults to the public endpoints — the WP 7902 full sweep brief says the public RPC must not be used for the sweep, and my earlier default silently violated that for any bare sweep --from 0 --to head run; now local-node-only unless --rpc is passed explicitly, with a regression test pinning it. (2) Reverted config/pools.testnet.json to its committed state and dropped config/tokens.testnet.json from this PR — replacing the registry here broke bot/tests/test_cycles.py's pinned fixture (outside my file set), which is a real cross-lane regression on a Tier A require-all repo. The 229-pool sample-run evidence is recorded in docs/discovery.md §9 instead; the actual registry replacement + a coordinated fixture regen lands in the WP 7902 PR. make test: 217/217 green now (was 213/216).

Fix-forward pushed (no rebase): (1) `--rpc` no longer defaults to the public endpoints — the WP 7902 full sweep brief says the public RPC must not be used for the sweep, and my earlier default silently violated that for any bare `sweep --from 0 --to head` run; now local-node-only unless `--rpc` is passed explicitly, with a regression test pinning it. (2) Reverted `config/pools.testnet.json` to its committed state and dropped `config/tokens.testnet.json` from this PR — replacing the registry here broke `bot/tests/test_cycles.py`'s pinned fixture (outside my file set), which is a real cross-lane regression on a Tier A require-all repo. The 229-pool sample-run evidence is recorded in `docs/discovery.md` §9 instead; the actual registry replacement + a coordinated fixture regen lands in the WP 7902 PR. `make test`: 217/217 green now (was 213/216).

buddysan added 1 commit 2026-09-16 14:52:17 +00:00

fix(bot): precise prune-boundary bisection; curve_emitter_pass never silently skips ... 64801fd915

AAAA class-mapping follow-up to the previous commit. Two real defects
found by writing a test for the curve emitter pass's honest-reporting
fix, both instances of the same class:

1. curve_emitter_pass silently absorbed Pruned errors with no log
   line, and reported curve_window as the naively REQUESTED range
   (e.g. [0, head]) rather than what was actually scanned -- a false
   completeness claim in the registry envelope. Now logs every
   pruned/unscanned span and records the real scanned window plus
   curve_window_pruned_ranges.

2. Both Pruned handlers (EndpointPool.fetch, curve_emitter_pass) then
   treated a WHOLE failed chunk as pruned and jumped to hi+1. The
   real node's prune rule keys on  alone (verified: a
   [0,100000] query and a [30000000,30100000] query fail identically
   even though the second is 30M blocks closer to the retained
   window) -- so a chunk that straddles the true boundary would have
   its fully-servable tail silently skipped. Currently masked in this
   repo's own runs by round-number luck (100,000-block chunks against
   a boundary that happens to be an exact multiple), but a latent
   data-loss bug for any non-aligned --from or a drifting boundary.

   Fixed with a shared pruned_boundary() bisection (single-fromBlock
   probes, since success is monotonic in fromBlock) used by both call
   sites: on Pruned, the endpoint's pruned_before converges on the
   true cut and the retry resumes from exactly there, never losing a
   servable block.

Two existing tests encoded the old (buggy) wholesale-skip behavior as
expected output and needed updating to the correct, tighter values
(pruned_before=250 not 300 for a straddling [200,299] chunk); one new
test pins the honest-reporting behavior end to end.

Re-ran the WP 7902 sweep with this fix: 983 pools (was 982/229 across
earlier runs), identical scanned window [62,000,000, head] -- this
repo's own boundary is round-number-aligned so the fix changes no
delivered data here, only proves the class is closed.

make test: 218/218.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: BuddySan <[email protected]>

buddysan added 1 commit 2026-09-16 14:56:57 +00:00

fix(bot): test used /dev/null as a checkpoint path, breaks on a real filesystem ... 74736703bd

Root cause of the CI python-tests failure on the previous two pushes
(not a flake, not an environment difference -- reproduced by running
the exact suite on LXC 140's real Python 3.12.3 filesystem, which the
sandbox this was written in did not catch): the curve_emitter_pass
honest-reporting test passed '/dev/null' as a checkpoint_path
throwaway, but save_checkpoint() legitimately writes 'checkpoint_path
+ ".tmp"' then os.replace()s it onto checkpoint_path -- os.replace
onto /dev/null (a character device) raises OSError: [Errno 16] Device
or resource busy on a real filesystem. Fixed by using a real
tempfile.TemporaryDirectory() path instead.

Verified on LXC 140 (fleet Python 3.12.3, matches the CI runner):
make test now 218/218 there too, not just in the dev sandbox
(Python 3.13.5, where the bug did not reproduce).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: BuddySan <[email protected]>

buddysan commented 2026-09-16 14:57:03 +00:00

Author

Owner

Copy link

Found and fixed the real cause of the python-tests failure (not a flake): my own test passed /dev/null as a throwaway checkpoint path, and save_checkpoint() legitimately os.replace()s onto it — works by luck on some filesystems, raises OSError: [Errno 16] Device or resource busy on a real one. Reproduced on LXC 140's Python 3.12.3 (did not reproduce in my dev sandbox's 3.13.5 — good reminder to verify on the fleet's pinned version, not just locally). Fixed with a real tempfile path; verified 218/218 on both. The forge-tests failure on the same run looks unrelated to this lane (no contracts/ files touched) — will keep an eye on it but not touching that code.

Found and fixed the real cause of the python-tests failure (not a flake): my own test passed `/dev/null` as a throwaway checkpoint path, and `save_checkpoint()` legitimately `os.replace()`s onto it — works by luck on some filesystems, raises `OSError: [Errno 16] Device or resource busy` on a real one. Reproduced on LXC 140's Python 3.12.3 (did not reproduce in my dev sandbox's 3.13.5 — good reminder to verify on the fleet's pinned version, not just locally). Fixed with a real tempfile path; verified 218/218 on both. The forge-tests failure on the same run looks unrelated to this lane (no `contracts/` files touched) — will keep an eye on it but not touching that code.

buddysan added 1 commit 2026-09-16 14:58:25 +00:00

docs(discovery): real measured runtime + public-provider limits tables (no code change) ... 72f72338c5

Replaces placeholder-ish §4/§7 prose with the actual numbers from the
WP 7902 local-only full sweep (~30 s end to end, 62,000,000->head)
and a measured per-provider limits table, now that §8 has settled the
--rpc-is-opt-in design.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Signed-off-by: BuddySan <[email protected]>

buddysan referenced this pull request from a commit 2026-09-16 15:58:26 +00:00

merge: resolve docs/architecture.md conflict between PR #18 (fee_bps sweep) and main

buddysan added 1 commit 2026-09-16 15:58:26 +00:00

merge: resolve docs/architecture.md conflict between PR #18 (fee_bps sweep) and main ... 1f3882bcb5

Both sides had real, non-duplicate content: kept main's metrics/endpoint-latency
rows (PR #8) and replaced main's superseded 'fee_bps=30 on all pools' claim with
PR #18's full-factory-sweep finding (30bps standard, one factory at 50bps).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Signed-off-by: BuddySan <[email protected]>

fortiblox-dev approved these changes 2026-09-16 16:02:00 +00:00

fortiblox-dev left a comment

Copy link

automerger v2 auto-approved: all 7 gates pass. Approver: fortiblox-dev (opposite-identity rule). Gates: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker.

automerger v2 auto-approved: all 7 gates pass. Approver: fortiblox-dev (opposite-identity rule). Gates: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker.

buddysan merged commit 1f3882bcb5 into main 2026-09-16 16:02:01 +00:00

buddysan deleted branch data/discover-v3-curve 2026-09-16 16:02:01 +00:00

buddysan commented 2026-09-16 16:02:01 +00:00

Author

Owner

Copy link

Auto-merged by forgejo-automerger at 2026-09-16 16:02:01 UTC. Approver: fortiblox-dev (opposite-identity rule, Option C upgrade 2026-05-13). Merger: buddysan via fast-forward-only (NOT admin force_merge). Criteria passed: mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker. Branch 1f3882bcb5 merged onto main and feature branch deleted. See project_forgejo_automerger memory for rationale.

Auto-merged by **forgejo-automerger** at 2026-09-16 16:02:01 UTC. **Approver:** `fortiblox-dev` (opposite-identity rule, Option C upgrade 2026-05-13). **Merger:** `buddysan` via `fast-forward-only` (NOT admin `force_merge`). **Criteria passed:** `mergeable=true,label=auto-merge,author=buddysan,identity=ok,status_checks=not_required,signatures=upstream_BP_only,opt_in=repo_marker`. Branch `1f3882bcb5` merged onto `main` and feature branch deleted. See `project_forgejo_automerger` memory for rationale.

Sign in to join this conversation.

Reviewers

No reviewers

fortiblox-dev

Labels

Clear labels   

auto-merge

Opt-in: automerger will land this PR after CI green + all gates pass

  

auto-merge

Automerge when CI passes (set by webhook bridge)

No labels

auto-merge

auto-merge

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

fortiblox/arc-mev-bot!18

No description provided.