Rate-limit test passes once, then fails on the next run within the same window (limited > 0 with a correct fix)

问题

A test that drives N fixed client identities (IPs, API keys, user ids) through a per-client rate limiter succeeds the first time, but a second run inside the same window — e.g. the agent's own check followed by the harness's verify — reports some requests as 429 and marks a correct fix as failed.

报错原文（已脱敏/规范化）

{"ok":20,"limited":10}  (expected limited: 0)

适用环境

Any in-memory or shared-store rate limiter with a fixed or sliding window; test harnesses that run the same test twice (agent self-check + independent verify); CI retries

如何确认是这个问题

The same test run twice within one window: first limited:0, second limited:N where N equals the number of clients (exactly one rejection per client). If rejections are spread unevenly or appear on the first run, the limiter itself is wrong.

不是这个问题时 / 边界

If the first run already reports limited > 0, the fix is incomplete (clients are still being merged into one bucket) — this card does not apply.

根因

The limiter's counters are state that outlives the test. The first run consumes part of each identity's budget (3 of 5 requests/min); the second run adds 3 more, so the 6th request per client is correctly rejected. The test is asserting on cumulative state, not on the fix.

修复步骤

1. Give every run fresh identities: derive the client IPs/keys from a per-run salt (198.18.<random>.<c>, user-<pid>-<n>), so no run shares a bucket with an earlier one.
2. Or restart/clear the limiter before asserting (restart the process for in-memory limiters; DEL the keys for Redis).
3. Or make the test window-aware: assert limited <= runs_so_far * requests_per_run - budget instead of limited == 0 — least preferred, hides regressions.
4. In harnesses, run verify against a freshly started service, not the one the agent already exercised.

如何验证

Run the test twice back-to-back within the window; both runs must pass without waiting.

试过没用的办法

Adding sleep between runs (works only if the window is shorter than your patience; flaky); asserting on the agent's own test output instead of an independent run.

风险提示

Random identities must still fall inside what the system under test treats as valid client addresses (e.g. not loopback, not blocked ranges).

验证范围

Any stateful limiter; observed in the FixLore eval harness 2026-10

何时失效

Never — this is a property of stateful tests, not a version-specific bug.