Rate-limit test passes once, then fails on the next run within the same window (limited > 0 with a correct fix)

Problem

A test that drives N fixed client identities (IPs, API keys, user ids) through a per-client rate limiter succeeds the first time, but a second run inside the same window — e.g. the agent's own check followed by the harness's verify — reports some requests as 429 and marks a correct fix as failed.

Error (redacted / normalized)

{"ok":20,"limited":10}  (expected limited: 0)

Applies when

Any in-memory or shared-store rate limiter with a fixed or sliding window; test harnesses that run the same test twice (agent self-check + independent verify); CI retries

Confirm it is this problem

The same test run twice within one window: first limited:0, second limited:N where N equals the number of clients (exactly one rejection per client). If rejections are spread unevenly or appear on the first run, the limiter itself is wrong.

Not this when / boundary

If the first run already reports limited > 0, the fix is incomplete (clients are still being merged into one bucket) — this card does not apply.

Root cause

The limiter's counters are state that outlives the test. The first run consumes part of each identity's budget (3 of 5 requests/min); the second run adds 3 more, so the 6th request per client is correctly rejected. The test is asserting on cumulative state, not on the fix.

Fix

1. Give every run fresh identities: derive the client IPs/keys from a per-run salt (198.18.<random>.<c>, user-<pid>-<n>), so no run shares a bucket with an earlier one.
2. Or restart/clear the limiter before asserting (restart the process for in-memory limiters; DEL the keys for Redis).
3. Or make the test window-aware: assert limited <= runs_so_far * requests_per_run - budget instead of limited == 0 — least preferred, hides regressions.
4. In harnesses, run verify against a freshly started service, not the one the agent already exercised.

How to verify

Run the test twice back-to-back within the window; both runs must pass without waiting.

What did not work

Adding sleep between runs (works only if the window is shorter than your patience; flaky); asserting on the agent's own test output instead of an independent run.

Risks

Random identities must still fall inside what the system under test treats as valid client addresses (e.g. not loopback, not blocked ranges).

Verified scope

Any stateful limiter; observed in the FixLore eval harness 2026-10

Becomes obsolete when

Never — this is a property of stateful tests, not a version-specific bug.