guides
Stop retrying flaky tests
Automatic retries convert a real bug into an intermittent one, and train your team to distrust every failure.
Tomás Rivera · Performance engineering · 2026-07-21 · 6 min read
Blanket test retries are one of those fixes that solve the symptom so effectively that the disease becomes permanent.
What retries actually do
A test that fails 20% of the time and is retried three times fails visibly less than 1% of the time. The bug is still there. It is now in production, where it also happens 20% of the time.
What to do instead
- Track failure rate per test so you know which tests are actually flaky
- Quarantine the worst offenders into a non-blocking job — visible, but not blocking merges
- Fix root causes: shared fixtures, real clocks, unseeded randomness, port collisions, test-order dependence
- Retry network operations, which genuinely are unreliable. Do not retry assertions.
Your next build could be twice as fast, at half the price
Start free. Migrating away is the same one line, and we publish that diff too.
Related
CI when your teammates are agentsAI coding agents push far more often than humans. Here is how that changes pipeline design and cost control.Why we publish instructions for leavingEvery migration page on this site includes the diff to migrate back off us. Here is the reasoning.What we learned running a million CI jobsQueue time, cache behaviour, right-sizing and the failure modes that only appear at scale.Docker Hub rate limits are a CI problem, not a Docker problemAnonymous pulls are rate-limited per IP, and CI runners share IPs. Here is how to stop finding out at 3am.