Guides
How to deal with flaky CI jobs
Tests fail randomly and people retry until they pass.
6 min read
Tests fail randomly and people retry until they pass.
Symptoms
- Same test fails on unchanged code
- Reruns are routine
- Real failures get retried too
Why it happens
Flakes usually come from shared state, timing assumptions or test-order dependence — and retrying hides all three.
How to fix it
- Track failure rate per test to identify the worst offenders
- Quarantine flaky tests into a non-blocking job rather than retrying blindly
- Fix the root cause: shared fixtures, real clocks, unseeded randomness, port collisions
- Never blanket-retry tests — it converts real bugs into intermittent ones
Your next build could be twice as fast, at half the price
Start free. Migrating away is the same one line, and we publish that diff too.
Related
How to collect runner metricsYou cannot tell whether a job is CPU-bound, memory-bound or disk-bound.How to track build duration percentilesAverage build time looks fine but developers still complain.How to export CI traces with OpenTelemetryCI timing data lives in a separate tool from the rest of your observability.How to query CI data programmaticallyYou need CI data in a dashboard or a weekly report.