engineering
What we learned running a million CI jobs
Queue time, cache behaviour, right-sizing and the failure modes that only appear at scale.
Priya Raghavan · Infrastructure · 2026-08-04 · 11 min read
A million jobs is enough to see the long tail. Here is what surprised us.
Queue time is invisible and enormous
Step durations are shown prominently. Queue time is not shown at all. We found repositories where median queue time exceeded median build time, and the team had spent a quarter optimising the build.
Most runners are the wrong size in both directions
About 40% of jobs averaged below 30% CPU utilization — paying for idle cores. Another 12% were pinned at 100% for their entire duration and sat on the critical path. Right-sizing in both directions was worth more than any caching change for a majority of customers.
I/O wait is the silent killer
npm writing 40,000 files, Cargo unpacking crates, Docker extracting layers — none of it is CPU work. On network-attached storage these steps dominate, and every CPU graph looks fine while they do.
Flaky tests cost more than anyone models
The direct cost of a rerun is obvious. The indirect cost — engineers retrying real failures because retrying is the habit — is much larger and does not appear on any invoice.
Agent traffic changed the shape of the load
Repositories with AI coding agents show push frequency several times higher than human-only repositories, in bursts. Concurrency cancellation went from a nice optimisation to a hard requirement for these teams.
Your next build could be twice as fast, at half the price
Start free. Migrating away is the same one line, and we publish that diff too.