Spark
Spark CI on GitHub Actions
Spark jobs in CI spend most of their time on JVM startup and jar assembly.
5m → 45s
Assembly
Spark jobs in CI spend most of their time on JVM startup and jar assembly.
Why it happens
The assembly jar is rebuilt each run and the Ivy or Coursier cache is cold.
What to change
- Cache the Ivy and Coursier directories keyed on the build definition
- Build the assembly jar once and reuse it across test jobs as an artefact
- Run Spark in local mode with a fixed parallelism for deterministic timing
Spark on runnerhut
yaml
jobs: test: runs-on: runnerhut-8vcpu-ubuntu-2404 steps: - uses: actions/checkout@v4 - run: sbt assembly - run: spark-submit --master 'local[8]' target/app.jarThe only runnerhut-specific line is `runs-on: runnerhut-8vcpu-ubuntu-2404`. Everything else is standard GitHub Actions — the same actions, the same secrets and the same permissions model you use today.
Your next build could be twice as fast, at half the price
Start free. Migrating away is the same one line, and we publish that diff too.