Skip to content

Spark

Spark CI on GitHub Actions

Spark jobs in CI spend most of their time on JVM startup and jar assembly.

5m → 45s
Assembly

Spark jobs in CI spend most of their time on JVM startup and jar assembly.

Why it happens

The assembly jar is rebuilt each run and the Ivy or Coursier cache is cold.

What to change

  1. Cache the Ivy and Coursier directories keyed on the build definition
  2. Build the assembly jar once and reuse it across test jobs as an artefact
  3. Run Spark in local mode with a fixed parallelism for deterministic timing

Spark on runnerhut

yaml
jobs:
test:
runs-on: runnerhut-8vcpu-ubuntu-2404
steps:
- uses: actions/checkout@v4
- run: sbt assembly
- run: spark-submit --master 'local[8]' target/app.jar
A working starting point

The only runnerhut-specific line is `runs-on: runnerhut-8vcpu-ubuntu-2404`. Everything else is standard GitHub Actions — the same actions, the same secrets and the same permissions model you use today.

Your next build could be twice as fast, at half the price

Start free. Migrating away is the same one line, and we publish that diff too.