e2e suite takes 10 minutes, one test at a time #246

closed cmc opened this on 2026-09-21 21:22 UTC · builds · assigned to cmc

Discussion

cmc 2026-09-21 21:22 UTC

The test job takes about 11m30s and grows with every test added. Nearly all of it is one package run one test at a time.

Measured locally (12 cores, go test ./... -count=1, 637s total):

  • e2e is 604s of it. The other 27 packages finish in about 40s, in parallel, and are not the problem.
  • 207 tests, median 2.48s, p90 4.49s. A flat tail, no hot spot.
  • Every test calls startInstance, which runs go build for gitbayd. Go's build cache covers the compile but not the final link, so that is 208 links of a byte-identical 35MB binary at about 0.8s each.
  • No test calls t.Parallel.

Two changes, both inside e2e:

  1. Build gitbayd, gitbay and gitbay-runner once per process rather than per test.
  2. Run the tests in parallel. Five call t.Setenv, which panics in a parallel test, and stay serial.

freePorts needs replacing first. It picks a port by binding :0, reading the number back and closing, and between that close and the bind inside gitbayd another instance can be handed the same port. Serially that window is never contended. In parallel it is, and the loser does not fail cleanly — waitForPort only asks whether something is listening, so a test whose port was taken talks to another test's server.

Bazel was considered and rejected. Its caching is already covered: the runner mounts a persistent per-repository HOME, so GOCACHE and GOMODCACHE are warm across builds, and -count=1 defeats test-result caching in any build system. Its parallelism is per target, and e2e is one target, so it would need an explicit shard count to do what t.Parallel does. The suite drives real git, ssh, sshd, gpg and podman, which its sandbox is built to prevent.

referenced in commit abe9818bdd by cmc: e2e: drop the minimum-timeout guard

2026-09-21 21:48 UTC

referenced in commit 76df81288e by cmc: e2e: run the tests in parallel

2026-09-21 21:48 UTC

closed by merge request !454: e2e: build binaries once, run the tests in parallel

2026-09-21 21:48 UTC

referenced in commit f8dd492920 by cmc: CHANGELOG: v1.34.0

2026-09-22 04:19 UTC