38bdad0c9f · krz/gitbay

1

# Runner isolation with rootless podman

2

3

Ref #144. Milestone v1.14.0. Splits from #115, whose concurrency half (`-jobs N`)

4

is done.

5

6

## Problem

7

8

A build runs whatever a repository's `.gitbay/ci.yml` says, through `sh -c`, as

9

the runner process's own user (`cmd/gitbay-runner/main.go:309`).

10

11

`deploy/gitbay-runner.override.conf` constrains the *service* —

12

`NoNewPrivileges`, `ProtectSystem=full`, `RestrictSUIDSGID`, CPU and IO weight —

13

and the runner's key carries scope `runner`, so a build cannot administer the

14

instance. Three things it can still do:

15

16

- **Read the runner's SSH private key.** It is on disk, owned by `ci-runner`,

17

and a step runs as `ci-runner`. Scope `runner` is not nothing: it claims

18

builds, posts logs, and marks builds done, for every repository the runner

19

serves.

20

- **Read the runner's environment.** `cmd.Env = append(os.Environ(), …)`

21

(`main.go:311`) hands each step the runner's entire environment, not a

22

constructed one.

23

- **See other builds in flight.** `-jobs N` puts N workspaces under one

24

`-workdir`, all readable by the same user.

25

26

Acceptable while every repository on the instance is the operator's. It stops

27

being acceptable the moment a fork's CI runs, which `registration = "open"`

28

makes reachable.

29

30

## Decision

31

32

Rootless podman, chosen over bubblewrap and a hand-rolled `unshare` sandbox.

33

34

The deciding factor was not isolation strength — bubblewrap would have been

35

enough for that, at one small package with no daemon. It was `image:` per job.

36

An OCI runtime is the only option that gets there, and a CI system where every

37

build runs against whatever the host happens to have installed is a CI system

38

people work around rather than with.

39

40

The cost is real and should be stated rather than discovered: podman is a

41

substantially larger dependency on bay1 than anything the runner has needed so

42

far, it needs subuid/subgid delegation for `ci-runner`, and it introduces image

43

storage that grows without pruning. The runner stops being a single Go binary

44

plus `git`.

45

46

## Design

47

48

### What runs where

49

50

The split matters more than the flags:

51

52

- **The runner clones**, outside any container, using its own key. The container

53

never sees `GIT_SSH_COMMAND`, the key, or the runner's environment.

54

- **The container runs the steps**, with the already-cloned workspace bind

55

mounted read-write at a fixed path.

56

57

That alone closes the key-theft path, independently of how good the sandbox is.

58

59

### Per-step or per-job

60

61

Per job, one container for all of a job's steps. Steps in a job share state

62

today — a build step writes what a test step reads — and per-step containers

63

would break that or force a layer-caching scheme nobody asked for. `sh -c` per

64

step stays, inside the one container.

65

66

### The environment

67

68

Stop inheriting. Build the step environment explicitly: `PATH`, `HOME`, `CI`,

69

the build's own variables, and secrets when `b.Trusted`. `os.Environ()` must not

70

appear in the container's environment.

71

72

Secrets stay env vars inside the container. They are visible to `podman inspect`

73

on the host, which is the runner's own user — the same user that already holds

74

them in memory, so this is not a new exposure. It is worth a comment saying so,

75

because it looks like one.

76

77

### `image:` in ci.yml

78

79

`Job` gains `Image string`. Empty means the instance default, which is

80

configuration on the runner (`-image`), not a value baked into the binary.

81

82

Validation belongs in `ci.Parse` beside the existing job checks: a reference,

83

not a command line. Reject anything with a shell metacharacter or whitespace —

84

this string reaches `podman run`, and the whole point is that a repository's

85

config file cannot become an argument injection.

86

87

### Network

88

89

Default on. A build that cannot fetch dependencies is useless to most projects,

90

and this design's threat model is about what a build can *reach on the host*,

91

not about exfiltration. `network: none` per job is a plausible later addition

92

and explicitly out of scope here.

93

94

### Failure modes

95

96

**Podman missing or broken.** The runner must refuse to run builds rather than

97

falling back to running them unsandboxed. A fallback that silently drops

98

isolation is worse than a stopped runner, because nothing surfaces it. Check at

99

startup, fail loudly, and say what is wrong.

100

101

**Image pull failure.** Fail the build with the pull error in the log. Do not

102

retry indefinitely; do not fall back to another image.

103

104

**Storage growth.** Images accumulate. Prune on a schedule, and document it —

105

an unbounded cache on a 40GB host is a slow outage.

106

107

### Host changes on bay1

108

109

- `apt install podman` (Debian 13 has it; kernel 6.12, cgroup2, user namespaces

110

enabled with `max_user_namespaces` = 31556 — all already verified).

111

- `/etc/subuid` and `/etc/subgid` entries for `ci-runner`.

112

- The systemd override needs `Delegate=yes` for rootless cgroup management, and

113

`ProtectSystem=full` must be checked against podman's storage under

114

`~/.local/share/containers` — it may need an explicit `ReadWritePaths`.

115

116

These are deployment changes, not code, and belong in `deploy/` with the rest.

117

118

## Sequencing

119

120

Three merge requests, because the middle one is the risky one and should be

121

reviewable on its own:

122

123

1. **Environment hygiene.** Stop inheriting `os.Environ()`; construct the step

124

environment explicitly. Independently valuable, no new dependency, and

125

testable today.

126

2. **Podman execution.** `Image` in `ci.yml` with validation, the clone/run

127

split, container invocation, startup check, failure modes.

128

3. **Deployment.** `deploy/` changes, subuid/subgid, the systemd override,

129

pruning, and the operational notes.

130

131

## Tests

132

133

- A step cannot read the runner's SSH key.

134

- A step's environment contains what was constructed and nothing from the

135

runner's own environment.

136

- Two concurrent builds cannot see each other's workspaces.

137

- An `image:` containing a shell metacharacter is refused by `ci.Parse`.

138

- A missing podman fails the runner at startup rather than running a build

139

unsandboxed — the fallback that must not exist.

140

- A pull failure fails the build with the error in the log.

141

142

The isolation tests need podman on the machine running them. Gate them so the

143

suite still passes without it, and make the skip visible: a silently skipped

144

isolation test is how this regresses.

145

146

## What this does not do

147

148

No `network: none`, no per-step images, no image caching strategy beyond

149

pruning, no resource limits per build beyond what the service already applies.

150

Each is a separate decision.

@@ -0,0 +1,150 @@
	1	# Runner isolation with rootless podman
	2
	3	Ref #144. Milestone v1.14.0. Splits from #115, whose concurrency half (`-jobs N`)
	4	is done.
	5
	6	## Problem
	7
	8	A build runs whatever a repository's `.gitbay/ci.yml` says, through `sh -c`, as
	9	the runner process's own user (`cmd/gitbay-runner/main.go:309`).
	10
	11	`deploy/gitbay-runner.override.conf` constrains the service —
	12	`NoNewPrivileges`, `ProtectSystem=full`, `RestrictSUIDSGID`, CPU and IO weight —
	13	and the runner's key carries scope `runner`, so a build cannot administer the
	14	instance. Three things it can still do:
	15
	16	- Read the runner's SSH private key. It is on disk, owned by `ci-runner`,
	17	and a step runs as `ci-runner`. Scope `runner` is not nothing: it claims
	18	builds, posts logs, and marks builds done, for every repository the runner
	19	serves.
	20	- Read the runner's environment. `cmd.Env = append(os.Environ(), …)`
	21	(`main.go:311`) hands each step the runner's entire environment, not a
	22	constructed one.
	23	- See other builds in flight. `-jobs N` puts N workspaces under one
	24	`-workdir`, all readable by the same user.
	25
	26	Acceptable while every repository on the instance is the operator's. It stops
	27	being acceptable the moment a fork's CI runs, which `registration = "open"`
	28	makes reachable.
	29
	30	## Decision
	31
	32	Rootless podman, chosen over bubblewrap and a hand-rolled `unshare` sandbox.
	33
	34	The deciding factor was not isolation strength — bubblewrap would have been
	35	enough for that, at one small package with no daemon. It was `image:` per job.
	36	An OCI runtime is the only option that gets there, and a CI system where every
	37	build runs against whatever the host happens to have installed is a CI system
	38	people work around rather than with.
	39
	40	The cost is real and should be stated rather than discovered: podman is a
	41	substantially larger dependency on bay1 than anything the runner has needed so
	42	far, it needs subuid/subgid delegation for `ci-runner`, and it introduces image
	43	storage that grows without pruning. The runner stops being a single Go binary
	44	plus `git`.
	45
	46	## Design
	47
	48	### What runs where
	49
	50	The split matters more than the flags:
	51
	52	- The runner clones, outside any container, using its own key. The container
	53	never sees `GIT_SSH_COMMAND`, the key, or the runner's environment.
	54	- The container runs the steps, with the already-cloned workspace bind
	55	mounted read-write at a fixed path.
	56
	57	That alone closes the key-theft path, independently of how good the sandbox is.
	58
	59	### Per-step or per-job
	60
	61	Per job, one container for all of a job's steps. Steps in a job share state
	62	today — a build step writes what a test step reads — and per-step containers
	63	would break that or force a layer-caching scheme nobody asked for. `sh -c` per
	64	step stays, inside the one container.
	65
	66	### The environment
	67
	68	Stop inheriting. Build the step environment explicitly: `PATH`, `HOME`, `CI`,
	69	the build's own variables, and secrets when `b.Trusted`. `os.Environ()` must not
	70	appear in the container's environment.
	71
	72	Secrets stay env vars inside the container. They are visible to `podman inspect`
	73	on the host, which is the runner's own user — the same user that already holds
	74	them in memory, so this is not a new exposure. It is worth a comment saying so,
	75	because it looks like one.
	76
	77	### `image:` in ci.yml
	78
	79	`Job` gains `Image string`. Empty means the instance default, which is
	80	configuration on the runner (`-image`), not a value baked into the binary.
	81
	82	Validation belongs in `ci.Parse` beside the existing job checks: a reference,
	83	not a command line. Reject anything with a shell metacharacter or whitespace —
	84	this string reaches `podman run`, and the whole point is that a repository's
	85	config file cannot become an argument injection.
	86
	87	### Network
	88
	89	Default on. A build that cannot fetch dependencies is useless to most projects,
	90	and this design's threat model is about what a build can reach on the host,
	91	not about exfiltration. `network: none` per job is a plausible later addition
	92	and explicitly out of scope here.
	93
	94	### Failure modes
	95
	96	Podman missing or broken. The runner must refuse to run builds rather than
	97	falling back to running them unsandboxed. A fallback that silently drops
	98	isolation is worse than a stopped runner, because nothing surfaces it. Check at
	99	startup, fail loudly, and say what is wrong.
	100
	101	Image pull failure. Fail the build with the pull error in the log. Do not
	102	retry indefinitely; do not fall back to another image.
	103
	104	Storage growth. Images accumulate. Prune on a schedule, and document it —
	105	an unbounded cache on a 40GB host is a slow outage.
	106
	107	### Host changes on bay1
	108
	109	- `apt install podman` (Debian 13 has it; kernel 6.12, cgroup2, user namespaces
	110	enabled with `max_user_namespaces` = 31556 — all already verified).
	111	- `/etc/subuid` and `/etc/subgid` entries for `ci-runner`.
	112	- The systemd override needs `Delegate=yes` for rootless cgroup management, and
	113	`ProtectSystem=full` must be checked against podman's storage under
	114	`~/.local/share/containers` — it may need an explicit `ReadWritePaths`.
	115
	116	These are deployment changes, not code, and belong in `deploy/` with the rest.
	117
	118	## Sequencing
	119
	120	Three merge requests, because the middle one is the risky one and should be
	121	reviewable on its own:
	122
	123	1. Environment hygiene. Stop inheriting `os.Environ()`; construct the step
	124	environment explicitly. Independently valuable, no new dependency, and
	125	testable today.
	126	2. Podman execution. `Image` in `ci.yml` with validation, the clone/run
	127	split, container invocation, startup check, failure modes.
	128	3. Deployment. `deploy/` changes, subuid/subgid, the systemd override,
	129	pruning, and the operational notes.
	130
	131	## Tests
	132
	133	- A step cannot read the runner's SSH key.
	134	- A step's environment contains what was constructed and nothing from the
	135	runner's own environment.
	136	- Two concurrent builds cannot see each other's workspaces.
	137	- An `image:` containing a shell metacharacter is refused by `ci.Parse`.
	138	- A missing podman fails the runner at startup rather than running a build
	139	unsandboxed — the fallback that must not exist.
	140	- A pull failure fails the build with the error in the log.
	141
	142	The isolation tests need podman on the machine running them. Gate them so the
	143	suite still passes without it, and make the skip visible: a silently skipped
	144	isolation test is how this regresses.
	145
	146	## What this does not do
	147
	148	No `network: none`, no per-step images, no image caching strategy beyond
	149	pruning, no resource limits per build beyond what the service already applies.
	150	Each is a separate decision.

Commit 38bdad0c9f

Commit `38bdad0c9f`