deploy, wiki: KillMode=mixed so a stop reaches only the runner !324
2 files changed, +11 −1
Layout: unified · split
.gitbay/wiki/Admin.org +4 −1
| @@ -478,7 +478,10 @@ run with =-isolation none=, set it back to =yes=. | |||
| 478 | 478 | ||
| 479 | *Restarting the runner is safe.* On SIGTERM it stops claiming, finishes | 479 | *Restarting the runner is safe.* On SIGTERM it stops claiming, finishes |
| 480 | the build in flight, reports it, and exits; the drop-in's | 480 | the build in flight, reports it, and exits; the drop-in's |
| 481 | =TimeoutStopSec=50min= covers the longest build. So =make deploy-runner= | 481 | =TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed= |
| 482 | is what makes the signal reach the runner alone — under systemd's default | ||
| 483 | the build's container and the log session are signalled with it, and the | ||
| 484 | runner drains a build that is already dead. So =make deploy-runner= | ||
| 482 | waits for a running build rather than orphaning it, and a build's result | 485 | waits for a running build rather than orphaning it, and a build's result |
| 483 | is retried for half a minute if gitbayd is restarting at that moment. A | 486 | is retried for half a minute if gitbayd is restarting at that moment. A |
| 484 | second SIGTERM ends the runner at once, abandoning the build to the | 487 | second SIGTERM ends the runner at once, abandoning the build to the |
deploy/gitbay-runner.override.conf +7
| @@ -49,6 +49,13 @@ ExecStopPost=-/usr/bin/pkill -u ci-runner -x catatonit | |||
| 49 | # is 45m plus half a minute of report retries — before systemd kills it. | 49 | # is 45m plus half a minute of report retries — before systemd kills it. |
| 50 | # `make deploy-runner` therefore waits for a running build (#179). | 50 | # `make deploy-runner` therefore waits for a running build (#179). |
| 51 | TimeoutStopSec=50min | 51 | TimeoutStopSec=50min |
| 52 | # The drain only works if the stop signal reaches the runner alone. The | ||
| 53 | # default control-group mode sends SIGTERM to every process in the | ||
| 54 | # cgroup at once — the ssh session streaming the log and the build's | ||
| 55 | # container with it — so the runner drained a build whose steps were | ||
| 56 | # already dead. mixed signals the main process only; whatever is left | ||
| 57 | # when it exits is killed. | ||
| 58 | KillMode=mixed | ||
| 52 | ExecStart= | 59 | ExecStart= |
| 53 | ExecStart=/usr/local/bin/gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work -poll 5s -timeout 45m -repos krz/gitbay,cmc/ci-smoke -isolation podman -image localhost/gitbay-ci:1 -cpus 3 | 60 | ExecStart=/usr/local/bin/gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work -poll 5s -timeout 45m -repos krz/gitbay,cmc/ci-smoke -isolation podman -image localhost/gitbay-ci:1 -cpus 3 |
| 54 | Nice=10 | 61 | Nice=10 |