chore(gitlab): signals, improve upgrade commands

knowledge base/gitlab/README.md

@@ -186,6 +186,14 @@ sudo yum check-update
 sudo gitlab-backup create
 tmux new-session -As 'gitlab-upgrade' "sudo yum update 'gitlab-ee'"
 
+# DB version upgrade
+sudo gitlab-ctl pg-upgrade
+sudo gitlab-ctl pg-upgrade -V '16'
+# Check there is enough disk space for two copies of the database
+test $(( $(sudo du -s '/var/opt/gitlab/postgresql/data' | awk '{print $1}') * 2 )) -lt \
+  $(sudo df --output='avail' --direct '/var/opt/gitlab/postgresql/data' | tail -n 1) \
+&& sudo gitlab-ctl pg-upgrade -V '16'
+
 # Reset the root user's password.
 sudo gitlab-rake 'gitlab:password:reset[root]'
 sudo gitlab-rails console
@@ -752,6 +760,7 @@ Solution: set the correct ownership with
 - [GitLab HA Scaling Runner Vending Machine for AWS EC2 ASG]
 - [GitLab maintenance mode]
 - [Forks]
+- [Upgrade packaged PostgreSQL server]
 
 <!--
   Reference
@@ -803,6 +812,7 @@ Solution: set the correct ownership with
 [tls]: https://docs.gitlab.com/charts/installation/tls.html
 [tutorial: use buildah in a rootless container with gitlab runner operator on openshift]: https://docs.gitlab.com/ee/ci/docker/buildah_rootless_tutorial.html
 [uninstall the linux package (omnibus)]: https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/doc/installation/index.md#uninstall-the-linux-package-omnibus
+[upgrade packaged postgresql server]: https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server
 [upgrade path tool]: https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/
 [use kaniko to build docker images]: https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
 

knowledge base/gitlab/runner.md

@@ -42,13 +42,17 @@ gitlab-runner exec docker \
   --env 'DOCKER_AUTH_CONFIG={ "credsStore": "ecr-login" }' \
   --docker-volumes "$HOME/.aws/credentials:/root/.aws/credentials:ro"
   'job-requiring-ecr-access'
+
+# Force a configuration file reload.
+sudo kill -HUP $(pidof 'gitlab-runner')
+sudo kill -s 'SIGHUP' $(pidof 'gitlab-runner')
 ```
 
 </details>
 
 Each runner executor is assigned 1 task at a time by default.
 
-Runners seem to require the main instance to give the full certificate chain upon connection.
+Runners require the main instance to give the full certificate chain upon connection.
 
 The `runners.autoscaler.policy.periods` setting appears to be a full blown cron job, not just a time frame.
 
@@ -78,6 +82,23 @@ Meaning it will reapply policy 1 at the 31st minute of every hour in the period
 
 </details>
 
+One can use system signals to interact with runners.
+
+<details style="margin-top: -1em; padding: 0 0 1em 1em;">
+
+| Signal                 | Command it operates on | Effect                                                                                                        | Example                                |
+| ---------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
+| `SIGINT`               | `register`             | Cancel ongoing runner registrations.<br/>Delete runners if already registered.                                |                                        |
+| `SIGINT`<br/>`SIGTERM` | `run`<br/>`run-single` | Abort all running builds and exit as soon as possible.<br/>Use twice to exit immediately (forceful shutdown). |                                        |
+| `SIGQUIT`              | `run`<br/>`run-single` | Stop accepting new builds and exit as soon as currently running builds finish (graceful shutdown).            | `sudo kill -SIGQUIT <main_runner_pid>` |
+| `SIGHUP`               | run                    | Force reloading the configuration file.                                                                       | `sudo kill -SIGHUP <main_runner_pid>`  |
+
+> Do **not** use `killall` or `pkill` for graceful shutdowns if one is using the `shell` or `docker` executors.<br/>
+> This causes improper handling of the signals due to subprocesses being killed as well. Only use it on the main process
+> handling the jobs.
+
+</details>
+
 ## Pull images from private AWS ECR registries
 
 1. Create an IAM Role in one's AWS account and attach it the
@@ -313,8 +334,8 @@ Procedure:
 
    </details>
 
-1. Install the gitlab runner on the **manager** instance.<br/>
-   Configure it to use the `docker-autoscaler` executor.
+1. Install the gitlab runner on the **manager** instance.
+1. Configure the runner to use the `docker-autoscaler` executor.
 
    <details style="margin-top: -1em; padding-bottom: 1em;">
 
@@ -849,6 +870,7 @@ Improvements:
 - [Autoscaling GitLab Runner on AWS EC2]
 - [Instance executor]
 - [Docker Autoscaler executor]
+- [Signals]
 
 <!--
   Reference
@@ -876,6 +898,7 @@ Improvements:
 [install and register gitlab runner for autoscaling with docker machine]: https://docs.gitlab.com/runner/executors/docker_machine.html
 [install gitlab runner]: https://docs.gitlab.com/runner/install/
 [instance executor]: https://docs.gitlab.com/runner/executors/instance.html
+[signals]: https://docs.gitlab.com/runner/commands/#signals
 [store registration tokens or runner tokens in secrets]: https://docs.gitlab.com/runner/install/kubernetes.html#store-registration-tokens-or-runner-tokens-in-secrets
 
 <!-- Others -->

snippets/ansible/tasks.yml

@@ -668,3 +668,13 @@
         acme_version: 2
         account_key_src: path/to/acme_account.key.pem
         certificate: path/to/certificate.crt.pem
+
+- name: GitLab
+  block:
+    - name: Install configured fleeting plugins
+      when: runner_executor in [ "docker-autoscaler", "instance" ]
+      become: true
+      ansible.builtin.command:
+        chdir: /root
+        cmd: gitlab-runner fleeting install
+        creates: /root/.config/fleeting/plugins

snippets/curl.sh

@@ -3,6 +3,7 @@
 # Sources:
 # - https://everything.curl.dev/usingcurl/connections/name.html
 
+
 # Use different names.
 # Kinda like '--resolve' but to aliases and supports ports.
 curl --connect-to 'super.fake.domain:443:localhost:8443' 'https://super.fake.domain'
@@ -10,3 +11,14 @@ curl --connect-to 'super.fake.domain:443:localhost:8443' 'https://super.fake.dom
 # Forcefully resolve hosts to given addresses.
 # The resolution *must* be an address, not an FQDN.
 curl --resolve 'super.fake.domain:8443:127.0.0.1' 'https://super.fake.domain:8443'
+
+
+curl -fs 'https://gitlab.com/api/v4/runners/all?per_page=100&paused=false' -H 'PRIVATE-TOKEN: glpat-m-…'
+curl --url 'https://gitlab.com/api/v4/runners/all' \
+	--fail --silent \
+	--header 'PRIVATE-TOKEN: glpat-m-…' \
+	--url-query 'per_page=100' --url-query 'paused=false'
+
+curl -fsX 'PUT' 'https://gitlab.com/api/v4/runners/{}' -H 'PRIVATE-TOKEN: glpat-m-…' -F 'paused=true'
+curl --fail --silent --request 'PUT' 'https://gitlab.com/api/v4/runners/{}' \
+	--header 'PRIVATE-TOKEN: glpat-m-…' --form 'paused=true'

snippets/gitlab/gitlab-runner.sh

@@ -25,6 +25,7 @@ gitlab-runner register --url 'https://gitlab.example.com' --registration-token '
 
 # Just list configured runners
 gitlab-runner list -c '/etc/gitlab-runner/config.toml'
+curl -fs 'https://gitlab.com/api/v4/runners/all?per_page=100' -H 'PRIVATE-TOKEN: glpat-m-…'
 
 # Check configured runners can connect to the main instance
 gitlab-runner verify -c '/etc/gitlab-runner/config.toml'
@@ -36,6 +37,20 @@ curl -fs 'https://gitlab.com/api/v4/runners/all?status=offline&per_page=100' -H
 | jq '.[].id' \
 | xargs -I 'runner_id' curl -fsX 'DELETE' "https://gitlab.com/api/v4/runners/runner_id" 'PRIVATE-TOKEN: glpat-m-…'
 
+# Force reloading the configuration file
+sudo kill -HUP $(pidof 'gitlab-runner')
+sudo kill -s 'SIGHUP' $(pgrep 'gitlab-runner')
+
+# Stop accepting new builds and exit as soon as currently running builds finish
+# A.K.A. graceful shutdown
+sudo kill -QUIT $(pgrep 'gitlab-runner')
+sudo kill -s 'SIGQUIT' $(pidof 'gitlab-runner')
+
+# Pause active runners
+curl -fs 'https://gitlab.com/api/v4/runners/all?per_page=100&paused=false' -H 'PRIVATE-TOKEN: glpat-m-…' \
+| jq '.[].id' - \
+| xargs -I '{}' curl -fsX 'PUT' 'https://gitlab.com/api/v4/runners/{}' -H 'PRIVATE-TOKEN: glpat-m-…' -F 'paused=true'
+
 
 ###
 # docker+machine executor

snippets/gitlab/package installation.sh

@@ -121,6 +121,14 @@ sudo gitlab-ctl backup-etc
 tmux new-session -A -s 'gitlab-upgrade' "sudo yum update 'gitlab-ee'"  # 'gitlab-ee-16.11.3' if version-specific
 sudo gitlab-rake 'gitlab:check'
 
+# DB version upgrade
+sudo gitlab-ctl pg-upgrade
+sudo gitlab-ctl pg-upgrade -V '16'
+# Check there is enough disk space for two copies of the database
+test $(( $(sudo du -s '/var/opt/gitlab/postgresql/data' | awk '{print $1}') * 2 )) -lt \
+	$(sudo df --output='avail' --direct '/var/opt/gitlab/postgresql/data' | tail -n 1) \
+&& sudo gitlab-ctl pg-upgrade -V '16'
+
 # Password reset
 sudo gitlab-rake 'gitlab:password:reset[root]'
 sudo gitlab-rails console

snippets/process management.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env sh
+
+# https://askubuntu.com/questions/180336/how-to-find-the-process-id-pid-of-a-running-terminal-program
+# https://bash.cyberciti.biz/guide/Sending_signal_to_Processes
+
+# Find process IDs
+pidof 'gitlab-runner'
+pgrep 'gitlab-runner'
+# Limit search to specific owners
+pgrep -u 'root,daemon' 'sshd'
+
+
+# List available signals
+kill -l
+
+# Send signals to processes
+kill -9 '1234'
+kill -KILL '1234'
+kill -SIGKILL '1234'
+kill -s 'SIGHUP' '3969'
+pkill -KILL 'firefox'
+pkill --signal 'HUP' 'prometheus'
+killall -s 'SIGKILL' 'firefox-bin'