brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 13:44:24 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(kb/gitlab): review and rearrange contents

Browse Source
This commit is contained in:
Michele Cereda
2024-02-21 00:24:46 +01:00
parent 087bdc340d
commit 3d85b8f6a9
1 changed files with 357 additions and 284 deletions
Download Patch File Download Diff File Expand all files Collapse all files

641
knowledge base/gitlab.md
View File

@@ -1,19 +1,358 @@
# Gitlab
## Table of contents <!-- omit in toc -->
1. [Omnibus](#omnibus)
1. [Kubernetes](#kubernetes)
1. [Helm chart](#helm-chart)
1. [Operator](#operator)
1. [CI/CD](#cicd)
1. [Make a job in a pipeline run only when some specific files change](#make-a-job-in-a-pipeline-run-only-when-some-specific-files-change)
1. [Get the version of the helper image to use for a runner](#get-the-version-of-the-helper-image-to-use-for-a-runner)
1. [Helm chart](#helm-chart)
1. [Chart maintenance](#chart-maintenance)
1. [Chart deployment](#chart-deployment)
1. [Minikube](#minikube)
1. [Kubernetes operator](#kubernetes-operator)
1. [Manage kubernetes clusters](#manage-kubernetes-clusters)
1. [Gotchas](#gotchas)
1. [Further readings](#further-readings)
1. [Sources](#sources)
1. [Sources](#sources)
## Omnibus
<details>
<summary>Configuration</summary>
```sh
# Edit and validate.
sudo vim '/etc/gitlab/gitlab.rb'
sudo ruby -c '/etc/gitlab/gitlab.rb'
# Make Gitlab aware of the changes.
sudo gitlab-ctl reconfigure
```
Backup settings for AWS buckets.</br>
See [Back up Gitlab using Amazon S3]:
```rb
# If using an IAM Profile, don't configure 'aws_access_key_id' and
# 'aws_secret_access_key' but set "'use_iam_profile' => true" instead.
gitlab_rails['backup_upload_connection'] = {
'provider' => 'AWS',
'region' => 'eu-west-1',
'aws_access_key_id' => 'AKIAKIAKI',
'aws_secret_access_key' => 'secret123'
}
# It appears one can use prefixes by appending them to the bucket name.
# See https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3376.
gitlab_rails['backup_upload_remote_directory'] = 'bucket-name/prefix'
# Use multipart uploads when the archive's size exceeds 100MB.
gitlab_rails['backup_multipart_chunk_size'] = 104857600
# Only keep 7 days worth of backups.
gitlab_rails['backup_keep_time'] = 604800
```
</details>
<details>
<summary>Maintenance</summary>
```sh
# Create backups.
sudo gitlab-backup create BACKUP='prefix_override' STRATEGY='copy'
# Create empty backup archives for testing purposes.
# See https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#excluding-specific-data-from-the-backup
sudo gitlab-backup create … \
SKIP='db,repositories,uploads,builds,artifacts,pages,lfs,terraform_state,registry,packages,ci_secure_files'
```
</details>
## Kubernetes
### Helm chart
Gitlab offers an official helm chart to to allow for deployments on kubernetes clusters.
Follow the [deployment] guide for details and updated information.
<details>
<summary>Deployment</summary>
1. Prepare the environment:
```sh
export \
ENVIRONMENT='minikube' \
NAMESPACE='gitlab' \
VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab"
```
1. Validate the values and install the chart.
> The installation took > 20m on a MacBook Pro 16-inch 2019 with Intel i7 and 16GB RAM.
```sh
# validation
helm upgrade --install \
--namespace "${NAMESPACE}" \
--values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
'gitlab' \
'gitlab/gitlab' \
--dry-run
# installation
helm upgrade --install \
--atomic \
--create-namespace \
--namespace "${NAMESPACE}" \
--timeout 0 \
--values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
'gitlab' \
'gitlab/gitlab' \
--debug
```
1. Keep an eye on the installation:
```sh
kubectl get events \
--namespace "${NAMESPACE}" \
--sort-by '.metadata.creationTimestamp' \
--watch
# requires `watch` from 'procps-ng'
# `brew install watch`
watch kubectl get all --namespace "${NAMESPACE}"
# requires `k9s`
# `brew install k9s`
k9s --namespace "${NAMESPACE}"
```
1. Get the login password for the `root` user:
```sh
kubectl get secret 'gitlab-gitlab-initial-root-password' \
--namespace "${NAMESPACE}" \
-o jsonpath='{.data.password}' \
| base64 --decode
```
1. Open the login page:
```sh
export URL="https://$(kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $2}')"
xdg-open "${URL}" # on linux
open "${URL}" # on mac os x
```
1. Have fun!
To delete everything:
```sh
helm uninstall --namespace "${NAMESPACE}" 'gitlab'
kubectl delete --ignore-not-found namespace "${NAMESPACE}"
```
</details>
<details>
<summary>Maintenance</summary>
- Add and update Gitlab's helm repository:
```sh
helm repo add 'gitlab' 'https://charts.gitlab.io/'
helm repo update
```
- Look up the chart's version:
```sh
$ helm search repo 'gitlab/gitlab'
NAME CHART VERSION APP VERSION DESCRIPTION
gitlab/gitlab 4.9.3 13.9.3 Web-based Git-repository manager with wiki and ...
```
- Fetch the chart:
```sh
helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR"
helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION"
```
- Get the default values for the chart:
```sh
helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml"
helm inspect values --version "$CHART_VERSION" 'gitlab/gitlab' > "${VALUES_DIR}/values-${CHART_VERSION}.yaml"
```
```sh
export VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab"
helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml"
```
- Create a dedicated values file with the changes one needs (see the helm chart gotchas below):
```yaml
global:
edition: ce
ingress:
configureCertmanager: false
time_zone: UTC
certmanager:
install: false
gitlab-runner:
install: false
```
- Upgrade the stored chart to a new version:
```sh
helm repo update
rm -r "${CHART_DIR}/gitlab"
helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION"
```
</details>
<details>
<summary>Minikube</summary>
When testing with a minikube installation with 8GiB RAM, kubernetes complained being out of memory.<br/>
Be sure to give your cluster enough resources:
```sh
# on linux
minikube start --kubernetes-version "${K8S_VERSION}" --cpus '4' --memory '12GiB'
# on mac os x
minikube start --kubernetes-version "${K8S_VERSION}" --cpus '8' --memory '12GiB' # docker-desktop (no Ingresses)
minikube start --kubernetes-version "${K8S_VERSION}" --cpus '8' --memory '12GiB' --vm # hyperkit vm (to be able to use Ingresses)
```
or consider using the [minimal Minikube example values file] as reference, as stated in [CPU and RAM Resource Requirements](https://docs.gitlab.com/charts/installation/deployment.html#cpu-and-ram-resource-requirements)
1. Finish preparing the environment:
```sh
export K8S_VERSION='v1.16.15'
```
1. Enable the `ingress` and `metrics-server` addons:
```sh
minikube addons enable 'ingress'
minikube addons enable 'metrics-server'
```
1. When using the `LoadBalancer` Ingress type (the default), start a tunnel in a different shell to let the installation finish:
```sh
minikube tunnel -c
```
1. Install the chart as described above.
1. Add minikube's IP address to the `/etc/hosts` file:
```sh
kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $3 " " $2}' | sudo tee -a '/etc/hosts'
```
</details>
<details>
<summary>Gotchas</summary>
- Use self-signed certs and avoid using certmanager setting up the following:
```yaml
global:
ingress:
configureCertmanager: false
certmanager:
install: false
```
- Avoid using a load balancer (mainly for local testing) setting the ingress type to `NodePort`:
```yaml
nginx-ingress:
controller:
service:
type: NodePort
```
- As of 2021-01-15, a clean minikube cluster with only gitlab installed takes up about 1 vCPU and 6+ GiB RAM:
```sh
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
minikube 965m 12% 6375Mi 53%
```
```sh
$ kubectl get pods --namespace 'gitlab'
NAMESPACE NAME READY STATUS RESTARTS AGE
gitlab gitlab-gitaly-0 1/1 Running 0 71m
gitlab gitlab-gitlab-exporter-547cf7fbff-xzqjp 1/1 Running 0 71m
gitlab gitlab-gitlab-shell-5c5b8dd9cd-g4z7b 1/1 Running 0 71m
gitlab gitlab-gitlab-shell-5c5b8dd9cd-ppbtk 1/1 Running 0 71m
gitlab gitlab-migrations-2-j6lt6 0/1 Completed 0 8m27s
gitlab gitlab-minio-6dd7d96ddb-xxq9w 1/1 Running 0 71m
gitlab gitlab-minio-create-buckets-2-q5zfg 0/1 Completed 0 8m27s
gitlab gitlab-nginx-ingress-controller-7fc8cbf49d-b9lqm 1/1 Running 0 71m
gitlab gitlab-nginx-ingress-controller-7fc8cbf49d-ng589 1/1 Running 0 71m
gitlab gitlab-nginx-ingress-default-backend-7ff88b95f-lv5vt 1/1 Running 0 71m
gitlab gitlab-postgresql-0 2/2 Running 0 71m
gitlab gitlab-prometheus-server-6cfb57f575-cs669 2/2 Running 0 71m
gitlab gitlab-redis-master-0 2/2 Running 0 71m
gitlab gitlab-registry-6c75496fc7-fgbvb 1/1 Running 0 8m16s
gitlab gitlab-registry-6c75496fc7-fhsqs 1/1 Running 0 8m27s
gitlab gitlab-sidekiq-all-in-1-v1-64b9c56675-lf29p 1/1 Running 0 8m27s
gitlab gitlab-task-runner-7897bb897d-br5g5 1/1 Running 0 7m54s
gitlab gitlab-webservice-default-7846fb55d6-4pspg 2/2 Running 0 7m37s
gitlab gitlab-webservice-default-7846fb55d6-tmjqm 2/2 Running 0 8m27s
```
with a **spike** of 5 vCPUs upon installation (specially for sidekiq). Keep this in mind when sizing the test cluster
- Disable TLS setting up the following values:
```yaml
global:
hosts:
https: false
ingress:
tls:
enabled: false
```
- Use a suffix in the ingresses hosts setting up the `global.hosts.hostSuffix` value:
```sh
$ helm template \
--namespace "${NAMESPACE}" \
--values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
--set global.hosts.hostSuffix='test' \
'gitlab' \
'gitlab/gitlab' \
| yq -r 'select(.kind == "Ingress") | .spec.rules[].host' -
gitlab-test.f.q.dn
minio-test.f.q.dn
registry-test.f.q.dn
```
</details>
### Operator
See the [operator guide] and the [operator code] for details.
## CI/CD
@@ -70,193 +409,6 @@ Status: Downloaded newer image for gitlab/gitlab-runner-helper:x86_64-8fa89735
docker.io/gitlab/gitlab-runner-helper:x86_64-8fa89735
```
## Helm chart
Gitlab offers an official helm chart to offer its services on a kubernetes cluster.
Follow the [deployment] guide for details.
### Chart maintenance
- add and update Gitlab's helm repository:
```sh
helm repo add 'gitlab' 'https://charts.gitlab.io/'
helm repo update
```
- lookup the chart's version:
```sh
$ helm search repo 'gitlab/gitlab'
NAME CHART VERSION APP VERSION DESCRIPTION
gitlab/gitlab 4.9.3 13.9.3 Web-based Git-repository manager with wiki and ...
```
- fetch the chart:
```sh
helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR"
helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION"
```
- get the default values for the chart:
```sh
helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml"
helm inspect values --version "$CHART_VERSION" 'gitlab/gitlab' > "${VALUES_DIR}/values-${CHART_VERSION}.yaml"
```
```sh
export VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab"
helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml"
```
- create a dedicated values file with the changes one needs (see [gotchas](#gotchas)):
```yaml
global:
edition: ce
ingress:
configureCertmanager: false
time_zone: UTC
certmanager:
install: false
gitlab-runner:
install: false
```
- upgrade the stored chart to a new version:
```sh
helm repo update
rm -r "${CHART_DIR}/gitlab"
helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION"
```
### Chart deployment
1. prepare the environment:
```sh
export \
ENVIRONMENT='minikube' \
NAMESPACE='gitlab' \
VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab"
```
1. validate the values and install the chart (took > 20m on a MacBook Pro 16-inch 2019 with Intel i7 and 16GB RAM):
```sh
# validation
helm upgrade --install \
--namespace "${NAMESPACE}" \
--values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
'gitlab' \
'gitlab/gitlab' \
--dry-run
# installation
helm upgrade --install \
--atomic \
--create-namespace \
--namespace "${NAMESPACE}" \
--timeout 0 \
--values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
'gitlab' \
'gitlab/gitlab' \
--debug
```
1. keep an eye on the installation:
```sh
kubectl get events \
--namespace "${NAMESPACE}" \
--sort-by '.metadata.creationTimestamp' \
--watch
# requires `watch` from 'procps-ng' (`brew install watch`)
watch kubectl get all --namespace "${NAMESPACE}"
# requires `k9s` (`brew install k9s`)
k9s --namespace "${NAMESPACE}"
```
1. get the login password for user `root`:
```sh
kubectl get secret 'gitlab-gitlab-initial-root-password' \
--namespace "${NAMESPACE}" \
-o jsonpath='{.data.password}' \
| base64 --decode
```
1. open the login page:
```sh
export URL="https://$(kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $2}')"
xdg-open "${URL}" # on linux
open "${URL}" # on mac os x
```
1. have fun!
To delete everything:
```sh
helm uninstall --namespace "${NAMESPACE}" 'gitlab'
kubectl delete --ignore-not-found namespace "${NAMESPACE}"
```
### Minikube
When testing with a minikube installation with 8GiB RAM, kubernetes complained being out of memory.<br/>
Be sure to give your cluster enough resources:
```sh
# on linux
minikube start --kubernetes-version "${K8S_VERSION}" --cpus 4 --memory 12GiB
# on mac os x
minikube start --kubernetes-version "${K8S_VERSION}" --cpus 8 --memory 12GiB # docker-desktop (no Ingresses)
minikube start --kubernetes-version "${K8S_VERSION}" --cpus 8 --memory 12GiB --vm # hyperkit vm (to be able to use Ingresses)
```
or consider using the [minimal Minikube example values file] as reference, as stated in [CPU and RAM Resource Requirements](https://docs.gitlab.com/charts/installation/deployment.html#cpu-and-ram-resource-requirements)
1. finish preparing the environment:
```sh
export K8S_VERSION='v1.16.15'
```
1. enable the `ingress` and `metrics-server` addons:
```sh
minikube addons enable 'ingress'
minikube addons enable 'metrics-server'
```
1. to use the `LoadBalancer` Ingress type (the default), start a tunnel in a different shell to let the installation finish:
```sh
minikube tunnel -c
```
1. install the chart as described [above](#chart-deployment)
1. add minikube's IP address to the `/etc/hosts` file:
```sh
kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $3 " " $2}' | sudo tee -a '/etc/hosts'
```
## Kubernetes operator
See the [operator guide] and the [operator code] for details.
## Manage kubernetes clusters
See [adding and removing kubernetes clusters] for more information.
@@ -265,87 +417,7 @@ For now the Gitlab instance can manage only kubernetes clusters external to the
## Gotchas
- use self-signed certs and avoid using certmanager setting up the following:
```yaml
global:
ingress:
configureCertmanager: false
certmanager:
install: false
```
- avoid using a load balancer (mainly for local testing) setting the ingress type to `NodePort`:
```yaml
nginx-ingress:
controller:
service:
type: NodePort
```
- as of 2021-01-15, a clean minikube cluster with only gitlab installed takes up about 1 vCPU and 6+ GiB RAM:
```sh
$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
minikube 965m 12% 6375Mi 53%
```
```sh
$ kubectl get pods --namespace gitlab
NAMESPACE NAME READY STATUS RESTARTS AGE
gitlab gitlab-gitaly-0 1/1 Running 0 71m
gitlab gitlab-gitlab-exporter-547cf7fbff-xzqjp 1/1 Running 0 71m
gitlab gitlab-gitlab-shell-5c5b8dd9cd-g4z7b 1/1 Running 0 71m
gitlab gitlab-gitlab-shell-5c5b8dd9cd-ppbtk 1/1 Running 0 71m
gitlab gitlab-migrations-2-j6lt6 0/1 Completed 0 8m27s
gitlab gitlab-minio-6dd7d96ddb-xxq9w 1/1 Running 0 71m
gitlab gitlab-minio-create-buckets-2-q5zfg 0/1 Completed 0 8m27s
gitlab gitlab-nginx-ingress-controller-7fc8cbf49d-b9lqm 1/1 Running 0 71m
gitlab gitlab-nginx-ingress-controller-7fc8cbf49d-ng589 1/1 Running 0 71m
gitlab gitlab-nginx-ingress-default-backend-7ff88b95f-lv5vt 1/1 Running 0 71m
gitlab gitlab-postgresql-0 2/2 Running 0 71m
gitlab gitlab-prometheus-server-6cfb57f575-cs669 2/2 Running 0 71m
gitlab gitlab-redis-master-0 2/2 Running 0 71m
gitlab gitlab-registry-6c75496fc7-fgbvb 1/1 Running 0 8m16s
gitlab gitlab-registry-6c75496fc7-fhsqs 1/1 Running 0 8m27s
gitlab gitlab-sidekiq-all-in-1-v1-64b9c56675-lf29p 1/1 Running 0 8m27s
gitlab gitlab-task-runner-7897bb897d-br5g5 1/1 Running 0 7m54s
gitlab gitlab-webservice-default-7846fb55d6-4pspg 2/2 Running 0 7m37s
gitlab gitlab-webservice-default-7846fb55d6-tmjqm 2/2 Running 0 8m27s
```
with a **spike** of 5 vCPUs upon installation (specially for sidekiq). Keep this in mind when sizing the test cluster
- disable TLS setting up the following values:
```yaml
global:
hosts:
https: false
ingress:
tls:
enabled: false
```
- use a suffix in the ingresses hosts setting up the `global.hosts.hostSuffix` value:
```sh
$ helm template \
--namespace "${NAMESPACE}" \
--values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
--set global.hosts.hostSuffix="test" \
'gitlab' \
'gitlab/gitlab' \
| yq -r 'select(.kind == "Ingress") | .spec.rules[].host' -
gitlab-test.f.q.dn
minio-test.f.q.dn
registry-test.f.q.dn
```
- use an access token to clone a project
- Use access tokens to clone projects:
```sh
git clone "https://oauth2:${ACCESS_TOKEN}@somegitlab.com/vendor/package.git"
@@ -362,12 +434,13 @@ For now the Gitlab instance can manage only kubernetes clusters external to the
- [Adding and removing Kubernetes clusters]
- Gitlab's [operator code] and relative [guide][operator guide]
## Sources
All the references in the [further readings] section, plus the following:
### Sources
- [Configuring private dns zones and upstream nameservers in kubernetes]
- [Using GitLab token to clone without authentication]
- [Back up GitLab Using Amazon S3]
- [Support object storage bucket prefixes]
- [Back up GitLab excluding specific data from the backup]
<!--
References
@@ -375,19 +448,19 @@ All the references in the [further readings] section, plus the following:
<!-- Upstream -->
[adding and removing kubernetes clusters]: https://docs.gitlab.com/ee/user/project/clusters/add_remove_clusters.html
[back up gitlab excluding specific data from the backup]: https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#excluding-specific-data-from-the-backup
[back up gitlab using amazon s3]: https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html?tab=Linux+package+%28Omnibus%29#using-amazon-s3
[chart]: https://docs.gitlab.com/charts/
[command-line options]: https://docs.gitlab.com/charts/installation/command-line-options.html
[deployment]: https://docs.gitlab.com/charts/installation/deployment.html
[global settings]: https://docs.gitlab.com/charts/charts/globals.html
[minimal minikube example values file]: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/values-minikube-minimum.yaml
[operator code]: https://gitlab.com/gitlab-org/gl-openshift/gitlab-operator
[operator guide]: https://docs.gitlab.com/charts/installation/operator.html
[operator code]: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator
[operator guide]: https://docs.gitlab.com/operator/
[runners on kubernetes]: https://docs.gitlab.com/runner/install/kubernetes.html
[support object storage bucket prefixes]: https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3376
[tls]: https://docs.gitlab.com/charts/installation/tls.html
<!-- In-article sections -->
[further readings]: #further-readings
<!-- Others -->
[configuring private dns zones and upstream nameservers in kubernetes]: https://kubernetes.io/blog/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes/
[using gitlab token to clone without authentication]: https://stackoverflow.com/questions/25409700/using-gitlab-token-to-clone-without-authentication#29570677