diff --git a/knowledge base/gitlab.md b/knowledge base/gitlab.md index f1ef33c..87065e2 100644 --- a/knowledge base/gitlab.md +++ b/knowledge base/gitlab.md @@ -1,19 +1,358 @@ # Gitlab -## Table of contents - +1. [Omnibus](#omnibus) +1. [Kubernetes](#kubernetes) + 1. [Helm chart](#helm-chart) + 1. [Operator](#operator) 1. [CI/CD](#cicd) 1. [Make a job in a pipeline run only when some specific files change](#make-a-job-in-a-pipeline-run-only-when-some-specific-files-change) 1. [Get the version of the helper image to use for a runner](#get-the-version-of-the-helper-image-to-use-for-a-runner) -1. [Helm chart](#helm-chart) - 1. [Chart maintenance](#chart-maintenance) - 1. [Chart deployment](#chart-deployment) - 1. [Minikube](#minikube) -1. [Kubernetes operator](#kubernetes-operator) 1. [Manage kubernetes clusters](#manage-kubernetes-clusters) 1. [Gotchas](#gotchas) 1. [Further readings](#further-readings) -1. [Sources](#sources) + 1. [Sources](#sources) + +## Omnibus + +
+ Configuration + +```sh +# Edit and validate. +sudo vim '/etc/gitlab/gitlab.rb' +sudo ruby -c '/etc/gitlab/gitlab.rb' + +# Make Gitlab aware of the changes. +sudo gitlab-ctl reconfigure +``` + +Backup settings for AWS buckets.
+See [Back up Gitlab using Amazon S3]: + +```rb +# If using an IAM Profile, don't configure 'aws_access_key_id' and +# 'aws_secret_access_key' but set "'use_iam_profile' => true" instead. +gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-west-1', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123' +} + +# It appears one can use prefixes by appending them to the bucket name. +# See https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3376. +gitlab_rails['backup_upload_remote_directory'] = 'bucket-name/prefix' + +# Use multipart uploads when the archive's size exceeds 100MB. +gitlab_rails['backup_multipart_chunk_size'] = 104857600 + +# Only keep 7 days worth of backups. +gitlab_rails['backup_keep_time'] = 604800 +``` + +
+ +
+ Maintenance + +```sh +# Create backups. +sudo gitlab-backup create BACKUP='prefix_override' STRATEGY='copy' + +# Create empty backup archives for testing purposes. +# See https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#excluding-specific-data-from-the-backup +sudo gitlab-backup create … \ + SKIP='db,repositories,uploads,builds,artifacts,pages,lfs,terraform_state,registry,packages,ci_secure_files' +``` + +
+ +## Kubernetes + +### Helm chart + +Gitlab offers an official helm chart to to allow for deployments on kubernetes clusters. + +Follow the [deployment] guide for details and updated information. + +
+ Deployment + +1. Prepare the environment: + + ```sh + export \ + ENVIRONMENT='minikube' \ + NAMESPACE='gitlab' \ + VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab" + ``` + +1. Validate the values and install the chart. + + > The installation took > 20m on a MacBook Pro 16-inch 2019 with Intel i7 and 16GB RAM. + + ```sh + # validation + helm upgrade --install \ + --namespace "${NAMESPACE}" \ + --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \ + 'gitlab' \ + 'gitlab/gitlab' \ + --dry-run + + # installation + helm upgrade --install \ + --atomic \ + --create-namespace \ + --namespace "${NAMESPACE}" \ + --timeout 0 \ + --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \ + 'gitlab' \ + 'gitlab/gitlab' \ + --debug + ``` + +1. Keep an eye on the installation: + + ```sh + kubectl get events \ + --namespace "${NAMESPACE}" \ + --sort-by '.metadata.creationTimestamp' \ + --watch + + # requires `watch` from 'procps-ng' + # `brew install watch` + watch kubectl get all --namespace "${NAMESPACE}" + + # requires `k9s` + # `brew install k9s` + k9s --namespace "${NAMESPACE}" + ``` + +1. Get the login password for the `root` user: + + ```sh + kubectl get secret 'gitlab-gitlab-initial-root-password' \ + --namespace "${NAMESPACE}" \ + -o jsonpath='{.data.password}' \ + | base64 --decode + ``` + +1. Open the login page: + + ```sh + export URL="https://$(kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $2}')" + + xdg-open "${URL}" # on linux + open "${URL}" # on mac os x + ``` + +1. Have fun! + +To delete everything: + +```sh +helm uninstall --namespace "${NAMESPACE}" 'gitlab' +kubectl delete --ignore-not-found namespace "${NAMESPACE}" +``` + +
+ +
+ Maintenance + +- Add and update Gitlab's helm repository: + + ```sh + helm repo add 'gitlab' 'https://charts.gitlab.io/' + helm repo update + ``` + +- Look up the chart's version: + + ```sh + $ helm search repo 'gitlab/gitlab' + NAME CHART VERSION APP VERSION DESCRIPTION + gitlab/gitlab 4.9.3 13.9.3 Web-based Git-repository manager with wiki and ... + ``` + +- Fetch the chart: + + ```sh + helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" + helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION" + ``` + +- Get the default values for the chart: + + ```sh + helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml" + helm inspect values --version "$CHART_VERSION" 'gitlab/gitlab' > "${VALUES_DIR}/values-${CHART_VERSION}.yaml" + ``` + + ```sh + export VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab" + helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml" + ``` + +- Create a dedicated values file with the changes one needs (see the helm chart gotchas below): + + ```yaml + global: + edition: ce + ingress: + configureCertmanager: false + time_zone: UTC + certmanager: + install: false + gitlab-runner: + install: false + ``` + +- Upgrade the stored chart to a new version: + + ```sh + helm repo update + rm -r "${CHART_DIR}/gitlab" + helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION" + ``` + +
+ +
+ Minikube + +When testing with a minikube installation with 8GiB RAM, kubernetes complained being out of memory.
+Be sure to give your cluster enough resources: + +```sh +# on linux +minikube start --kubernetes-version "${K8S_VERSION}" --cpus '4' --memory '12GiB' + +# on mac os x +minikube start --kubernetes-version "${K8S_VERSION}" --cpus '8' --memory '12GiB' # docker-desktop (no Ingresses) +minikube start --kubernetes-version "${K8S_VERSION}" --cpus '8' --memory '12GiB' --vm # hyperkit vm (to be able to use Ingresses) +``` + +or consider using the [minimal Minikube example values file] as reference, as stated in [CPU and RAM Resource Requirements](https://docs.gitlab.com/charts/installation/deployment.html#cpu-and-ram-resource-requirements) + +1. Finish preparing the environment: + + ```sh + export K8S_VERSION='v1.16.15' + ``` + +1. Enable the `ingress` and `metrics-server` addons: + + ```sh + minikube addons enable 'ingress' + minikube addons enable 'metrics-server' + ``` + +1. When using the `LoadBalancer` Ingress type (the default), start a tunnel in a different shell to let the installation finish: + + ```sh + minikube tunnel -c + ``` + +1. Install the chart as described above. + +1. Add minikube's IP address to the `/etc/hosts` file: + + ```sh + kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $3 " " $2}' | sudo tee -a '/etc/hosts' + ``` + +
+ +
+ Gotchas + +- Use self-signed certs and avoid using certmanager setting up the following: + + ```yaml + global: + ingress: + configureCertmanager: false + certmanager: + install: false + ``` + +- Avoid using a load balancer (mainly for local testing) setting the ingress type to `NodePort`: + + ```yaml + nginx-ingress: + controller: + service: + type: NodePort + ``` + +- As of 2021-01-15, a clean minikube cluster with only gitlab installed takes up about 1 vCPU and 6+ GiB RAM: + + ```sh + $ kubectl top nodes + NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% + minikube 965m 12% 6375Mi 53% + ``` + + ```sh + $ kubectl get pods --namespace 'gitlab' + NAMESPACE NAME READY STATUS RESTARTS AGE + gitlab gitlab-gitaly-0 1/1 Running 0 71m + gitlab gitlab-gitlab-exporter-547cf7fbff-xzqjp 1/1 Running 0 71m + gitlab gitlab-gitlab-shell-5c5b8dd9cd-g4z7b 1/1 Running 0 71m + gitlab gitlab-gitlab-shell-5c5b8dd9cd-ppbtk 1/1 Running 0 71m + gitlab gitlab-migrations-2-j6lt6 0/1 Completed 0 8m27s + gitlab gitlab-minio-6dd7d96ddb-xxq9w 1/1 Running 0 71m + gitlab gitlab-minio-create-buckets-2-q5zfg 0/1 Completed 0 8m27s + gitlab gitlab-nginx-ingress-controller-7fc8cbf49d-b9lqm 1/1 Running 0 71m + gitlab gitlab-nginx-ingress-controller-7fc8cbf49d-ng589 1/1 Running 0 71m + gitlab gitlab-nginx-ingress-default-backend-7ff88b95f-lv5vt 1/1 Running 0 71m + gitlab gitlab-postgresql-0 2/2 Running 0 71m + gitlab gitlab-prometheus-server-6cfb57f575-cs669 2/2 Running 0 71m + gitlab gitlab-redis-master-0 2/2 Running 0 71m + gitlab gitlab-registry-6c75496fc7-fgbvb 1/1 Running 0 8m16s + gitlab gitlab-registry-6c75496fc7-fhsqs 1/1 Running 0 8m27s + gitlab gitlab-sidekiq-all-in-1-v1-64b9c56675-lf29p 1/1 Running 0 8m27s + gitlab gitlab-task-runner-7897bb897d-br5g5 1/1 Running 0 7m54s + gitlab gitlab-webservice-default-7846fb55d6-4pspg 2/2 Running 0 7m37s + gitlab gitlab-webservice-default-7846fb55d6-tmjqm 2/2 Running 0 8m27s + ``` + + with a **spike** of 5 vCPUs upon installation (specially for sidekiq). Keep this in mind when sizing the test cluster + +- Disable TLS setting up the following values: + + ```yaml + global: + hosts: + https: false + ingress: + tls: + enabled: false + ``` + +- Use a suffix in the ingresses hosts setting up the `global.hosts.hostSuffix` value: + + ```sh + $ helm template \ + --namespace "${NAMESPACE}" \ + --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \ + --set global.hosts.hostSuffix='test' \ + 'gitlab' \ + 'gitlab/gitlab' \ + | yq -r 'select(.kind == "Ingress") | .spec.rules[].host' - + + gitlab-test.f.q.dn + minio-test.f.q.dn + registry-test.f.q.dn + ``` + +
+ +### Operator + +See the [operator guide] and the [operator code] for details. ## CI/CD @@ -70,193 +409,6 @@ Status: Downloaded newer image for gitlab/gitlab-runner-helper:x86_64-8fa89735 docker.io/gitlab/gitlab-runner-helper:x86_64-8fa89735 ``` -## Helm chart - -Gitlab offers an official helm chart to offer its services on a kubernetes cluster. - -Follow the [deployment] guide for details. - -### Chart maintenance - -- add and update Gitlab's helm repository: - - ```sh - helm repo add 'gitlab' 'https://charts.gitlab.io/' - helm repo update - ``` - -- lookup the chart's version: - - ```sh - $ helm search repo 'gitlab/gitlab' - NAME CHART VERSION APP VERSION DESCRIPTION - gitlab/gitlab 4.9.3 13.9.3 Web-based Git-repository manager with wiki and ... - ``` - -- fetch the chart: - - ```sh - helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" - helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION" - ``` - -- get the default values for the chart: - - ```sh - helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml" - helm inspect values --version "$CHART_VERSION" 'gitlab/gitlab' > "${VALUES_DIR}/values-${CHART_VERSION}.yaml" - ``` - - ```sh - export VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab" - helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml" - ``` - -- create a dedicated values file with the changes one needs (see [gotchas](#gotchas)): - - ```yaml - global: - edition: ce - ingress: - configureCertmanager: false - time_zone: UTC - certmanager: - install: false - gitlab-runner: - install: false - ``` - -- upgrade the stored chart to a new version: - - ```sh - helm repo update - rm -r "${CHART_DIR}/gitlab" - helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION" - ``` - -### Chart deployment - -1. prepare the environment: - - ```sh - export \ - ENVIRONMENT='minikube' \ - NAMESPACE='gitlab' \ - VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab" - ``` - -1. validate the values and install the chart (took > 20m on a MacBook Pro 16-inch 2019 with Intel i7 and 16GB RAM): - - ```sh - # validation - helm upgrade --install \ - --namespace "${NAMESPACE}" \ - --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \ - 'gitlab' \ - 'gitlab/gitlab' \ - --dry-run - - # installation - helm upgrade --install \ - --atomic \ - --create-namespace \ - --namespace "${NAMESPACE}" \ - --timeout 0 \ - --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \ - 'gitlab' \ - 'gitlab/gitlab' \ - --debug - ``` - -1. keep an eye on the installation: - - ```sh - kubectl get events \ - --namespace "${NAMESPACE}" \ - --sort-by '.metadata.creationTimestamp' \ - --watch - - # requires `watch` from 'procps-ng' (`brew install watch`) - watch kubectl get all --namespace "${NAMESPACE}" - - # requires `k9s` (`brew install k9s`) - k9s --namespace "${NAMESPACE}" - ``` - -1. get the login password for user `root`: - - ```sh - kubectl get secret 'gitlab-gitlab-initial-root-password' \ - --namespace "${NAMESPACE}" \ - -o jsonpath='{.data.password}' \ - | base64 --decode - ``` - -1. open the login page: - - ```sh - export URL="https://$(kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $2}')" - - xdg-open "${URL}" # on linux - open "${URL}" # on mac os x - ``` - -1. have fun! - -To delete everything: - -```sh -helm uninstall --namespace "${NAMESPACE}" 'gitlab' -kubectl delete --ignore-not-found namespace "${NAMESPACE}" -``` - -### Minikube - -When testing with a minikube installation with 8GiB RAM, kubernetes complained being out of memory.
-Be sure to give your cluster enough resources: - -```sh -# on linux -minikube start --kubernetes-version "${K8S_VERSION}" --cpus 4 --memory 12GiB - -# on mac os x -minikube start --kubernetes-version "${K8S_VERSION}" --cpus 8 --memory 12GiB # docker-desktop (no Ingresses) -minikube start --kubernetes-version "${K8S_VERSION}" --cpus 8 --memory 12GiB --vm # hyperkit vm (to be able to use Ingresses) -``` - -or consider using the [minimal Minikube example values file] as reference, as stated in [CPU and RAM Resource Requirements](https://docs.gitlab.com/charts/installation/deployment.html#cpu-and-ram-resource-requirements) - -1. finish preparing the environment: - - ```sh - export K8S_VERSION='v1.16.15' - ``` - -1. enable the `ingress` and `metrics-server` addons: - - ```sh - minikube addons enable 'ingress' - minikube addons enable 'metrics-server' - ``` - -1. to use the `LoadBalancer` Ingress type (the default), start a tunnel in a different shell to let the installation finish: - - ```sh - minikube tunnel -c - ``` - -1. install the chart as described [above](#chart-deployment) - -1. add minikube's IP address to the `/etc/hosts` file: - - ```sh - kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $3 " " $2}' | sudo tee -a '/etc/hosts' - ``` - -## Kubernetes operator - -See the [operator guide] and the [operator code] for details. - ## Manage kubernetes clusters See [adding and removing kubernetes clusters] for more information. @@ -265,87 +417,7 @@ For now the Gitlab instance can manage only kubernetes clusters external to the ## Gotchas -- use self-signed certs and avoid using certmanager setting up the following: - - ```yaml - global: - ingress: - configureCertmanager: false - certmanager: - install: false - ``` - -- avoid using a load balancer (mainly for local testing) setting the ingress type to `NodePort`: - - ```yaml - nginx-ingress: - controller: - service: - type: NodePort - ``` - -- as of 2021-01-15, a clean minikube cluster with only gitlab installed takes up about 1 vCPU and 6+ GiB RAM: - - ```sh - $ kubectl top nodes - NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% - minikube 965m 12% 6375Mi 53% - ``` - - ```sh - $ kubectl get pods --namespace gitlab - NAMESPACE NAME READY STATUS RESTARTS AGE - gitlab gitlab-gitaly-0 1/1 Running 0 71m - gitlab gitlab-gitlab-exporter-547cf7fbff-xzqjp 1/1 Running 0 71m - gitlab gitlab-gitlab-shell-5c5b8dd9cd-g4z7b 1/1 Running 0 71m - gitlab gitlab-gitlab-shell-5c5b8dd9cd-ppbtk 1/1 Running 0 71m - gitlab gitlab-migrations-2-j6lt6 0/1 Completed 0 8m27s - gitlab gitlab-minio-6dd7d96ddb-xxq9w 1/1 Running 0 71m - gitlab gitlab-minio-create-buckets-2-q5zfg 0/1 Completed 0 8m27s - gitlab gitlab-nginx-ingress-controller-7fc8cbf49d-b9lqm 1/1 Running 0 71m - gitlab gitlab-nginx-ingress-controller-7fc8cbf49d-ng589 1/1 Running 0 71m - gitlab gitlab-nginx-ingress-default-backend-7ff88b95f-lv5vt 1/1 Running 0 71m - gitlab gitlab-postgresql-0 2/2 Running 0 71m - gitlab gitlab-prometheus-server-6cfb57f575-cs669 2/2 Running 0 71m - gitlab gitlab-redis-master-0 2/2 Running 0 71m - gitlab gitlab-registry-6c75496fc7-fgbvb 1/1 Running 0 8m16s - gitlab gitlab-registry-6c75496fc7-fhsqs 1/1 Running 0 8m27s - gitlab gitlab-sidekiq-all-in-1-v1-64b9c56675-lf29p 1/1 Running 0 8m27s - gitlab gitlab-task-runner-7897bb897d-br5g5 1/1 Running 0 7m54s - gitlab gitlab-webservice-default-7846fb55d6-4pspg 2/2 Running 0 7m37s - gitlab gitlab-webservice-default-7846fb55d6-tmjqm 2/2 Running 0 8m27s - ``` - - with a **spike** of 5 vCPUs upon installation (specially for sidekiq). Keep this in mind when sizing the test cluster - -- disable TLS setting up the following values: - - ```yaml - global: - hosts: - https: false - ingress: - tls: - enabled: false - ``` - -- use a suffix in the ingresses hosts setting up the `global.hosts.hostSuffix` value: - - ```sh - $ helm template \ - --namespace "${NAMESPACE}" \ - --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \ - --set global.hosts.hostSuffix="test" \ - 'gitlab' \ - 'gitlab/gitlab' \ - | yq -r 'select(.kind == "Ingress") | .spec.rules[].host' - - - gitlab-test.f.q.dn - minio-test.f.q.dn - registry-test.f.q.dn - ``` - -- use an access token to clone a project +- Use access tokens to clone projects: ```sh git clone "https://oauth2:${ACCESS_TOKEN}@somegitlab.com/vendor/package.git" @@ -362,12 +434,13 @@ For now the Gitlab instance can manage only kubernetes clusters external to the - [Adding and removing Kubernetes clusters] - Gitlab's [operator code] and relative [guide][operator guide] -## Sources - -All the references in the [further readings] section, plus the following: +### Sources - [Configuring private dns zones and upstream nameservers in kubernetes] - [Using GitLab token to clone without authentication] +- [Back up GitLab Using Amazon S3] +- [Support object storage bucket prefixes] +- [Back up GitLab excluding specific data from the backup] [adding and removing kubernetes clusters]: https://docs.gitlab.com/ee/user/project/clusters/add_remove_clusters.html +[back up gitlab excluding specific data from the backup]: https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#excluding-specific-data-from-the-backup +[back up gitlab using amazon s3]: https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html?tab=Linux+package+%28Omnibus%29#using-amazon-s3 [chart]: https://docs.gitlab.com/charts/ [command-line options]: https://docs.gitlab.com/charts/installation/command-line-options.html [deployment]: https://docs.gitlab.com/charts/installation/deployment.html [global settings]: https://docs.gitlab.com/charts/charts/globals.html [minimal minikube example values file]: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/values-minikube-minimum.yaml -[operator code]: https://gitlab.com/gitlab-org/gl-openshift/gitlab-operator -[operator guide]: https://docs.gitlab.com/charts/installation/operator.html +[operator code]: https://gitlab.com/gitlab-org/cloud-native/gitlab-operator +[operator guide]: https://docs.gitlab.com/operator/ [runners on kubernetes]: https://docs.gitlab.com/runner/install/kubernetes.html +[support object storage bucket prefixes]: https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3376 [tls]: https://docs.gitlab.com/charts/installation/tls.html - -[further readings]: #further-readings - [configuring private dns zones and upstream nameservers in kubernetes]: https://kubernetes.io/blog/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes/ [using gitlab token to clone without authentication]: https://stackoverflow.com/questions/25409700/using-gitlab-token-to-clone-without-authentication#29570677