brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
3282e0b84546c673dd5f5c443264938c4ea469af
oam/knowledge base/gitlab.md
Michele Cereda 3282e0b845 chore(kb/gitlab): specify conditions for jobs
2024-04-20 01:08:18 +02:00

22 KiB
Raw Blame History

Gitlab

  1. Omnibus
  2. Kubernetes
    1. Helm chart
    2. Operator
  3. Repository management
    1. Different owners for parts of the code base
  4. CI/CD pipelines
    1. Specify when to run jobs
      1. Make a job in a pipeline run only when some specific files change
    2. Get the version of the helper image to use for a runner
  5. Manage kubernetes clusters
  6. Runners
    1. Autoscaling
      1. Docker Machine
  7. Troubleshooting
    1. Use access tokens to clone projects
    2. Pipeline fails with error You are not allowed to download code from this project
  8. Further readings
    1. Sources

Omnibus

Configuration

The application of configuration changes is handled by Chef Infra.
It runs checks, ensures directories, permissions, and services are in place and working, and restarts components if any of their configuration files have changed.

# Edit and validate.
sudo vim '/etc/gitlab/gitlab.rb'
sudo ruby -c '/etc/gitlab/gitlab.rb'
sudo gitlab-ctl check-config

# Make Gitlab aware of the changes.
sudo gitlab-ctl reconfigure

Backup settings for AWS buckets.
See Back up Gitlab using Amazon S3:

# If using an IAM Profile, don't configure 'aws_access_key_id' and
# 'aws_secret_access_key' but set "'use_iam_profile' => true" instead.
gitlab_rails['backup_upload_connection'] = {
  'provider' => 'AWS',
  'region' => 'eu-west-1',
  'aws_access_key_id' => 'AKIAKIAKI',
  'aws_secret_access_key' => 'secret123'
}

# It appears one can use prefixes by appending them to the bucket name.
# See https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3376.
gitlab_rails['backup_upload_remote_directory'] = 'bucket-name/prefix'

# Use multipart uploads when the archive's size exceeds 100MB.
gitlab_rails['backup_multipart_chunk_size'] = 104857600

# Only keep 7 days worth of backups.
gitlab_rails['backup_keep_time'] = 604800
Maintenance 
# Check the components' state.
sudo gitlab-ctl status

# Create backups.
sudo gitlab-backup create BACKUP='prefix_override' STRATEGY='copy'

# Create empty backup archives for testing purposes.
# See https://docs.gitlab.com/ee/administration/backup_restore/backup_gitlab.html#excluding-specific-data-from-the-backup
sudo gitlab-backup create … \
  SKIP='db,repositories,uploads,builds,artifacts,pages,lfs,terraform_state,registry,packages,ci_secure_files'

Kubernetes

Helm chart

Gitlab offers an official helm chart to to allow for deployments on kubernetes clusters.

Follow the deployment guide for details and updated information.

Deployment

  1. Prepare the environment:

    export \
  ENVIRONMENT='minikube' \
  NAMESPACE='gitlab' \
  VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab"

  2. Validate the values and install the chart.

    The installation took > 20m on a MacBook Pro 16-inch 2019 with Intel i7 and 16GB RAM.

    # validation
helm upgrade --install \
  --namespace "${NAMESPACE}" \
  --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
  'gitlab' \
  'gitlab/gitlab' \
  --dry-run

# installation
helm upgrade --install \
  --atomic \
  --create-namespace \
  --namespace "${NAMESPACE}" \
  --timeout 0 \
  --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
  'gitlab' \
  'gitlab/gitlab' \
  --debug

  3. Keep an eye on the installation:

    kubectl get events \
  --namespace "${NAMESPACE}" \
  --sort-by '.metadata.creationTimestamp' \
  --watch

# requires `watch` from 'procps-ng'
# `brew install watch`
watch kubectl get all --namespace "${NAMESPACE}"

# requires `k9s`
# `brew install k9s`
k9s --namespace "${NAMESPACE}"

  4. Get the login password for the root user:

    kubectl get secret 'gitlab-gitlab-initial-root-password' \
  --namespace "${NAMESPACE}" \
  -o jsonpath='{.data.password}' \
| base64 --decode

  5. Open the login page:

    export URL="https://$(kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $2}')"

xdg-open "${URL}"   # on linux
open "${URL}"       # on mac os x

  6. Have fun!

To delete everything:

helm uninstall --namespace "${NAMESPACE}" 'gitlab'
kubectl delete --ignore-not-found namespace "${NAMESPACE}"
Maintenance

  • Add and update Gitlab's helm repository:

    helm repo add 'gitlab' 'https://charts.gitlab.io/'
helm repo update

  • Look up the chart's version:

    $ helm search repo 'gitlab/gitlab'
NAME             CHART VERSION   APP VERSION     DESCRIPTION
gitlab/gitlab    4.9.3           13.9.3          Web-based Git-repository manager with wiki and ...

  • Fetch the chart:

    helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR"
helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION"

  • Get the default values for the chart:

    helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml"
helm inspect values --version "$CHART_VERSION" 'gitlab/gitlab' > "${VALUES_DIR}/values-${CHART_VERSION}.yaml"

    export VALUES_DIR="$(git rev-parse --show-toplevel)/kubernetes/helm/gitlab"
helm inspect values 'gitlab/gitlab' > "${VALUES_DIR}/values.yaml"

  • Create a dedicated values file with the changes one needs (see the helm chart gotchas below):

    global:
  edition: ce
  ingress:
    configureCertmanager: false
  time_zone: UTC
certmanager:
  install: false
gitlab-runner:
  install: false

  • Upgrade the stored chart to a new version:

    helm repo update
rm -r "${CHART_DIR}/gitlab"
helm fetch 'gitlab/gitlab' --untar --untardir "$CHART_DIR" --version "$CHART_VERSION"
Minikube

When testing with a minikube installation with 8GiB RAM, kubernetes complained being out of memory.
Be sure to give your cluster enough resources:

# on linux
minikube start --kubernetes-version "${K8S_VERSION}" --cpus '4' --memory '12GiB'

# on mac os x
minikube start --kubernetes-version "${K8S_VERSION}" --cpus '8' --memory '12GiB'       # docker-desktop (no Ingresses)
minikube start --kubernetes-version "${K8S_VERSION}" --cpus '8' --memory '12GiB' --vm  # hyperkit vm (to be able to use Ingresses)

or consider using the minimal Minikube example values file as reference, as stated in CPU and RAM Resource Requirements

  1. Finish preparing the environment:

    export K8S_VERSION='v1.16.15'

  2. Enable the ingress and metrics-server addons:

    minikube addons enable 'ingress'
minikube addons enable 'metrics-server'

  3. When using the LoadBalancer Ingress type (the default), start a tunnel in a different shell to let the installation finish:

    minikube tunnel -c

  4. Install the chart as described above.

  5. Add minikube's IP address to the /etc/hosts file:

    kubectl get ingresses --namespace 'gitlab' | grep 'webservice' | awk '{print $3 "  " $2}' | sudo tee -a '/etc/hosts'
Gotchas

  • Use self-signed certs and avoid using certmanager setting up the following:

    global:
  ingress:
    configureCertmanager: false
certmanager:
  install: false

  • Avoid using a load balancer (mainly for local testing) setting the ingress type to NodePort:

    nginx-ingress:
  controller:
    service:
      type: NodePort

  • As of 2021-01-15, a clean minikube cluster with only gitlab installed takes up about 1 vCPU and 6+ GiB RAM:

    $ kubectl top nodes
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
minikube   965m         12%    6375Mi          53%

    $ kubectl get pods --namespace 'gitlab'
NAMESPACE     NAME                                                   READY   STATUS        RESTARTS   AGE
gitlab        gitlab-gitaly-0                                        1/1     Running       0          71m
gitlab        gitlab-gitlab-exporter-547cf7fbff-xzqjp                1/1     Running       0          71m
gitlab        gitlab-gitlab-shell-5c5b8dd9cd-g4z7b                   1/1     Running       0          71m
gitlab        gitlab-gitlab-shell-5c5b8dd9cd-ppbtk                   1/1     Running       0          71m
gitlab        gitlab-migrations-2-j6lt6                              0/1     Completed     0          8m27s
gitlab        gitlab-minio-6dd7d96ddb-xxq9w                          1/1     Running       0          71m
gitlab        gitlab-minio-create-buckets-2-q5zfg                    0/1     Completed     0          8m27s
gitlab        gitlab-nginx-ingress-controller-7fc8cbf49d-b9lqm       1/1     Running       0          71m
gitlab        gitlab-nginx-ingress-controller-7fc8cbf49d-ng589       1/1     Running       0          71m
gitlab        gitlab-nginx-ingress-default-backend-7ff88b95f-lv5vt   1/1     Running       0          71m
gitlab        gitlab-postgresql-0                                    2/2     Running       0          71m
gitlab        gitlab-prometheus-server-6cfb57f575-cs669              2/2     Running       0          71m
gitlab        gitlab-redis-master-0                                  2/2     Running       0          71m
gitlab        gitlab-registry-6c75496fc7-fgbvb                       1/1     Running       0          8m16s
gitlab        gitlab-registry-6c75496fc7-fhsqs                       1/1     Running       0          8m27s
gitlab        gitlab-sidekiq-all-in-1-v1-64b9c56675-lf29p            1/1     Running       0          8m27s
gitlab        gitlab-task-runner-7897bb897d-br5g5                    1/1     Running       0          7m54s
gitlab        gitlab-webservice-default-7846fb55d6-4pspg             2/2     Running       0          7m37s
gitlab        gitlab-webservice-default-7846fb55d6-tmjqm             2/2     Running       0          8m27s

    with a spike of 5 vCPUs upon installation (specially for sidekiq). Keep this in mind when sizing the test cluster

  • Disable TLS setting up the following values:

    global:
  hosts:
    https: false
  ingress:
    tls:
      enabled: false

  • Use a suffix in the ingresses hosts setting up the global.hosts.hostSuffix value:

    $ helm template \
    --namespace "${NAMESPACE}" \
    --values "${VALUES_DIR}/values.${ENVIRONMENT}.yaml" \
    --set global.hosts.hostSuffix='test' \
    'gitlab' \
    'gitlab/gitlab' \
  | yq -r 'select(.kind == "Ingress") | .spec.rules[].host' -

gitlab-test.f.q.dn
minio-test.f.q.dn
registry-test.f.q.dn

Operator

See the operator guide and the operator code for details.

Repository management

Different owners for parts of the code base

Refer to code owners for more and updated information.

Leverage code owners.

By adding code owners to the repository, they become eligible approvers in the project for MRs that contain those files.
Enable the eligible approvers merge request approval rule in the project's Settings > Merge requests.

Require code owner approval for protected branches in the project's Settings > Repository > Protected branches.

Gotchas:

  • Specifying owners for paths overwrites the previous owners list.
    There seems to be no way to inherit and add (and not just overwrite) owners that would not require the list being repeated.
  • There is as of 2024-04-10 no way to assign ownership by using aliases for roles (like maintainers or developers); only groups or users are allowed.
    This feature is being added, but it has been open for over 3y now. See Ability to reference Maintainers or Developers from CODEOWNERS.

CI/CD pipelines

Refer to CI/CD pipelines and CI/CD pipeline templates.
Also check Use CI/CD configuration from other files and Use extends to reuse configuration sections.

Specify when to run jobs

Refer Specify when jobs run with rules.

Use the rules key and specify the conditions the job needs.

The only/except keywords have been deprecated by the rules keyword, and cannot be used together.
This means one might be forced to use only/except if one is including a pipeline that is already using them.

Conditions are validated in order until one applies. The rest are ignored.
If no condition applies, the job is skipped.

Make a job in a pipeline run only when some specific files change

docker-build:
  only:
    changes:
      - cmd/*
      - go.*
      - Dockerfile

Multiple entries in the condition are validated in an OR fashion. In the example above, the condition will make the job run only when a change occurs:

  • to any file in the cmd directory
  • to any file in the repository's root directory which name starts with go (like go.mod or go.sum)
  • to the Dockerfile in the repository's root directory

Get the version of the helper image to use for a runner

The gitlab/gitlab-runner-helper images are tagged using the runner's os, architecture, and git revision.

One needs to know the version of Gitlab and of the runner one wants to use.
Usually, the runner's version is the one most similar to Gitlab's version (e.g. Gitlab: 13.6.2 → gitlab-runner: 13.6.0).

To get the tag to use for the helper, check the runner's version:

$ docker run --rm --name 'runner' 'gitlab/gitlab-runner:alpine-v13.6.0' --version
Version:      13.6.0
Git revision: 8fa89735
Git branch:   13-6-stable
GO version:   go1.13.8
Built:        2020-11-21T06:16:31+0000
OS/Arch:      linux/amd64

In this case, the os is Linux, the architecture is amd64 and the revision is 8fa89735. So, following their instructions, the tag will be x86_64-8fa89735:

$ docker pull 'gitlab/gitlab-runner-helper:x86_64-8fa89735'
x86_64-8fa89735: Pulling from gitlab/gitlab-runner-helper
a1514ca1e64d: Pull complete
Digest: sha256:4e239257280eb0fa750f1ef30975dacdef5f5346bfaa9e6d60e58d440d8cd0f1
Status: Downloaded newer image for gitlab/gitlab-runner-helper:x86_64-8fa89735
docker.io/gitlab/gitlab-runner-helper:x86_64-8fa89735

Manage kubernetes clusters

See adding and removing kubernetes clusters for more information.

For now the Gitlab instance can manage only kubernetes clusters external to the one it is running into.

Runners

brew install 'gitlab-runner'

gitlab-runner exec docker 'job-name'
gitlab-runner exec docker \
  --env 'AWS_ACCESS_KEY_ID=AKIA…' --env 'AWS_SECRET_ACCESS_KEY=F…s' --env 'AWS_REGION=eu-east-1' \
  --env 'DOCKER_AUTH_CONFIG={ "credsStore": "ecr-login" }' \
  --docker-volumes "$HOME/.aws/credentials:/root/.aws/credentials:ro"
  'job-requiring-ecr-access'

Autoscaling

Docker Machine

Supported cloud providers.

Pitfalls:

Troubleshooting

Use access tokens to clone projects

git clone "https://oauth2:${ACCESS_TOKEN}@somegitlab.com/vendor/package.git"

Pipeline fails with error You are not allowed to download code from this project

Error message example:

Getting source from Git repository 00:00
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in /builds/myProj/myRepo/.git/
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://gitlab.com/myProj/myRepo.git/': The requested URL returned error: 403

Root cause: the user starting the pipeline does not have enough privileges to the repository.

Solution: give that user developer access or have somebody else with enough privileges run it.

Further readings

Sources