diff --git a/examples/kubernetes/values.gitlab-runner.yaml b/examples/kubernetes/values.gitlab-runner.yaml index 6cdb926..0b33f7c 100644 --- a/examples/kubernetes/values.gitlab-runner.yaml +++ b/examples/kubernetes/values.gitlab-runner.yaml @@ -2,8 +2,7 @@ ## # Source: -# `helm template --namespace 'gitlab' --values "values.gitlab-runner.yaml" --set global.hosts.hostSuffix='test' \ -# 'gitlab-runner' 'gitlab/gitlab-runner'` +# `helm template --namespace 'gitlab' 'gitlab-runner' 'gitlab/gitlab-runner'` ## useTini: false diff --git a/knowledge base/kubernetes/README.md b/knowledge base/kubernetes/README.md index 7c18452..443b87e 100644 --- a/knowledge base/kubernetes/README.md +++ b/knowledge base/kubernetes/README.md @@ -3,15 +3,13 @@ Open source container orchestration engine for containerized applications.
Hosted by the [Cloud Native Computing Foundation][cncf]. -## Table of content - 1. [Basics](#basics) -1. [The control plane](#the-control-plane) - 1. [The API server](#the-api-server) +1. [Control plane](#control-plane) + 1. [API server](#api-server) 1. [`kube-scheduler`](#kube-scheduler) 1. [`kube-controller-manager`](#kube-controller-manager) 1. [`cloud-controller-manager`](#cloud-controller-manager) -1. [The worker nodes](#the-worker-nodes) +1. [Worker nodes](#worker-nodes) 1. [`kubelet`](#kubelet) 1. [`kube-proxy`](#kube-proxy) 1. [Container runtime](#container-runtime) @@ -51,19 +49,21 @@ In cloud environments, nodes are also available in grouped sets (_node pools_) c Nodes host application workloads in the form of [_pods_][pods]. -The [_control plane_](#the-control-plane) manages the nodes and the pods in the cluster. It is itself a set of pods which expose the APIs and interfaces used to define, deploy, and manage the lifecycle of the cluster's resources.
-In higher environments, the control plane usually runs across multiple **dedicated** nodes to provide improved fault-tolerance and high availability. +The [_control plane_](#control-plane) manages the nodes and the pods in the cluster. It is itself a set of pods +which expose the APIs and interfaces used to define, deploy, and manage the lifecycle of the cluster's resources.
+In higher environments, the control plane usually runs across multiple **dedicated** nodes to provide improved +fault-tolerance and high availability. ![Cluster components](components.svg) -## The control plane +## Control plane Makes global decisions about the cluster (like scheduling).
Detects and responds to cluster events (like starting up a new pod when a deployment has less replicas then it requests). The control plane is composed by: -- [the API server](#the-api-server); +- [the API server](#api-server); - The _distributed store_ for the cluster's configuration data.
The current store of choice is [`etcd`][etcd]. - [the scheduler](#kube-scheduler); @@ -71,21 +71,37 @@ The control plane is composed by: - [the cloud controller](#cloud-controller-manager). Control plane components run on one or more cluster nodes.
-For ease of use, setup scripts typically start all control plane components on the **same** host and avoid **running** other workloads on it. +For ease of use, setup scripts typically start all control plane components on the **same** host and avoid **running** +other workloads on it. -### The API server +### API server -The API server exposes the Kubernetes API. It is the front end for, and the core of, the Kubernetes control plane.
-`kube-apiserver` is the main implementation of the Kubernetes API server, and is designed to scale horizontally (by deploying more instances) and balance traffic between its instances. +Exposes the Kubernetes API. It is the front end for, and the core of, the Kubernetes control plane.
+`kube-apiserver` is the main implementation of the Kubernetes API server, and is designed to scale horizontally (by +deploying more instances) and balance traffic between its instances. -The API server exposes the HTTP API that lets end users, different parts of a cluster, and external components communicate with one another. This lets you query and manipulate the state of API objects in Kubernetes and can be accessed through command-line tools or directly using REST calls. The serialized state of the objects is stored by writing them into `etcd`'s store. +The API server exposes the HTTP API that lets end users, different parts of a cluster and external components +communicate with one another, or query and manipulate the state of API objects in Kubernetes.
+Can be accessed through command-line tools or directly using REST calls.
+The serialized state of the objects is stored by writing them into `etcd`'s store. -Consider using one of the client libraries if you are writing an application using the Kubernetes API. The complete API details are documented using OpenAPI. +Suggested the use of one of the available client libraries if writing an application using the Kubernetes API.
+The complete API details are documented using OpenAPI. -Kubernetes supports multiple API versions, each at a different API path (like `/api/v1` or `/apis/rbac.authorization.k8s.io/v1alpha1`); the server handles the conversion between API versions transparently. Versioning is done at the API level rather than at the resource or field level, ensuring that the API presents a clear and consistent view of system resources and behavior, and enabling controlling access to end-of-life and/or experimental APIs. All the different versions are representations of the same persisted data. +Kubernetes supports multiple API versions, each at a different API path (e.g.: `/api/v1`, +`/apis/rbac.authorization.k8s.io/v1alpha1`).
+All the different versions are representations of the same persisted data.
+The server handles the conversion between API versions transparently. -To make it easier to evolve and to extend them, Kubernetes implements API groups that can be enabled or disabled. API resources are distinguished by their **API group**, **resource type**, **namespace** (for namespaced resources), and **name**.
-New API resources and new resource fields can be added often and frequently. Elimination of resources or fields requires following the [API deprecation policy]. +Versioning is done at the API level, rather than at the resource or field level, to ensure the API presents a clear and +consistent view of system resources and behavior.
+Also enables controlling access to end-of-life and/or experimental APIs. + +API groups can be enabled or disabled.
+API resources are distinguished by their **API group**, **resource type**, **namespace** (for namespaced resources), and +**name**.
+New API resources and new resource fields can be added often and frequently.
+Elimination of resources or fields requires following the [API deprecation policy]. The Kubernetes API can be extended: @@ -108,33 +124,42 @@ Scheduling decisions take into account: ### `kube-controller-manager` Runs _controller_ processes.
-Each controller is a separate process logically speaking; they are all compiled into a single binary and run in a single process to reduce complexity. +Each controller is a separate process logically speaking; they are all compiled into a single binary and run in a single +process to reduce complexity. Examples of these controllers are: - the node controller, which notices and responds when nodes go down; -- the replication controller, which maintains the correct number of pods for every replication controller object in the system; +- the replication controller, which maintains the correct number of pods for every replication controller object in the + system; - the job controller, which checks one-off tasks (_job_) objects and creates pods to run them to completion; - the EndpointSlice controller, which populates _EndpointSlice_ objects providing a link between services and pods; - the ServiceAccount controller, which creates default ServiceAccounts for new namespaces. ### `cloud-controller-manager` -Embeds cloud-specific control logic, linking clusters to one's cloud provider's API and separating the components that interact with that cloud platform from the components that only interact with clusters. +Embeds cloud-specific control logic, linking clusters to one's cloud provider's API and separating the components that +interact with that cloud platform from the components that only interact with clusters. -They only run controllers that are specific to one's cloud provider. If you are running Kubernetes on your own premises, or in a learning environment inside your own PC, your cluster will have no cloud controller managers. +Clusters only run controllers that are specific to one's cloud provider.
+If running Kubernetes on one's own premises, or in a learning environment inside one's own PC, the cluster will have no +cloud controller managers. -As with the `kube-controller-manager`, it combines several logically independent control loops into a single binary that you run as a single process. It can scale horizontally to improve performance or to help tolerate failures. +As with the `kube-controller-manager`, cloud controller managers combine several logically independent control loops +into single binaries run as single processes.
+It can scale horizontally to improve performance or to help tolerate failures. The following controllers can have cloud provider dependencies: -- the node controller, which checks the cloud provider to determine if a node has been deleted in the cloud after it stops responding; +- the node controller, which checks the cloud provider to determine if a node has been deleted in the cloud after it + stops responding; - the route controller, which sets up routes in the underlying cloud infrastructure; - the service controller, which creates, updates and deletes cloud provider load balancers. -## The worker nodes +## Worker nodes -Each and every node runs components providing a runtime environment for the cluster, and syncing with the control plane to maintain workloads running as requested. +Each and every node runs components providing a runtime environment for the cluster, and syncing with the control plane +to maintain workloads running as requested. ### `kubelet` @@ -147,19 +172,23 @@ It only manages containers created by Kubernetes. Network proxy running on each node and implementing part of the Kubernetes Service concept. -It maintains all the network rules on nodes which allow network communication to the Pods from network sessions inside or outside of your cluster. +It maintains all the network rules on nodes which allow network communication to the Pods from network sessions inside +or outside of one's cluster. -It uses the operating system's packet filtering layer, if there is one and it's available; if not, it forwards the traffic itself. +It uses the operating system's packet filtering layer, if there is one and it's available; if not, it just forwards the +traffic itself. ### Container runtime -The software that is responsible for running containers. +The software responsible for running containers. -Kubernetes supports container runtimes like `containerd`, `CRI-O`, and any other implementation of the Kubernetes CRI (Container Runtime Interface). +Kubernetes supports container runtimes like `containerd`, `CRI-O`, and any other implementation of the Kubernetes CRI +(Container Runtime Interface). ### Addons -Addons use Kubernetes resources (_DaemonSet_, _Deployment_, etc) to implement cluster features, and as such namespaced resources for addons belong within the `kube-system` namespace. +Addons use Kubernetes resources (_DaemonSet_, _Deployment_, etc) to implement cluster features.
+As such, namespaced resources for addons belong within the `kube-system` namespace. See [addons] for an extended list of the available addons. @@ -175,14 +204,19 @@ Configuration files are written in YAML (preferred) or JSON format and are compo ### Pods The smallest deployable unit of computing that one can create and manage in Kubernetes.
-Pods contain one or more relatively tightly coupled application containers; they are always co-located (executed on the same host) and co-scheduled (executed together), and **share** context, storage and network resources, and a specification for how to run them. +Pods contain one or more relatively tightly coupled application containers; they are always co-located (executed on the +same host) and co-scheduled (executed together), and **share** context, storage and network resources, and a +specification for how to run them. -Pods are (and _should be_) usually created trough other workload resources (like _Deployments_, _StatefulSets_, or _Jobs_) and **not** directly.
-Such parent resources leverage and manage _ReplicaSets_, which in turn manage copies of the same pod. When deleted, **all** the resources they manage are deleted with them. +Pods are (and _should be_) usually created trough other workload resources (like _Deployments_, _StatefulSets_, or +_Jobs_) and **not** directly.
+Such parent resources leverage and manage _ReplicaSets_, which in turn manage copies of the same pod. When deleted, +**all** the resources they manage are deleted with them. Gotchas: -- If a Container specifies a memory or CPU `limit` but does **not** specify a memory or CPU `request`, Kubernetes automatically assigns it a resource `request` spec equal to the given `limit`. +- If a Container specifies a memory or CPU `limit` but does **not** specify a memory or CPU `request`, Kubernetes + automatically assigns it a resource `request` spec equal to the given `limit`. ## Autoscaling @@ -209,8 +243,10 @@ Autoscaling of Nodes by number requires the [Cluster Autoscaler]. 1. The Cluster Autoscaler routinely checks for pending Pods. 1. Pods fill up the available Nodes. 1. When Pods start to fail for lack of available resources, Nodes are added to the cluster. -1. When Pods are not failing due to lack of available resources and one or more Nodes are underused, the Autoscaler tries to fit the existing Pods in less Nodes. -1. If one or more Nodes can result unused from the previous step (DaemonSets are usually not taken into consideration), the Autoscaler will terminate them. +1. When Pods are not failing due to lack of available resources and one or more Nodes are underused, the Autoscaler + tries to fit the existing Pods in less Nodes. +1. If one or more Nodes can result unused from the previous step (DaemonSets are usually not taken into consideration), + the Autoscaler will terminate them. Autoscaling of Nodes by size requires add-ons like [Karpenter]. @@ -220,19 +256,24 @@ Also see [configuration best practices] and the [production best practices check - Prefer an **updated** version of Kubernetes.
The upstream project maintains release branches for the most recent three minor releases.
- Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support. + Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received + approximately 9 months of patch support. - Prefer **stable** versions of Kubernetes and **multiple nodes** for production clusters. - Prefer **consistent** versions of Kubernetes components throughout **all** nodes.
- Components support [version skew][version skew policy] up to a point, with specific tools placing additional restrictions. + Components support [version skew][version skew policy] up to a point, with specific tools placing additional + restrictions. - Consider keeping **separation of ownership and control** and/or group related resources.
Leverage [Namespaces]. - Consider **organizing** cluster and workload resources.
Leverage [Labels][labels and selectors]; see [recommended Labels]. - Avoid sending traffic to pods which are not ready to manage it.
- [Readiness probes][Configure Liveness, Readiness and Startup Probes] signal services to not forward requests until the probe verifies its own pod is up.
- [Liveness probes][configure liveness, readiness and startup probes] ping the pod for a response and check its health; if the check fails, they kill the current pod and launch a new one. + [Readiness probes][Configure Liveness, Readiness and Startup Probes] signal services to not forward requests until the + probe verifies its own pod is up.
+ [Liveness probes][configure liveness, readiness and startup probes] ping the pod for a response and check its health; + if the check fails, they kill the current pod and launch a new one. - Avoid workloads and nodes fail due limited resources being available.
- Set [resource requests and limits][resource management for pods and containers] to reserve a minimum amount of resources for pods and limit their hogging abilities. + Set [resource requests and limits][resource management for pods and containers] to reserve a minimum amount of + resources for pods and limit their hogging abilities. - Prefer smaller container images. - Prioritize critical workloads.
Leverage [quality of service](#quality-of-service). @@ -305,7 +346,8 @@ When a Pod is created, it is also assigned one of the following QoS classes: qosClass: Burstable ``` -- _BestEffort_, when the Pod does not meet the criteria for the other QoS classes (its Containers have **no** memory or CPU limits **nor** requests) +- _BestEffort_, when the Pod does not meet the criteria for the other QoS classes (its Containers have **no** memory or + CPU limits **nor** requests) ```yaml spec: @@ -319,37 +361,50 @@ When a Pod is created, it is also assigned one of the following QoS classes: ## Containers with high privileges -Kubernetes [introduced a Security Context][security context design proposal] as a mitigation solution to some workloads requiring to change one or more Node settings for performance, stability, or other issues (e.g. [ElasticSearch]).
-This is usually achieved executing the needed command from an InitContainer with higher privileges than normal, which will have access to the Node's resources and breaks the isolation Containers are usually famous for. If compromised, an attacker can use this highly privileged container to gain access to the underlying Node. +Kubernetes [introduced a Security Context][security context design proposal] as a mitigation solution to some workloads +requiring to change one or more Node settings for performance, stability, or other issues (e.g. [ElasticSearch]).
+This is usually achieved executing the needed command from an InitContainer with higher privileges than normal, which +will have access to the Node's resources and breaks the isolation Containers are usually famous for. If compromised, an +attacker can use this highly privileged container to gain access to the underlying Node. From the design proposal: -> A security context is a set of constraints that are applied to a Container in order to achieve the following goals (from the [Security design][Security Design Proposal]): +> A security context is a set of constraints that are applied to a Container in order to achieve the following goals +> (from the [Security design][Security Design Proposal]): > > - ensure a **clear isolation** between the Container and the underlying host it runs on; > - **limit** the ability of the Container to negatively impact the infrastructure or other Containers. > -> [The main idea is that] **Containers should only be granted the access they need to perform their work**. The Security Context takes advantage of containerization features such as the ability to [add or remove capabilities][Runtime privilege and Linux capabilities in Docker containers] to give a process some privileges, but not all the privileges of the `root` user. +> \[The main idea is that] **Containers should only be granted the access they need to perform their work**. The +> Security Context takes advantage of containerization features such as the ability to +> [add or remove capabilities][Runtime privilege and Linux capabilities in Docker containers] to give a process some +> privileges, but not all the privileges of the `root` user. ### Capabilities -Adding capabilities to a Container is **not** making it _privileged_, **nor** allowing _privilege escalation_. It is just giving the Container the ability to write to specific files or devices depending on the given capability. +Adding capabilities to a Container is **not** making it _privileged_, **nor** allowing _privilege escalation_. It is +just giving the Container the ability to write to specific files or devices depending on the given capability. -This means having a capability assigned does **not** automatically make the Container able to wreak havoc on a Node, and this practice **can be a legitimate use** of this feature instead. +This means having a capability assigned does **not** automatically make the Container able to wreak havoc on a Node, and +this practice **can be a legitimate use** of this feature instead. From the feature's `man` page: -> Linux divides the privileges traditionally associated with superuser into distinct units, known as _capabilities_, which can be independently enabled and disabled. Capabilities are a per-thread attribute. +> Linux divides the privileges traditionally associated with superuser into distinct units, known as _capabilities_, +> which can be independently enabled and disabled. Capabilities are a per-thread attribute. This also means a Container will be **limited** to its contents, plus the capabilities it has been assigned. -Some capabilities are assigned to all Containers by default, while others (the ones which could cause more issues) require to be **explicitly** set using the Containers' `securityContext.capabilities.add` property.
-If a Container is _privileged_ (see [Privileged container vs privilege escalation]), it will have access to **all** the capabilities, with no regards of what are explicitly assigned to it. +Some capabilities are assigned to all Containers by default, while others (the ones which could cause more issues) +require to be **explicitly** set using the Containers' `securityContext.capabilities.add` property.
+If a Container is _privileged_ (see [Privileged container vs privilege escalation]), it will have access to **all** the +capabilities, with no regards of what are explicitly assigned to it. Check: - [Linux capabilities], to see what capabilities can be assigned to a process **in a Linux system**; -- [Runtime privilege and Linux capabilities in Docker containers] for the capabilities available **inside Kubernetes**, and +- [Runtime privilege and Linux capabilities in Docker containers] for the capabilities available **inside Kubernetes**, + and - [Container capabilities in Kubernetes] for a handy table associating capabilities in Kubernetes to their Linux variant. ### Privileged container vs privilege escalation @@ -357,33 +412,43 @@ Check: A _privileged container_ is very different from a _container leveraging privilege escalation_. A **privileged container** does whatever a processes running directly on the Node can.
-It will have automatically assigned **all** [capabilities](#capabilities), and being `root` in this container is effectively being `root` on the Node it is running on. +It will have automatically assigned **all** [capabilities](#capabilities), and being `root` in this container is +effectively being `root` on the Node it is running on. -> For a Container to be _privileged_, its definition **requires the `securityContext.privileged` property set to `true`**. +> For a Container to be _privileged_, its definition **requires the `securityContext.privileged` property set to +> `true`**. **Privilege escalation** allows **a process inside the Container** to gain more privileges than its parent process.
-The process will be able to assume `root`-like powers, but will have access only to the **assigned** [capabilities](#capabilities) and generally have limited to no access to the Node like any other Container. +The process will be able to assume `root`-like powers, but will have access only to the **assigned** +[capabilities](#capabilities) and generally have limited to no access to the Node like any other Container. -> For a Container to _leverage privilege escalation_, its definition **requires the `securityContext.allowPrivilegeEscalation` property**: +> For a Container to _leverage privilege escalation_, its definition **requires the +> `securityContext.allowPrivilegeEscalation` property**: > > - to **either** be set to `true`, or > - to **not be set** at all **if**: > - the Container is already privileged, or > - the Container has `SYS_ADMIN` capabilities. > -> This property directly controls whether the [`no_new_privs`][No New Privileges Design Proposal] flag gets set on the Container's process. +> This property directly controls whether the [`no_new_privs`][No New Privileges Design Proposal] flag gets set on the +> Container's process. From the [design document for `no_new_privs`][No New Privileges Design Proposal]: -> In Linux, the `execve` system call can grant more privileges to a newly-created process than its parent process. Considering security issues, since Linux kernel v3.5, there is a new flag named `no_new_privs` added to prevent those new privileges from being granted to the processes. +> In Linux, the `execve` system call can grant more privileges to a newly-created process than its parent process. +> Considering security issues, since Linux kernel v3.5, there is a new flag named `no_new_privs` added to prevent those +> new privileges from being granted to the processes. > -> `no_new_privs` is inherited across `fork`, `clone` and `execve` and **can not be unset**. With `no_new_privs` set, `execve` promises not to grant the privilege to do anything that could not have been done without the `execve` call. +> `no_new_privs` is inherited across `fork`, `clone` and `execve` and **can not be unset**. With `no_new_privs` set, +> `execve` promises not to grant the privilege to do anything that could not have been done without the `execve` call. > -> For more details about `no_new_privs`, please check the [Linux kernel documentation][no_new_privs linux kernel documentation]. +> For more details about `no_new_privs`, please check the +> [Linux kernel documentation][no_new_privs linux kernel documentation]. > -> […] +> \[…] > -> To recap, below is a table defining the default behavior at the pod security policy level and what can be set as a default with a pod security policy: +> To recap, below is a table defining the default behavior at the pod security policy level and what can be set as a +> default with a pod security policy: > > | allowPrivilegeEscalation setting | uid = 0 or unset | uid != 0 | privileged/CAP_SYS_ADMIN | > | -------------------------------- | ------------------ | ------------------ | ------------------------ | @@ -411,11 +476,13 @@ All kubernetes clusters should: - be created using **IaC** ([terraform], [pulumi]); - have different node pools dedicated to different workloads; -- have at least one node pool composed by **non-preemptible** dedicated to critical services like Admission Controller Webhooks. +- have at least one node pool composed by **non-preemptible** dedicated to critical services like Admission Controller + Webhooks. Each node pool should: -- have a _meaningful_ **name** (like `--`) to make it easy to recognize the workloads running on it or the features of the nodes in it; +- have a _meaningful_ **name** (like `--`) to make it easy to recognize the workloads + running on it or the features of the nodes in it; - have a _minimum_ set of _meaningful_ **labels**, like: - cloud provider information; - node information and capabilities; @@ -423,7 +490,8 @@ Each node pool should: ## Edge computing -If planning to run Kubernetes on a Raspberry Pi, see [k3s] and the [Build your very own self-hosting platform with Raspberry Pi and Kubernetes] series of articles. +If planning to run Kubernetes on a Raspberry Pi, see [k3s] and the +[Build your very own self-hosting platform with Raspberry Pi and Kubernetes] series of articles. ## Troubleshooting @@ -512,8 +580,10 @@ Leverage the `preStop` hook instead of `postStart`. > Hooks **are not passed parameters**, and this includes environment variables > Use a script if you need them. See [container hooks] and [preStop hook doesn't work with env variables] -Since kubernetes version 1.9 and forth, volumeMounts behavior on secret, configMap, downwardAPI and projected have changed to Read-Only by default. -A workaround to the problem is to create an `emptyDir` Volume and copy the contents into it and execute/write whatever you need: +Since kubernetes version 1.9 and forth, volumeMounts behavior on secret, configMap, downwardAPI and projected have +changed to Read-Only by default. +A workaround to the problem is to create an `emptyDir` Volume and copy the contents into it and execute/write whatever +you need: ```yaml initContainers: @@ -585,7 +655,8 @@ Tools: - [`kubectx`+`kubens`][kubectx+kubens], alternative to [`kubie`][kubie] and [`kubeswitch`][kubeswitch] - [`kubeswitch`][kubeswitch], alternative to [`kubie`][kubie] and [`kubectx`+`kubens`][kubectx+kubens] - [`kube-ps1`][kube-ps1] -- [`kubie`][kubie], alternative to [`kubeswitch`][kubeswitch], and to [`kubectx`+`kubens`][kubectx+kubens] and [`kube-ps1`][kube-ps1] +- [`kubie`][kubie], alternative to [`kubeswitch`][kubeswitch], and to [`kubectx`+`kubens`][kubectx+kubens] and + [`kube-ps1`][kube-ps1] - [Minikube] - [Kubescape]