oam/knowledge base/cloud computing/aws/eks.md

Elastic Kubernetes Service

1. TL;DR
2. Requirements
3. Creation procedure
4. Create worker nodes
   1. Create managed node groups
   2. Schedule pods on Fargate
5. Secrets encryption through KMS
6. Troubleshooting
7. Further readings
   1. Sources

TL;DR

When one creates a cluster, one really creates just the cluster's control plane and its dedicated nodes.
Pods can be scheduled on any combination of self-managed nodes, managed node groups and Fargate, depending on the cluster's properties.

EKS automatically installs self-managed add-ons like the AWS VPC CNI plugin, kube-proxy and CoreDNS.
Disable them in the cluster's definition.

EKS automatically creates a Security Group for the control plane upon cluster creation.
This apparently cannot be avoided or customized in the cluster's definition (e.g. using IaC tools like Pulumi or Terraform):

error: aws:eks/cluster:Cluster resource 'cluster' has a problem: Value for unconfigurable attribute. Can't configure a value for "vpc_config.0.cluster_security_group_id": its value will be decided automatically based on the result of applying this configuration.

For some reason, giving resources a tag like aks:eks:cluster-name succeeds, but has no effect (it is not applied).

Usage

# Create clusters.
aws eks create-cluster \
   --name 'DeepThought' \
   --role-arn 'arn:aws:iam::000011112222:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS' \
   --resources-vpc-config 'subnetIds=subnet-11112222333344445,subnet-66667777888899990'

Requirements

• [suggestion] 1 (one) custom Cluster Service Role with the AmazonEKSClusterPolicy policy attached or similar permissions.

  Kubernetes clusters managed by EKS make calls to other AWS services on the user behalf to manage the resources that the cluster uses.
  For a cluster to be allowed to make those calls, it requires to have an IAM role assigned, and this role must have:

  • The AmazonEKSClusterPolicy policy attached to it, or
  • Comparable permissions.

  To create clusters which would not require access to any other AWS resource, one can assign the cluster the AWSServiceRoleForAmazonEKS service-linked role directly ^1,2.

  Amazon EKS uses the service-linked role named AWSServiceRoleForAmazonEKS - The role allows Amazon EKS to manage clusters in your account. The attached policies allow the role to manage the following resources: network interfaces, security groups, logs, and VPCs.

  Prior to October 3, 2023, AmazonEKSClusterPolicy was required on the IAM role for each cluster.

  Prior to April 16, 2020, AmazonEKSServicePolicy was also required and the suggested name was eksServiceRole. With the AWSServiceRoleForAmazonEKS service-linked role, that policy is no longer required for clusters created on or after April 16, 2020.

  Pro tip

  Should one want to use more advanced features like encryption with managed keys, the role will need access to the referenced resources.
  In this case it would probably be better to create a custom role instead of assigning permissions to the built-in one.

• [suggestion] 1+ (one or more) custom service role(s) for the pod executors, with the required policies attached or similar permissions.

  The reasons and required permissions vary depending on the type of executor.
  See the corresponding section under Create worker nodes.

Creation procedure

1. Create a VPC, if one does not have them already, with public and private subnets that meet EKS' requirements.

   Example in Cloudformation

2. Create a custom IAM role for the cluster if needed (see Requirements).
   

3. Attach the required policies to the role used in the cluster.

   Example in CLI

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": "sts:AssumeRole",
       "Principal": {
         "Service": "eks.amazonaws.com"
       }
     }]
   }
   

   aws iam create-role \
     --role-name 'DeepThinker' \
     --assume-role-policy-document 'file://eks-cluster-role-trust-policy.json'
   

   Example in Pulumi

   const cluster_assumeRole_policy = JSON.stringify({
     Version: "2012-10-17",
     Statement: [{
       Effect: "Allow",
       Action: "sts:AssumeRole",
       Principal: {
         Service: "eks.amazonaws.com",
       },
     }],
   });
   const cluster_service_role = new aws.iam.Role("cluster-service-role", {
     assumeRolePolicy: cluster_assumeRole_policy,
     name: "DeepThinker",
     …
   });
   

   
   

4. Create the cluster.

   Example in CLI

   aws eks create-cluster \
     --name 'DeepThought' \
     --role-arn 'arn:aws:iam::000011112222:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS' \
     --resources-vpc-config 'subnetIds=subnet-11112222333344445,subnet-66667777888899990'
   

   Example in Pulumi

   const cluster = new aws.eks.Cluster("cluster", {
     name: "DeepThought",
     roleArn: cluster_service_role.arn,
     vpcConfig: {
       subnetIds: [
         "subnet-11112222333344445",
         "subnet-66667777888899990",
       ],
     },
     …
   });
   

   
   

5. Connect to the cluster.

   $ aws eks update-kubeconfig --name 'DeepThought'
   Added new context arn:aws:eks:eu-east-1:000011112222:cluster/DeepThought to /home/itsAme/.kube/config
   
   $ kubectl cluster-info
   Kubernetes control plane is running at https://FB32A9C4A3D6BBC82695B1936BF4AAA3.gr7.eu-east-1.eks.amazonaws.com
   CoreDNS is running at https://FB32A9C4A3D6BBC82695B1936BF4AAA3.gr7.eu-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
   

6. Create some worker nodes.

7. TODO

Create worker nodes

See step 3 of the getting started guide.

Create managed node groups

See Choosing an Amazon EC2 instance type.

Additional requirements:

• [suggestion] 1 (one) custom Node Service Role with the AmazonEKSFargatePodExecutionRolePolicy policy attached or similar permissions.

  The EKS nodes' kubelet makes calls to the AWS APIs on one's behalf.
  Nodes receive permissions for these API calls through an IAM instance profile and associated policies.

  For a node to be allowed to make those calls, it requires to have an IAM instance profile assigned, and this profile must use a role with:

  • The AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly and AmazonEKS_CNI_Policy policies attached to it, or
  • Comparable permissions.

• If the nodes are to be created in private subnets, the cluster must provide its private API server endpoint.
  Set the cluster's vpc_config.0.endpoint_private_access attribute to true.

Procedure:

1. Create a custom IAM role for the nodes if needed (see Requirements).

2. Attach the required policies to the role used by the nodes.

   Example in CLI

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Principal": {
           "Service": "ec2.amazonaws.com"
         }
       }
     ]
   }
   

   aws iam create-role \
     --role-name 'DeepThinkerNode' \
     --assume-role-policy-document 'file://eks-node-role-trust-policy.json'
   aws iam attach-role-policy \
     --policy-arn 'arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy' \
     --role-name 'DeepThinkerNode'
   aws iam attach-role-policy \
     --policy-arn 'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly' \
     --role-name 'DeepThinkerNode'
   aws iam attach-role-policy \
     --policy-arn 'arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy' \
     --role-name 'DeepThinkerNode'
   

   Example in Pulumi

   const nodes_assumeRole_policy = JSON.stringify({
     Version: "2012-10-17",
     Statement: [{
       Effect: "Allow",
       Action: "sts:AssumeRole",
       Principal: {
         Service: "ec2.amazonaws.com",
       },
     }],
   });
   const nodes_service_role = new aws.iam.Role("nodes-service-role", {
     assumeRolePolicy: nodes_assumeRole_policy,
     managedPolicyArns: [
       // alternatively, use RolePolicyAttachments
       "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
       "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
       "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
     ],
     name: "DeepThinkerNode",
     …
   });
   

   
   

3. Create the desired node groups.

   Example in CLI

   aws eks create-nodegroup \
     --cluster-name 'DeepThought' \
     --nodegroup-name 'alpha' \
     --scaling-config 'minSize=1,maxSize=3,desiredSize=1' \
     --node-role-arn 'arn:aws:iam::000011112222:role/DeepThinkerNodeRole' \
     --subnets 'subnet-11112222333344445' 'subnet-66667777888899990'
   

   Example in Pulumi

   const ec2Nodes_group0 = new aws.eks.NodeGroup("ec2Nodes_group0", {
     clusterName: cluster.name,
     nodeRoleArn: nodes_service_role.arn,
     scalingConfig: {
       minSize: 1,
       maxSize: 3,
       desiredSize: 1,
     },
     subnetIds: cluster.vpcConfig.subnetIds,
     …
   });
   

   
   

Schedule pods on Fargate

Additional requirements:

• [suggestion] 1 (one) custom Fargate Service Role with the AmazonEKSFargatePodExecutionRolePolicy policy attached or similar permissions.

  To create pods on Fargate, the components running on Fargate must make calls to the AWS APIs on one's behalf.
  This is so that it can take actions such as pull container images from ECR or route logs to other AWS services.

  For a cluster to be allowed to make those calls, it requires to have a Fargate profile assigned, and this profile must use a role with:

  • The AmazonEKSFargatePodExecutionRolePolicy policy attached to it, or
  • Comparable permissions.

• 1+ (one or more) Fargate profile(s).

Procedure:

1. Create a custom IAM role for the Fargate profile if needed (see Requirements).

2. Attach the required policies to the role used by the profile.

   Example in CLI

   
   

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "sts:AssumeRole",
         "Principal": {
           "Service": "eks-fargate-pods.amazonaws.com"
         },
         "Condition": {
            "ArnLike": {
               "aws:SourceArn": "arn:aws:eks:region-code:111122223333:fargateprofile/my-cluster/*"
            }
         }
       }
     ]
   }
   

   aws iam create-role \
     --role-name 'DeepThinker' \
     --assume-role-policy-document 'file://eks-cluster-role-trust-policy.json'
   aws iam attach-role-policy \
     --policy-arn 'arn:aws:iam::aws:policy/AmazonEKSClusterPolicy' \
     --role-name 'DeepThinker'
   

   
   

3. TODO

Secrets encryption through KMS

See Enabling secret encryption on an existing cluster.

TL;DR:

1. Make sure the role used in the cluster has access to the used key with kms:DescribeKey and kms:CreateGrant permissions.

2. Configure the cluster:

   Example in CLI

   aws eks associate-encryption-config \
     --cluster-name 'DeepThought' \
     --encryption-config '[{
       "provider": { "keyArn": "arn:aws:kms:eu-west-1:000011112222:key/33334444-5555-6666-7777-88889999aaaa" },
       "resources": [ "secrets" ]
     }]'
   

Troubleshooting

See Amazon EKS troubleshooting.

Further readings

• Kubernetes
• EKS Workshop
• Pulumi
• Terraform

Sources

• Getting started with Amazon EKS - AWS Management Console and AWS CLI
• aws eks create-cluster
• aws eks create-nodegroup
• Using service-linked roles for Amazon EKS
• Service-linked role permissions for Amazon EKS
• Amazon EKS cluster IAM role
• Amazon EKS VPC and subnet requirements and considerations
• Amazon EKS security group requirements and considerations
• Amazon EKS clusters
• Amazon EKS add-ons
• Enabling secret encryption on an existing cluster
• Choosing an Amazon EC2 instance type