diff --git a/knowledge base/cloud computing/aws/ecs.md b/knowledge base/cloud computing/aws/ecs.md index fdb0f45..f30d26a 100644 --- a/knowledge base/cloud computing/aws/ecs.md +++ b/knowledge base/cloud computing/aws/ecs.md @@ -20,15 +20,16 @@ ## TL;DR _Tasks_ are the basic unit of deployment.
-Their details are specified in _task definitions_. - -_Standalone tasks_ are meant to perform some work, then stop much like batch processes.
-_Services_ run and maintain a defined number of instances of the same task simultaneously, and are meant to stay active -much like web servers. +They are instances of the set of containers specified in their own _task definition_. Tasks model and run one or more containers, much like Pods in Kubernetes.
Containers **cannot** run on ECS unless encapsulated in a task. +_Standalone tasks_ start a single task, which is meant to perform some work to completion and then stop (much like batch +processes would).
+_Services_ run and maintain a defined number of instances of the same task simultaneously, which are meant to stay +active and act as replicas of some service (much like web servers would). + Tasks are executed depending on their _launch type_ and _capacity providers_: - On EC2 instances that one owns, manages, and pays for. diff --git a/knowledge base/cloud computing/aws/efs.md b/knowledge base/cloud computing/aws/efs.md index 15bca50..c24fdb5 100644 --- a/knowledge base/cloud computing/aws/efs.md +++ b/knowledge base/cloud computing/aws/efs.md @@ -10,35 +10,91 @@ performance. ## TL;DR Built to scale on demand growing and shrinking automatically as files are added and removed.
-Accessible across most types of AWS compute instances, including EC2, ECS, EKS, Lambda, and Fargate. +Accessible across EC2, ECS, EKS, Lambda, and Fargate. -Supports the NFS v4.0 and v4.1 protocols. +Supports the **NFS v4.0** and **v4.1** protocols.
+A _mount target_ is required for any file system for clients to be able to use NFS to mount them. + +The file system's DNS name automatically resolves to the mount target's IP address in the Availability Zone of the +connecting EC2 instances.
+It follows the `{{ file-system-id }}.efs.{{ aws-region }}.amazonaws.com` convention. Available file system types: - _Regional_: redundant across **multiple** geographically separated AZs **within the same Region**. - _One Zone_: data stored within a **single AZ**, with all the limits it implies. -Default modes: +Available throughput modes: -- _General Purpose Performance_: ideal for latency-sensitive applications.
+- _Elastic_: scales automatically in real time to meet the needs of workloads' activity.
+ Only available for file systems using the General Purpose performance mode.
+ Default setting when not specified during creation. +- _Provisioned_: statically provides the specified level of throughput independently from the file system's size. +- _Bursting_: scales automatically with the amount of data in Standard storage. + +Available performance modes: + +- _General Purpose_: lowest per-operation latency.
+ Recommended for all file systems. Ideal for latency-sensitive applications.
Examples: web-serving environments, content-management systems, home directories, and general file serving. -- _Elastic Throughput_: designed to scale throughput performance automatically to meet the needs of workloads' activity. +- _Max I/O_: designed for highly parallelized workloads that **can** tolerate higher latencies than the General Purpose + mode.
+ **Not** supported by One Zone file systems or file systems using the Elastic throughput mode. + +Lifecycle management settings allow to automatically move files into and out of the lower-cost Infrequent Access storage +class based on access patterns.
+Leverages lifecycle policies. + +When creating file systems via the Console, the file system's lifecycle policy is configured by default with the +following settings: + +- Transition into IA set to 30 days since last access. +- TransitionToArchive set to 90 days since last access. +- Transition into Standard set to None. + +When creating file systems via the CLI or APIs, it is **not** possible to set lifecycle policies at the same time.
+One **must** wait until the file system is created, then use the `PutLifecycleConfiguration` API operation to update the +lifecycle policies. Provides file-system-access semantics like strong data consistency and file locking.
-Supports controlling access to file systems through POSIX permissions.
-Supports authentication, authorization, and encryption. +Supports: + +- Controlling access to file systems through POSIX permissions. +- Authentication and authorization. +- Encryption in transit and at rest. -EFS supports encryption in transit and encryption at rest.
Encryption at rest is enabled when creating a file system. In such case, all data and metadata is encrypted.
Encryption in transit is enabled when mounting a file system. Client access via NFS to EFS is controlled by both IAM policies and network security policies (i.e. security groups). Windows-based EC2 instances are **not** supported. +Automatic backups are enabled by default when creating file systems using the console.
+When creating file systems via the CLI or the APIs, automatic backups are enabled by default only when setting them up +to be One Zone file systems. + +
+ Usage + +```sh +# Get mount targets' information. +aws efs describe-mount-targets --query 'MountTargets[]' --file-system-id 'fs-0123456789abcdef0' + +# Get mount targets' IP address. +aws efs describe-mount-targets --query 'MountTargets[].IpAddress' --file-system-id 'fs-0123456789abcdef0' +aws efs describe-mount-targets --query 'MountTargets[].IpAddress' --mount-target-id 'fsmt-0123456789abcdef0' + +# Mount volumes. +mount -t 'nfs' -o 'nfsvers=4.0,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport' \ + 'fs-0123456789abcdef0.efs.eu-west-1.amazonaws.com:/' "$HOME/efs" +``` + +
+ ## Further readings - [Amazon Web Services] +- [How do I mount, unmount, automount, and on-premises mount my Amazon EFS file system?] ### Sources @@ -55,6 +111,7 @@ Windows-based EC2 instances are **not** supported. +[how do i mount, unmount, automount, and on-premises mount my amazon efs file system?]: https://repost.aws/knowledge-center/efs-mount-automount-unmount-steps [what is amazon elastic file system?]: https://docs.aws.amazon.com/efs/latest/ug/whatisefs.html diff --git a/snippets/aws/other commands.fish b/snippets/aws/other commands.fish index edfbe96..d4a9beb 100644 --- a/snippets/aws/other commands.fish +++ b/snippets/aws/other commands.fish @@ -104,6 +104,22 @@ aws ecs describe-services --cluster 'stg' --services 'grafana' aws ecs wait services-stable --cluster 'stg' --services 'grafana' +### +# EFS +# ------------------ +### + +# Get mount targets' information. +aws efs describe-mount-targets --query 'MountTargets[]' --file-system-id 'fs-0123456789abcdef0' + +# Get mount targets' IP address. +aws efs describe-mount-targets --query 'MountTargets[].IpAddress' --output 'text' --file-system-id 'fs-0123456789abcdef0' +aws efs describe-mount-targets --query 'MountTargets[].IpAddress' --output 'text' --mount-target-id 'fsmt-0123456789abcdef0' + +# Mount volumes. +mount -t 'nfs' -o 'nfsvers=4.0,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport' \ + 'fs-0123456789abcdef0.efs.eu-west-1.amazonaws.com:/' "$HOME/efs" + ### # EKS # ------------------ diff --git a/snippets/pulumi/aws/run loki in monolithic mode on ecs fargate with config file in efs.ts b/snippets/pulumi/aws/run loki in monolithic mode on ecs fargate with config file in efs.ts new file mode 100644 index 0000000..98b0abe --- /dev/null +++ b/snippets/pulumi/aws/run loki in monolithic mode on ecs fargate with config file in efs.ts @@ -0,0 +1,272 @@ +import * as aws from "@pulumi/aws"; + + +// requirements - start + +const vpc_output = aws.ec2.getVpcOutput({ + filters: [{ + name: "tag:Name", + values: "Default", + }], +}); +const privateSubnets_output = aws.ec2.getSubnetOutput({ + filters: [{ + name: "tag:Type", + values: [ "Private" ], + }], +}); +const ecsCluster_output = aws.ecs.getClusterOutput({ clusterName: "someCluster" }); +const dnsZone_output = aws.route53.getZoneOutput({ name: "example.org." }); + +// requirements - end + + +// resources - start + +/** + * The ECS service *will* fail and continuously restart until an acceptable configuration file for Loki named + * 'local-config.yaml' is uploaded to the EFS volume. + **/ + +const securityGroup = new aws.ec2.SecurityGroup( + "loki", + { + vpcId: vpc_output.apply((vpc: aws.ec2.Vpc) => vpc.id), + name: "Loki", + description: "Manage access to and from Loki", + tags: { + Name: "Loki", + Application: "Loki", + }, + + ingress: [ + { + description: "NFS access to EFS", + self: true, + protocol: "tcp", + fromPort: 2049, + toPort: 2049, + }, + { + description: "HTTP server", + cidrBlocks: [ "0.0.0.0/0" ], + ipv6CidrBlocks: [ "::/0" ], + protocol: "tcp", + fromPort: 3100, + toPort: 3100, + }, + { + description: "gRPC server", + cidrBlocks: [ "0.0.0.0/0" ], + ipv6CidrBlocks: [ "::/0" ], + protocol: "tcp", + fromPort: 9095, + toPort: 9095, + }, + ], + egress: [{ + description: "Allow all", + cidrBlocks: [ "0.0.0.0/0" ], + ipv6CidrBlocks: [ "::/0" ], + protocol: "-1", + fromPort: 0, + toPort: 0, + }], + }, +); + +const efsVolume_config = new aws.efs.FileSystem( + "loki-config", + { + creationToken: "loki-config", + tags: { + Name: "loki-config", + Application: "Loki", + }, + }, +); +privateSubnets_output.apply( + (subnets: aws.ec2.Subnet[]) => subnets.forEach( + subnet => new aws.efs.MountTarget( + `loki-config-${subnet.availabilityZone}`, + { + fileSystemId: efsVolume_config.id, + subnetId: subnet.id, + securityGroups: [ securityGroup.id ], + }, + ), + ), +); + +const taskDefinition = new aws.ecs.TaskDefinition( + "loki", + { + family: "Loki", + tags: { Application: "Loki" }, + + networkMode: "awsvpc", + requiresCompatibilities: [ "FARGATE" ], + cpu: "256", // Fargate requirement + memory: "512", // Fargate requirement + executionRoleArn: "arn:aws:iam::012345678901:role/ecsTaskExecutionRole", // logging requirement + volumes: [{ + name: "config", + efsVolumeConfiguration: { fileSystemId: efsVolume_config.id }, + }], + containerDefinitions: JSON.stringify([{ + name: "loki", + image: "grafana/loki:3.3.2", + essential: true, + environment: [], // specified to avoid showing changes on every run + systemControls: [], // specified to avoid showing changes on every run + volumesFrom: [], // specified to avoid showing changes on every run + mountPoints: [{ + sourceVolume: "config", + containerPath: "/etc/loki", + readOnly: true, + }], + healthCheck: { + command: [ + "CMD-SHELL", + "wget -qO- localhost:3100/ready || exit 1", + ], + startPeriod: 15, + }, + portMappings: [ + { + name: "http-server", + appProtocol: "http", + containerPort: 3100, + }, + { + name: "grpc-server", + appProtocol: "grpc", + containerPort: 9095, + }, + ], + logConfiguration: { + logDriver: "awslogs", + options: { + "awslogs-create-group": "true", + "awslogs-group": "/ecs/loki", + "awslogs-region": "eu-west-1", + "awslogs-stream-prefix": "ecs", + }, + }, + }]), + }, +); + +const targetGroup_http = new aws.alb.TargetGroup( + "loki-http", + { + vpcId: vpc_output.apply((vpc: aws.ec2.Vpc) => vpc.id), + name: "loki-http", + tags: { Application: "Loki" }, + + targetType: "ip", + ipAddressType: "ipv4", + protocol: "HTTP", + port: 3100, + healthCheck: { path: "/ready" }, + }, +); +const targetGroup_grpc = new aws.alb.TargetGroup( + "loki-grpc", + { + vpcId: vpc_output.apply((vpc: aws.ec2.Vpc) => vpc.id), + name: "loki-grpc", + tags: { Application: "Loki" }, + + targetType: "ip", + ipAddressType: "ipv4", + protocol: "HTTP", + protocolVersion: "GRPC", + port: 9095, + }, +); +const loadBalancer = new aws.alb.LoadBalancer( + "loki", + { + name: "Loki", + tags: { Application: "Loki" }, + + internal: true, + ipAddressType: "ipv4", + subnets: privateSubnets_output.apply((subnets: aws.ec2.Subnet[]) => subnets.map(subnet => subnet.id)), + securityGroups: [ securityGroup.id ], + accessLogs: { bucket: "" }, + }, +); +new aws.route53.Record( + "loki", + { + zoneId: dnsZone_output.apply((zone: aws.route53.Zone) => zone.zoneId), + name: "loki.example.org", + type: "A", + aliases: [{ + name: loadBalancer.dnsName, + zoneId: loadBalancer.zoneId, + evaluateTargetHealth: true, + }], + }, +); +new aws.alb.Listener( + "loki-http", + { + tags: { Application: "Loki" }, + loadBalancerArn: loadBalancer.arn, + port: 3100, + protocol: "HTTP", + defaultActions: [{ + order: 1, + targetGroupArn: targetGroup_http.arn, + type: "forward", + }], + }, +); +// new aws.alb.Listener( +// // FIXME: Listener protocol 'HTTP' is not supported with a target group with the protocol-version 'GRPC' +// "loki-grpc", +// { +// tags: { Application: "Loki" }, +// loadBalancerArn: loadBalancer.arn, +// port: 9095, +// protocol: "HTTP", +// defaultActions: [{ +// order: 1, +// targetGroupArn: targetGroup_grpc.arn, +// type: "forward", +// }], +// }, +// ); +new aws.ecs.Service( + "loki", + { + name: "Loki", + tags: { Application: "Loki" }, + + cluster: ecsCluster_output.arn, + taskDefinition: taskDefinition.arn, + desiredCount: 1, + launchType: "FARGATE", + networkConfiguration: { + subnets: privateSubnets_output.apply((subnets: aws.ec2.Subnet[]) => subnets.map(subnet => subnet.id)), + securityGroups: [ securityGroup.id ], + }, + loadBalancers: [ + { + containerName: "loki", + containerPort: 3100, + targetGroupArn: targetGroup_http.arn, + }, + // { + // containerName: "loki", + // containerPort: 9095, + // targetGroupArn: targetGroup_grpc.arn, + // }, + ], + }, +); + +// resources - end diff --git a/snippets/pulumi/aws/run loki in monolithic mode on ecs.ts b/snippets/pulumi/aws/run loki in monolithic mode on ecs fargate.ts similarity index 95% rename from snippets/pulumi/aws/run loki in monolithic mode on ecs.ts rename to snippets/pulumi/aws/run loki in monolithic mode on ecs fargate.ts index a2cffca..25d3059 100644 --- a/snippets/pulumi/aws/run loki in monolithic mode on ecs.ts +++ b/snippets/pulumi/aws/run loki in monolithic mode on ecs fargate.ts @@ -1,15 +1,27 @@ import * as aws from "@pulumi/aws"; + +// requirements - start + const vpc_output = aws.ec2.getVpcOutput({ filters: [{ name: "tag:Name", values: "Default", }], }); - +const privateSubnets_output = aws.ec2.getSubnetOutput({ + filters: [{ + name: "tag:Type", + values: [ "Private" ], + }], +}); +const ecsCluster_output = aws.ecs.getClusterOutput({ clusterName: "someCluster" }); const dnsZone_output = aws.route53.getZoneOutput({ name: "example.org." }); -const ecsCluster_output = aws.ecs.getClusterOutput({ clusterName: "someCluster" }); +// requirements - end + + +// resources - start const securityGroup = new aws.ec2.SecurityGroup( "loki", @@ -67,9 +79,9 @@ const taskDefinition = new aws.ecs.TaskDefinition( image: "grafana/loki:3.3.2", essential: true, environment: [], // specified to avoid showing changes on every run + systemControls: [], // specified to avoid showing changes on every run volumesFrom: [], // specified to avoid showing changes on every run mountPoints: [], // specified to avoid showing changes on every run - systemControls: [], // specified to avoid showing changes on every run healthCheck: { command: [ "CMD-SHELL", @@ -102,12 +114,6 @@ const taskDefinition = new aws.ecs.TaskDefinition( }, ); -const privateSubnets_output = aws.ec2.getSubnetOutput({ - filters: [{ - name: "tag:Type", - values: [ "Private" ], - }], -}); const targetGroup_http = new aws.alb.TargetGroup( "loki-http", { @@ -119,9 +125,7 @@ const targetGroup_http = new aws.alb.TargetGroup( ipAddressType: "ipv4", protocol: "HTTP", port: 3100, - healthCheck: { - path: "/ready", - }, + healthCheck: { path: "/ready" }, }, ); const targetGroup_grpc = new aws.alb.TargetGroup( @@ -148,9 +152,7 @@ const loadBalancer = new aws.alb.LoadBalancer( ipAddressType: "ipv4", subnets: privateSubnets_output.apply((subnets: aws.ec2.Subnet[]) => subnets.map(subnet => subnet.id)), securityGroups: [ securityGroup.id ], - accessLogs: { - bucket: "", - }, + accessLogs: { bucket: "" }, }, ); new aws.route53.Record( @@ -181,7 +183,7 @@ new aws.alb.Listener( }, ); // new aws.alb.Listener( -// FIXME: Listener protocol 'HTTP' is not supported with a target group with the protocol-version 'GRPC' +// // FIXME: Listener protocol 'HTTP' is not supported with a target group with the protocol-version 'GRPC' // "loki-grpc", // { // tags: { Application: "Loki" }, @@ -223,3 +225,5 @@ new aws.ecs.Service( ], }, ); + +// resources - end