From ac3554b5514d516d52487ffb708b3f0e06881bf6 Mon Sep 17 00:00:00 2001 From: Michele Cereda Date: Sun, 17 Nov 2024 01:22:24 +0100 Subject: [PATCH] chore(kb/aws): extend ec2 disk information --- knowledge base/cloud computing/aws/ebs.md | 67 +++++++++++++++++------ knowledge base/cloud computing/aws/ec2.md | 40 +++++++++++++- 2 files changed, 88 insertions(+), 19 deletions(-) diff --git a/knowledge base/cloud computing/aws/ebs.md b/knowledge base/cloud computing/aws/ebs.md index e497c10..0cb51bf 100644 --- a/knowledge base/cloud computing/aws/ebs.md +++ b/knowledge base/cloud computing/aws/ebs.md @@ -3,6 +3,7 @@ Persistent [block storage][what is block storage?] for [EC2 Instances][ec2]. 1. [TL;DR](#tldr) +1. [Volume types](#volume-types) 1. [Snapshots](#snapshots) 1. [Encryption](#encryption) 1. [Further readings](#further-readings) @@ -10,7 +11,7 @@ Persistent [block storage][what is block storage?] for [EC2 Instances][ec2]. ## TL;DR -
+
Real world use cases ```sh @@ -29,27 +30,51 @@ aws ec2 wait snapshot-completed --snapshot-ids 'snap-0123456789abcdef0'
+Every EBS volume is local to, and available in, a single Availability Zone. + +Multiple EBS volumes can be attached to a single instance as long as both the volumes and the instance are in the same +Availability Zone. + +Depending on the volume and instance types, multiple instances can mount a single volume at the same time. + Volumes can have their size **increased**, but **not** reduced.
-After increase, the filesystem **must** be [extended][Extend the file system after resizing an EBS volume] to take -advantage of the change in size.
-Apparently, Linux machines are able to do that automatically with a reboot. +The volumes' filesystem is **not** automatically extended during the change in size, and **must** be +[extended manually][Extend the file system after resizing an EBS volume] to take advantage of the change in size.
+Linux-based instances should™ be able to take care of that automatically after reboot. + +[Pricing][amazon ebs pricing] + +## Volume types + +Refer [Amazon EBS volume types]. + +| | `gp3` | `gp2` | `io2` | `io1` | `st1` | `sc1` | +| ------------------- | -------------- | -------------- | -------------- | -------------- | ---------------- | ---------------- | +| Class | SSD | SSD | SSD | SSD | HDD | HDD | +| Annual failure rate | 0.1% - 0.2% | 0.1% - 0.2% | 0.001% | 0.1% - 0.2% | 0.1% - 0.2% | 0.1% - 0.2% | +| Size | 1 GiB - 16 TiB | 1 GiB - 16 TiB | 4 GiB - 64 TiB | 4 GiB - 16 TiB | 125 GiB - 16 TiB | 125 GiB - 16 TiB | +| Max IOPS | 16,000 | 16,000 | 256,000 | 64,000 | 500 | 250 | +| Max throughput | 1,000 MiB/s | 250 MiB/s | 4,000 MiB/s | 1,000 MiB/s | 500 MiB/s | 250 MiB/s | +| Multi-attach | No | No | Yes | Yes | No | No | +| NVMe reservations | No | No | Yes | No | No | No | +| Bootable | Yes | Yes | Yes | Yes | No | No | ## Snapshots -The first snapshot is **complete**, with all the volume's blocks being copied. All successive snapshots of the same -volume are **incremental**, with only the changes being copied.
+A volume's first snapshot is a **complete** snapshot of it, with _all the volume's blocks_ being copied over.
+All successive snapshots of the same volume are **incremental**, with _only the changes_ being copied over.
Incremental snapshots are stored in EBS' standard tier. Snapshots can be unbearably slow depending on the amount of data needing to be copied.
-For comparison, the first snapshot of a 200 GiB volume took about 2h to complete. +For comparison, the first snapshot of a standard 200 GiB `gp3` volume took about 2h to complete. Snapshots can be [archived][archive amazon ebs snapshots] to save money should they **not** need frequent nor fast retrieval.
When archived, incremental snapshots are converted to **full snapshots** and moved to EBS' archive tier. -> The minimum archive period is 90 days.
-> If deleting or permanently restoring an archived snapshot before the minimum archive period, one is billed for all the -> remaining days in the archive tier, rounded to the nearest hour. +> The **minimum** archival period is **90 days**.
+> Archived snapshots deleted or permanently restored before the end of the minimum archival period are billed for the +> whole period. When access to archived snapshots is needed, they need to be restored to the standard tier before use. Restoring can take **up to 72h**. @@ -99,7 +124,7 @@ snapshot for encrypted volumes is itself encrypted or unencrypted.
-
+
The original snapshot is not encrypted 1. EC2 sends a `CreateGrant` request to KMS to be allowed to encrypt the volume that is being created from the snapshot. @@ -117,17 +142,20 @@ snapshot for encrypted volumes is itself encrypted or unencrypted. 1. EC2 uses the plaintext data key in the Nitro hardware to encrypt disk I/O to the volume.
The plaintext data key persists in memory as long as the volume is attached to the instance. +
+ When KMS keys become unusable, the effect is **almost immediately** subject to **eventual** consistency.
The key state of the impacted KMS keys change to reflect their new condition, and all requests to use those keys in cryptographic operations fail. -EC2 uses the **data** key, not the KMS key itself, to encrypt all disk I/O while a volume is attached to -the instance. As such, there is **no** immediate effect on the EC2 instance or its attached EBS volumes when performing -an action that makes a key unusable.
-When the encrypted EBS volume is detached from the instance, however, EBS removes the data key from the Nitro hardware. -As such, the next time the encrypted EBS volume is attached to an EC2 instance the attachment will fail due EBS being -unable to use the KMS key to decrypt the volume's encrypted data key. To use the EBS volume again, one must make the KMS -key usable again. +EC2 encrypts all I/O to and from attached volumes using the **data** key, not the KMS key itself.
+There is **no** immediate effect on the EC2 instance or its attached EBS volumes when performing actions that make KMS +keys unusable. + +EBS removes data keys from the hardware when encrypted EBS volumes are detached from instances.
+Attaching EBS volumes which data keys are encrypted with unusable KMS keys to EC2 instances will fail, because EBS will +not be able to use the KMS keys to decrypt the data key used for the volume.
+Make the KMS key usable again to be able to attach such EBS volumes. ## Further readings @@ -138,6 +166,7 @@ key usable again. - [Automate snapshot lifecycles] - [Choose the best Amazon EBS volume type for your self-managed database deployment] - [Extend the file system after resizing an EBS volume] +- [Pricing][amazon ebs pricing] ### Sources @@ -159,6 +188,8 @@ key usable again. [ec2]: ec2.md +[amazon ebs pricing]: https://aws.amazon.com/ebs/pricing/ +[amazon ebs volume types]: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volume-types.html [archive amazon ebs snapshots]: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-archive.html [automate snapshot lifecycles]: https://docs.aws.amazon.com/ebs/latest/userguide/snapshot-ami-policy.html [choose the best amazon ebs volume type for your self-managed database deployment]: https://aws.amazon.com/blogs/storage/how-to-choose-the-best-amazon-ebs-volume-type-for-your-self-managed-database-deployment/ diff --git a/knowledge base/cloud computing/aws/ec2.md b/knowledge base/cloud computing/aws/ec2.md index a5ac27a..20c8a45 100644 --- a/knowledge base/cloud computing/aws/ec2.md +++ b/knowledge base/cloud computing/aws/ec2.md @@ -3,6 +3,7 @@ 1. [TL;DR](#tldr) 1. [Burstable instances](#burstable-instances) 1. [Disks](#disks) + 1. [Ephemeral storage](#ephemeral-storage) 1. [Metrics](#metrics) 1. [Auto scaling](#auto-scaling) 1. [Lifecycle hooks](#lifecycle-hooks) @@ -121,7 +122,38 @@ If the average CPU usage over a 24-hour period **exceeds** the baseline, instanc ## Disks -Refer [EBS] and [Device names for volumes on Amazon EC2 instances]. +Refer [EBS]. + +Volumes being attached to an EC2 instance require a device name for the instance to refer to. The block device driver in +the OS then assigns the volume an internal device name when mounting it, which _can_ be different from the name given in +the volume's definition. +Refer [Device names for volumes on Amazon EC2 instances]. + +One or more Provisioned IOPS SSD (`io1` or `io2`) volumes can be attached to **up to 16 instances** as long as those +instances reside **in the same Availability Zone**.
+Refer [Attach an EBS volume to multiple EC2 instances using Multi-Attach]. + +The maximum number of EBS volumes that instances can have attached depends on the instance's type and size.
+Refer instance volume limits. + +Each instance a volume is attached to has **full read and write permission** to the shared volume.
+This allows to achieve higher application availability in applications that can manage concurrent write operations +effectively. + +### Ephemeral storage + +Refer [Instance store temporary block storage for EC2 instances] for temporary storage of information that changes +frequently (e.g. buffers, caches, scratch data, temporary content). + +_Instance stores_ consist of one or more virtual volumes exposed as block devices. + +The size of an instance store and the number of devices available, varies by instance type and size.
+Not every instance type provides instance store volumes. + +Virtual devices for instance store volumes are given device names in order from `ephemeral0` to `ephemeral23`. + +There is no additional charge for using the instance store volumes provided with instances.
+Instance store volumes are **included** as part of the usage cost of an instance. ## Metrics @@ -210,6 +242,7 @@ TODO - [Announcing Amazon EC2 per second billing] - [How can I send memory and disk metrics from my EC2 instances to CloudWatch?] - [Device names for volumes on Amazon EC2 instances] +- [Amazon EBS volume limits for Amazon EC2 instances] ### Sources @@ -227,6 +260,8 @@ TODO - [Amazon EC2 Auto Scaling lifecycle hooks] - [EC2 Image Builder] - [CompleteLifecycleAction] +- [Instance store temporary block storage for EC2 instances] +- [Attach an EBS volume to multiple EC2 instances using Multi-Attach] +[amazon ebs volume limits for amazon ec2 instances]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_limits.html [amazon ec2 auto scaling lifecycle hooks]: https://docs.aws.amazon.com/autoscaling/ec2/userguide/lifecycle-hooks.html [amazon ec2 auto scaling]: https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html [announcing amazon ec2 per second billing]: https://aws.amazon.com/about-aws/whats-new/2017/10/announcing-amazon-ec2-per-second-billing/ +[attach an ebs volume to multiple ec2 instances using multi-attach]: https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes-multi.html [best practices for handling ec2 spot instance interruptions]: https://aws.amazon.com/blogs/compute/best-practices-for-handling-ec2-spot-instance-interruptions/ [burstable performance instances]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html [change the instance type]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html @@ -256,6 +293,7 @@ TODO [how can i send memory and disk metrics from my ec2 instances to cloudwatch?]: https://repost.aws/knowledge-center/cloudwatch-memory-metrics-ec2 [how to clone instance ec2]: https://repost.aws/questions/QUOrWudF3vRL2Vqtrv0M9lfQ/how-to-clone-instance-ec2 [iam roles for amazon ec2]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html +[instance store temporary block storage for ec2 instances]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html [key concepts and definitions for burstable performance instances]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-credits-baseline-concepts.html [retrieve instance metadata]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html [standard mode for burstable performance instances]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances-standard-mode.html