diff --git a/knowledge base/cloud computing/aws/README.md b/knowledge base/cloud computing/aws/README.md index 6a36813..b414ca8 100644 --- a/knowledge base/cloud computing/aws/README.md +++ b/knowledge base/cloud computing/aws/README.md @@ -10,7 +10,7 @@ 1. [GuardDuty](#guardduty) 1. [EventBridge](#eventbridge) 1. [Inspector](#inspector) - 1. [RDS](#rds) + 1. [KMS](#kms) 1. [Security Hub](#security-hub) 1. [Resource constraints](#resource-constraints) 1. [Access control](#access-control) @@ -60,6 +60,7 @@ One can can rapidly remapping addresses to other instances in one's account and | [EventBridge] | FIXME | | [GuardDuty] | Threat detection | | [Inspector] | FIXME | +| [KMS] | Key management | | [OpenSearch] | ELK, logging | | [RDS] | Databases | | [S3] | Storage | @@ -89,8 +90,6 @@ This feature is automatically enabled by default when one starts using one of th ### Config -FIXME - Compliance service for assessing and auditing AWS resources. Provides an inventory of resources.
@@ -113,15 +112,11 @@ Sample templates for compliance standards and benchmarks are available. ### Detective -FIXME - Uses ML and graphs to try and identify the root cause of security issues.
Creates visualizations with details and context by leveraging events from VPC Flow Logs, CloudTrail and GuardDuty. ### GuardDuty -FIXME - Threat detection service. It continuously monitors accounts and workloads for malicious activity and delivers security findings for visibility and @@ -139,73 +134,62 @@ _Threat IP List_ is a blacklist of **public IPs and CIDRs** that will be used by ### EventBridge -FIXME +TODO ### Inspector -FIXME +TODO -### RDS +### KMS -Read replicas **can** be promoted to standalone DB instances.
-See [Working with DB instance read replicas]. +_Key material_ is the cryptographic secret of Keys that is used in encryption operations. -Disk free metrics are available in CloudWatch. +Enabling automatic key rotation for a KMS key makes the service generate new cryptographic material for the key every +year by default.
+Specify a custom rotation period to customize that time frame. -Automatic backups are **enabled** by default.
-RDS will automatically create storage volume snapshots of the **entire** DB instances.
-Backups occur during a daily user-configurable 30 minute period backup window and are kept for a configurable number of -up to 35 days (_backup retention period_). One can recover DB instances to any point in time during the backup retention -period. +Perform on-demand rotation should you need to immediately initiate key material rotation.
+This works regardless of whether the automatic key rotation is enabled or not. On-demand rotations do **not** change +existing automatic rotation schedules. -DB instances must be in the `available` state for automated backups to occur.
-Automated backups don't occur while DB instances are in other states (i.e., `storage_full`). +KMS saves **all** previous versions of the cryptographic material in perpetuity to allow decryption of any data +encrypted with keys.
+Rotated key material is **not** deleted until the key itself is deleted. -Automated backups don't occur while a DB snapshot copy is running in the same AWS Region for the same database. +Track the rotation of key material [CloudWatch], CloudTrail, and the KMS console.
+Alternatively, use the `GetKeyRotationStatus` operation to verify whether automatic rotation is enabled for a key and +identify any in progress on-demand rotations. Use the `ListKeyRotations` operation to view the details of completed +rotations. -Back up DB instances manually by creating DB snapshots.
-The first snapshot contains the data for the full database. Subsequent snapshots of the same database are incremental. +When using a rotated KMS key to encrypt data, KMS uses the **current** key material.
+When using the same rotated KMS key to decrypt ciphertext, KMS uses the version of the key material that was used for +encryption.
+One **cannot** select a particular version of key materials for decrypt operations. This automation allows to safely use +rotated KMS keys in applications and AWS services without code changes. -One can copy both automatic and manual DB snapshots, but only share manual DB snapshots. +Automatic key rotation has no effect on the data that KMS keys protect: it does **not** rotate the data generated by +rotated keys, re-encrypts any data protected by the keys, nor it will mitigate the effect of compromised data keys. -RDS backup storage for each Region is composed of both the automated backups and manual DB snapshots for that -Region.
-Moving snapshots to other Regions increases the backup storage in the destination Regions. +KMS supports automatic and on-demand key rotation only for symmetric encryption keys with key material that KMS itself +creates.
+Automatic rotation is optional for customer managed KMS keys. KMS rotates the key material for AWS managed keys on an +yearly basis. Rotation of AWS owned KMS keys is managed by the AWS service that owns the key. -Backups are stored in [S3]. +Key rotation only changes the key material, not the key's properties.
+The key is considered the same logical resource, regardless of whether or how many times its key material changes. -Should one choose to retain automated backups when deleting DB instances, its automated backups are saved for the full -retention period, otherwise all automated backups are deleted with the instance.
-After they are deleted, automated backups can't be recovered. +Creating a new key and using it in place of the original one has the same effect as rotating the key material in an +existing key.
+This is considered a _manual_ key rotation and is a good choice to rotate keys that are not eligible for automatic key +rotation. -Should one choose to have RDS create a final DB snapshot before deleting a DB instance, one can use that or previously -created manual snapshots to recover it. +AWS charges a monthly fee for the first and second rotation of key material maintained for each key.
+This price increase is capped at the second rotation. Any subsequent rotations will **not** be billed. -Manual snapshots are **not** deleted. - -One can store up to 100 manual snapshots per Region. - -One can choose any of the following retention periods for instances' Performance Insights data: - -- 7 days (default, free tier). -- _n_ months, where n is a number from 1 to 24.
- In CLI and IaC, this number must be _n*31_. - -One can export DB snapshot data to [S3] buckets. - -The export process runs in the background and does **not** affect the performance of active DB instances. - -RDS extracts data from snapshots and stores it in Apache Parquet format.
-By default, **all** data in the snapshots is exported; one can export specific sets of databases, schemas, or tables. - -After the export, one can analyze the data directly through -[Athena](https://docs.aws.amazon.com/athena/latest/ug/parquet-serde.html) or -[Redshift Spectrum](https://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-copy-from-columnar.html). +Each key counts as one when calculating key resource quotas, regardless of the number of rotated key material versions. ### Security Hub -FIXME - Aggregator of findings for security auditing. > Uses [Config] to check resources' configuration by leveraging compliancy rules. @@ -328,7 +312,6 @@ Examples: - [Using service-linked roles] - [Best Practices for Tagging AWS Resources] - [Automating DNS-challenge based LetsEncrypt certificates with AWS Route 53] -- [Working with DB instance read replicas] - AWS' [CLI] - [Configuring EC2 Disk alert using Amazon CloudWatch] - [aws.permissions.cloud] @@ -357,11 +340,9 @@ Examples: - [Use an IAM role in the AWS CLI] - [Creating a role to delegate permissions to an IAM user] - [How to use the PassRole permission with IAM roles] -- [Introduction to backups][rds introduction to backups] for RDS -- [Pricing and data retention for Performance Insights][rds pricing and data retention for performance insights] for - RDS - [Exporting DB snapshot data to Amazon S3] - [I'm trying to export a snapshot from Amazon RDS MySQL to Amazon S3, but I'm receiving an error. Why is this happening?] +- [Rotating AWS KMS keys] @@ -386,6 +367,7 @@ Examples: [ecs]: ecs.md [eks]: eks.md [opensearch]: opensearch.md +[rds]: rds.md [s3]: s3.md [sagemaker]: sagemaker.md @@ -406,8 +388,7 @@ Examples: [iam json policy elements: sid]: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_sid.html [nat gateways]: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html [not authorized to perform: sts:assumerole]: https://repost.aws/questions/QUOY5XngCtRyOX4Desaygz8Q/not-authorized-to-perform-sts-assumerole -[rds introduction to backups]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html -[rds pricing and data retention for performance insights]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Overview.cost.html +[rotating aws kms keys]: https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html [services that publish cloudwatch metrics]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html [subnets for your vpc]: https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html [test your roles' access policies using the aws identity and access management policy simulator]: https://aws.amazon.com/blogs/security/test-your-roles-access-policies-using-the-aws-identity-and-access-management-policy-simulator/ @@ -419,7 +400,6 @@ Examples: [what is amazon vpc?]: https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html [what is aws config?]: https://docs.aws.amazon.com/config/latest/developerguide/WhatIsConfig.html [what is cloudwatch]: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html -[working with db instance read replicas]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html [automating dns-challenge based letsencrypt certificates with aws route 53]: https://johnrix.medium.com/automating-dns-challenge-based-letsencrypt-certificates-with-aws-route-53-8ba799dd207b diff --git a/knowledge base/cloud computing/aws/cli.md b/knowledge base/cloud computing/aws/cli.md index ace875f..671b110 100644 --- a/knowledge base/cloud computing/aws/cli.md +++ b/knowledge base/cloud computing/aws/cli.md @@ -108,11 +108,6 @@ aws iam detach-user-policy --user-name 'me-user' \ aws iam delete-user-policy --user-name 'me-user' --policy-name 'user-ro-access-bucket' -# Show RDS instances. -aws rds describe-db-instances -aws rds describe-db-instances --output 'json' --query "DBInstances[?(DBInstanceIdentifier=='master-prod')]" - - # List hosted zones. aws route53 list-hosted-zones @@ -147,6 +142,7 @@ Subcommands not listed here are in their own service-specific article: [`ec2`][ec2 tldr] | [`ecr`][ecr tldr] | [`eks`][eks tldr] | +[`rds`][rds tldr] | [`s3`][s3 tldr] | [`ssm`][ssm tldr] @@ -280,6 +276,7 @@ Solutions: [ec2 tldr]: ec2.md#tldr [ecr tldr]: ecr.md#tldr [eks tldr]: eks.md#tldr +[rds tldr]: rds.md#tldr [s3 tldr]: s3.md#tldr [ssm tldr]: ssm.md#tldr diff --git a/knowledge base/cloud computing/aws/rds.md b/knowledge base/cloud computing/aws/rds.md new file mode 100644 index 0000000..c57c8fe --- /dev/null +++ b/knowledge base/cloud computing/aws/rds.md @@ -0,0 +1,294 @@ +# Amazon Relational Database Service + +1. [TL;DR](#tldr) +1. [Backup](#backup) + 1. [Automatic backups](#automatic-backups) + 1. [Manual backups](#manual-backups) + 1. [Export snapshots to S3](#export-snapshots-to-s3) +1. [Restore](#restore) +1. [Encryption](#encryption) +1. [Further readings](#further-readings) + 1. [Sources](#sources) + +## TL;DR + +
+ Usage + +```sh +# Show RDS instances. +aws rds describe-db-instances +aws rds describe-db-instances --output 'json' --query "DBInstances[?(DBInstanceIdentifier=='master-prod')]" + +# Restore instances from snapshots. +aws rds restore-db-instance-from-db-snapshot \ + --db-instance-identifier 'mynewdbinstance' \ + --db-snapshot-identifier 'mydbsnapshot' + +# Start export tasks. +aws rds start-export-task \ + --export-task-identifier 'db-finalSnapshot-2024' \ + --source-arn 'arn:aws:rds:eu-west-1:012345678901:snapshot:db-prod-final-2024' \ + --s3-bucket-name 'backups' \ + --iam-role-arn 'arn:aws:iam::012345678901:role/CustomRdsS3Exporter' \ + --kms-key-id 'arn:aws:kms:eu-west-1:012345678901:key/abcdef01-2345-6789-abcd-ef0123456789' + +# Get export tasks' status. +aws rds describe-export-tasks +aws rds describe-export-tasks --export-task-identifier 'my-snapshot-export' + +# Cancel tasks. +aws rds cancel-export-task --export-task-identifier 'my_export' +``` + +
+ +Read replicas **can** be promoted to standalone DB instances.
+See [Working with DB instance read replicas]. + +Disk free metrics are available in CloudWatch. + +One can choose any of the following retention periods for instances' Performance Insights data: + +- 7 days (default, free tier). +- _n_ months, where n is a number from 1 to 24.
+ In CLI and IaC, this number must be _n*31_. + +## Backup + +RDS backup storage for each Region is calculated from both the automated backups and manual DB snapshots for that +Region.
+Moving snapshots to other Regions increases the backup storage in the destination Regions. + +Backups are stored in [S3]. + +Should one choose to retain automated backups when deleting DB instances, those backups are saved for the full retention +period; otherwise, all automated backups are deleted with the instance.
+After automated backups are deleted, they **cannot** be recovered. + +Should one choose to have RDS create a final DB snapshot before deleting a DB instance, one can use that or previously +created manual snapshots to recover it. + +### Automatic backups + +Automatic backups are storage volume snapshots of **entire** DB instances. + +Automatic backups are **enabled** by default.
+Setting the backup retention period to 0 disables them, setting it to a nonzero value (re)enables them. + +> Enabling automatic backups takes the affected instances offline to have a backup created immediately.
+> It **will** cause outages. + +Automatic backups occur **daily** during the instances' backup window, configured in 30 minute periods. Should backups +require more time than allotted to the backup window, they will continue after the window ends and until they finish. + +Backups are retained for up to 35 days (_backup retention period_).
+One can recover DB instances to any point in time from the backup retention period. + +The backup window can't overlap with the weekly maintenance window for DB instance or Multi-AZ DB cluster.
+During automatic backup windows storage I/O might be suspended briefly while the backup process initializes. +Initialization typically takes up to a few seconds. One might also experience elevated latencies for a few minutes +during backups for Multi-AZ deployments.
+For MariaDB, MySQL, Oracle and PostgreSQL Multi-AZ deployments, I/O activity isn't suspended on the primary instance as +the backup is taken from the standby.
+Automated backups might occasionally be skipped if instances or clusters are running heavy workloads at the time backups +are supposed to start. + +DB instances must be in the `available` state for automated backups to occur.
+Automated backups don't occur while DB instances are in other states (i.e., `storage_full`). + +Automated backups aren't created while a DB instance or cluster is stopped.
+RDS doesn't include time spent in the stopped state when the backup retention window is calculated. This means backups +can be retained longer than the backup retention period if a DB instance has been stopped. + +Automated backups don't occur while a DB snapshot copy is running in the same AWS Region for the same database. + +### Manual backups + +Back up DB instances manually by creating DB snapshots.
+The first snapshot contains the data for the full database. Subsequent snapshots of the same database are incremental. + +One can copy both automatic and manual DB snapshots, but only share manual DB snapshots. + +Manual snapshots **never** expire and are retained indefinitely. + +One can store up to 100 manual snapshots per Region. + +### Export snapshots to S3 + +One can export DB snapshot data to [S3] buckets.
+RDS spins up an instance from the snapshot, extracts data from it and stores the data in Apache Parquet format.
+By default **all** data in the snapshots is exported, but one can specify specific sets of databases, schemas, or tables +to export. + +- The export process runs in the background and does **not** affect the performance of active DB instances. +- Multiple export tasks for the same DB snapshot cannot run simultaneously. This applies to both full and partial + exports. +- Exporting snapshots from DB instances that use magnetic storage isn't supported. +- The following characters aren't supported in table column names: + + ```plaintext + , ; { } ( ) \n \t = (space) / + ``` + + Tables containing those characters in column names are skipped during export. +- PostgreSQL _temporary_ and _unlogged_ tables are skipped during export. +- Large objects in the data, like BLOBs or CLOBs, close to or greater than 500 MB will make the export fail. +- Large rows close to or greater than 2 GB will make their table being skipped during export. +- Data exported from snapshots to S3 cannot be restored to new DB instances. +- The snapshot export tasks require a role with write-access permission to the destination S3 bucket: + + ```json + { + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": "sts:AssumeRole", + "Principal": { + "Service": "export.rds.amazonaws.com" + } + }] + } + ``` + + ```json + { + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Action": [ + "s3:PutObject*", + "s3:ListBucket", + "s3:GetObject*", + "s3:DeleteObject*", + "s3:GetBucketLocation" + ], + "Resource": [ + "arn:aws:s3:::bucket", + "arn:aws:s3:::bucket/*" + ] + }] + } + ``` + +After the export, one can analyze the data directly through +[Athena](https://docs.aws.amazon.com/athena/latest/ug/parquet-serde.html) or +[Redshift Spectrum](https://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-copy-from-columnar.html). + +
+ In the Console + +The _Export to Amazon S3_ console option appears only for snapshots that can be exported to Amazon S3.
+Snapshots might not be available for export because of the following reasons: + +- The DB engine isn't supported for S3 export. +- The DB instance version isn't supported for S3 export. +- S3 export isn't supported in the AWS Region where the snapshot was created. + +
+
+ Using the CLI + +```sh +# Start new tasks. +$ aws rds start-export-task \ + --export-task-identifier 'db-finalSnapshot-2024' \ + --source-arn 'arn:aws:rds:eu-west-1:012345678901:snapshot:db-prod-final-2024' \ + --s3-bucket-name 'backups' --s3-prefix 'rds' \ + --iam-role-arn 'arn:aws:iam::012345678901:role/CustomRdsS3Exporter' \ + --kms-key-id 'arn:aws:kms:eu-west-1:012345678901:key/abcdef01-2345-6789-abcd-ef0123456789' +{ + "ExportTaskIdentifier": "db-finalSnapshot-2024", + "IamRoleArn": "arn:aws:iam::012345678901:role/CustomRdsS3Exporter", + "KmsKeyId": "arn:aws:kms:eu-west-1:012345678901:key/abcdef01-2345-6789-abcd-ef0123456789", + "PercentProgress": 0, + "S3Bucket": "backups", + "S3Prefix": "rds", + "SnapshotTime": "2024-06-17T09:04:41.387000+00:00", + "SourceArn": "arn:aws:rds:eu-west-1:012345678901:snapshot:db-prod-final-2024", + "Status": "STARTING", + "TotalExtractedDataInGB": 0 +} + +# Get tasks' status. +$ aws rds describe-export-tasks +$ aws rds describe-export-tasks --export-task-identifier 'db-finalSnapshot-2024' +$ aws rds describe-export-tasks --query 'ExportTasks[].WarningMessage' --output 'yaml' + +# Cancel tasks. +$ aws rds cancel-export-task --export-task-identifier 'my_export' +{ + "Status": "CANCELING", + "S3Prefix": "", + "ExportTime": "2019-08-12T01:23:53.109Z", + "S3Bucket": "DOC-EXAMPLE-BUCKET", + "PercentProgress": 0, + "KmsKeyId": "arn:aws:kms:AWS_Region:123456789012:key/K7MDENG/bPxRfiCYEXAMPLEKEY", + "ExportTaskIdentifier": "my_export", + "IamRoleArn": "arn:aws:iam::123456789012:role/export-to-s3", + "TotalExtractedDataInGB": 0, + "TaskStartTime": "2019-11-13T19:46:00.173Z", + "SourceArn": "arn:aws:rds:AWS_Region:123456789012:snapshot:export-example-1" +} +``` + +
+ +## Restore + +DB instances **can** be restored from DB snapshots.
+Instances **cannot** be restored with less storage. + +When increasing allocated storage, increases must be by at least of 10%. Trying to increase the value by less than 10% +will result in an error.
+The allocated storage **cannot** be increased when restoring RDS for SQL Server DB instances. + +```sh +aws rds restore-db-instance-from-db-snapshot \ + --db-instance-identifier 'mynewdbinstance' \ + --db-snapshot-identifier 'mydbsnapshot' +``` + +## Encryption + +RDS automatically integrates with AWS KMS for key management. + +By default, RDS uses the RDS AWS managed key (`aws/rds`) for encryption.
+This key can't be managed, rotated, nor deleted by users. + +RDS will automatically put databases into a terminal state when access to the KMS key is required but the key has been +disabled or deleted, or its permissions have been somehow revoked.
+This change could be immediate or deferred depending on the use case that required access to the KMS key.
+In this terminal state, DB instances are no longer available and their databases' current state can't be recovered. To +restore DB instances, one must first re-enable access to the KMS key for RDS, and then restore the instances from their +latest available backup. + +## Further readings + +- [Working with DB instance read replicas] + +### Sources + +- [Pricing and data retention for Performance Insights] +- [Introduction to backups] +- [Restoring from a DB snapshot] +- [AWS KMS key management] + + + + + +[s3]: s3.md + + + +[aws kms key management]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.Keys.html +[introduction to backups]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html +[pricing and data retention for performance insights]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.Overview.cost.html +[restoring from a db snapshot]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RestoreFromSnapshot.html +[working with db instance read replicas]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html + + diff --git a/knowledge base/cloud computing/aws/s3.md b/knowledge base/cloud computing/aws/s3.md index 52769a4..ac4bd61 100644 --- a/knowledge base/cloud computing/aws/s3.md +++ b/knowledge base/cloud computing/aws/s3.md @@ -1,6 +1,7 @@ # Simple Storage Service 1. [TL;DR](#tldr) +1. [Storage tiers](#storage-tiers) 1. [Lifecycle configuration](#lifecycle-configuration) 1. [Further readings](#further-readings) 1. [Sources](#sources) @@ -8,7 +9,7 @@ ## TL;DR
- Common usage + Usage ```sh # List all buckets. @@ -47,11 +48,18 @@ aws s3 cp 'file.txt' 's3://my-bucket/' \ 'full=id=79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be' aws s3 cp 'mydoc.txt' 's3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mykey' -# Handling file streams. +# Handle file streams. +# Useful for piping: +# - setting the source to '-' sends data from stdin +# - setting the destination to '-' sends data to stdout aws s3 cp - 's3://my-bucket/stream.txt' aws s3 cp - 's3://my-bucket/stream.txt' --expected-size '54760833024' aws s3 cp 's3://my-bucket/stream.txt' - +# Directly print the contents of files to stdout. +aws s3 cp --quiet 's3://my-bucket/file.txt' '-' +aws s3 cp --quiet 's3://my-bucket/file.txt' '/dev/stdout' + # Remove objects. aws s3 rm 's3://my-bucket/prefix-name' --recursive --dryrun @@ -105,20 +113,35 @@ aws s3api list-objects-v2 \
+## Storage tiers + +| | Standard | Intelligent-Tiering | Express One Zone | Standard Infrequent Access | One Zone Infrequent Access | Glacier Instant Retrieval | Glacier Flexible Retrieval | Glacier Deep Archive | +| ---------------------- | ------------ | ------------------- | ------------------------- | -------------------------- | -------------------------- | ------------------------- | -------------------------- | -------------------- | +| Retrieval charge | ✗ | ✗ | ✗ | per GB retrieved | per GB retrieved | per GB retrieved | per GB retrieved | per GB retrieved | +| Latency | milliseconds | milliseconds | single-digit milliseconds | milliseconds | milliseconds | milliseconds | minutes to hours | hours | +| Minimum storage charge | ✗ | ✗ | 1 hour | 30 days | 30 days | 90 days | 90 days | 180 days | +| Availability Zones | 3+ | 3+ | 1 | 3+ | 1 | 3+ | 3+ | 3+ | + ## Lifecycle configuration > Adding, removing or changing lifecycle rules takes a while.
> Wait a couple of minutes after the operation to make sure all the bucket's properties are synced. -When one has multiple rules in an S3 Lifecycle configuration, an object can become eligible for multiple S3 Lifecycle actions. In such cases, Amazon S3 follows these general rules: +When multiple rules are applied through S3 Lifecycle configurations, objects can become eligible for multiple S3 +Lifecycle actions. In such cases: -1. Permanent deletion takes precedence over transition. -1. Transition takes precedence over creation of delete markers. -1. When an object is eligible for both a S3 Glacier Flexible Retrieval and S3 Standard-IA (or S3 One Zone-IA) transition, Amazon S3 chooses the S3 Glacier Flexible Retrieval transition. +1. Permanent deletion takes precedence over transitions. +1. Transitions takes precedence over creation of delete markers. +1. When objects are eligible for transition to both S3 Glacier Flexible Retrieval and S3 Standard-IA (or One Zone-IA), + precedence is given to S3 Glacier Flexible Retrieval transition. -Propagation delay: When you add an S3 Lifecycle configuration to a bucket, there is usually some lag before a new or updated Lifecycle configuration is fully propagated to all the Amazon S3 systems. Expect a delay of a few minutes before the configuration fully takes effect. This delay can also occur when you delete an S3 Lifecycle configuration. +When adding S3 Lifecycle configurations to buckets, there is usually some lag before a new or updated Lifecycle +configuration is fully propagated to all the S3's systems.
+Expect a delay of a few minutes before any change in configuration fully takes effect. This includes configuration +deletions. -Objects can only go down the tiers, not up. Many other constraints apply, like no transition done for objects <128KiB.
+Objects can only go down the tiers, not up.
+Other constraints apply, like no transition done for objects smaller than 128KiB.
See [General considerations for transitions][lifecycle general considerations for transitions]. Examples: [1][lifecycle configuration examples], [2][s3 lifecycle rules examples] @@ -132,6 +155,7 @@ Examples: [1][lifecycle configuration examples], [2][s3 lifecycle rules example ### Sources +- [Amazon S3 Storage Classes] - [General considerations for transitions][lifecycle general considerations for transitions] - [Lifecycle configuration examples][lifecycle configuration examples] - [CLI subcommand reference] @@ -139,7 +163,8 @@ Examples: [1][lifecycle configuration examples], [2][s3 lifecycle rules example - [How S3 Intelligent-Tiering works] @@ -151,6 +176,7 @@ Examples: [1][lifecycle configuration examples], [2][s3 lifecycle rules example [s3 lifecycle rules examples]: ../../../examples/aws/s3.lifecycle-rules +[amazon s3 storage classes]: https://aws.amazon.com/s3/storage-classes/ [cli subcommand reference]: https://docs.aws.amazon.com/cli/latest/reference/s3/ [expiring amazon s3 objects based on last accessed date to decrease costs]: https://aws.amazon.com/blogs/architecture/expiring-amazon-s3-objects-based-on-last-accessed-date-to-decrease-costs/ [find out the size of your amazon s3 buckets]: https://aws.amazon.com/blogs/storage/find-out-the-size-of-your-amazon-s3-buckets/ diff --git a/snippets/aws.fish b/snippets/aws.fish index 2c262ef..8eadab3 100644 --- a/snippets/aws.fish +++ b/snippets/aws.fish @@ -99,3 +99,16 @@ set instance_id 'i-0915612f182914822' \ aws imagebuilder list-image-recipes aws imagebuilder get-image-recipe --image-recipe-arn 'arn:aws:imagebuilder:eu-west-1:012345678901:image-recipe/my-custom-image/1.0.12' + + +aws rds start-export-task \ + --export-task-identifier 'db-finalSnapshot-2024' \ + --source-arn 'arn:aws:rds:eu-west-1:012345678901:snapshot:db-prod-final-2024' \ + --s3-bucket-name 'backups' --s3-prefix 'rds' \ + --iam-role-arn 'arn:aws:iam::012345678901:role/CustomRdsS3Exporter' \ + --kms-key-id 'arn:aws:kms:eu-west-1:012345678901:key/abcdef01-2345-6789-abcd-ef0123456789' + +# Max 5 running at any given time, RDS cannot queue +echo {1..5} | xargs -p -n '1' -I '{}' aws rds start-export-task … + +aws rds describe-export-tasks --query 'ExportTasks[].WarningMessage' --output 'json' diff --git a/snippets/pulumi/aws rds s3 exporter iam role.ts b/snippets/pulumi/aws rds s3 exporter iam role.ts index fb4e176..037743a 100644 --- a/snippets/pulumi/aws rds s3 exporter iam role.ts +++ b/snippets/pulumi/aws rds s3 exporter iam role.ts @@ -1,4 +1,5 @@ /** + * https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html * https://repost.aws/knowledge-center/rds-mysql-export-snapshot */