oam/knowledge base/cloud computing/aws/s3.md

Simple Storage Service

1. TL;DR
2. Storage tiers
3. Lifecycle configuration
4. Further readings
   1. Sources

TL;DR

Usage

# List all buckets.
aws s3 ls
aws s3api list-buckets --output 'json' --query 'Buckets[].Name'
aws s3api list-buckets --output 'yaml-stream' | yq -r '.[].Buckets[].Name' -

# List prefixes and objects in buckets.
# Adding the trailing '/' or '--recurse' lists the content of prefixes.
aws s3 ls 's3://my-bucket'
aws s3 ls --recursive 's3://my-bucket/prefix/'
aws s3 ls 's3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/'

# Find the size of buckets or objects.
# It will list all the contents *and* give a total size at the end.
aws s3 ls --human-readable --recursive --summarize 's3://my-bucket'
aws s3 ls … 's3://my-bucket/prefix/'

# Create buckets.
aws s3 mb 's3://my-bucket'

# Copy files to or from buckets.
aws s3 cp 'test.txt' 's3://my-bucket/test4.txt'
aws s3 cp 'test.txt' 's3://my-bucket/test2.txt' --expires '2024-10-01T20:30:00Z'
aws s3 cp 's3://my-bucket/test.txt' 'test2.txt'
aws s3 cp 's3://my-bucket/test.txt' 's3://my-bucket/test5.txt'
aws s3 cp 's3://my-bucket/test.txt' 's3://my-other-bucket/'
aws s3 cp 's3://my-bucket' '.' --recursive
aws s3 cp 'myDir' 's3://my-bucket/' --recursive --exclude "*.jpg"
aws s3 cp 's3://my-bucket/logs/' 's3://my-bucket2/logs/' --recursive \
  --exclude "*" --include "*.log"
aws s3 cp 's3://my-bucket/test.txt' 's3://my-bucket/test2.txt' \
    --acl 'public-read-write'
aws s3 cp 'file.txt' 's3://my-bucket/' \
  --grants read=uri='http://acs.amazonaws.com/groups/global/AllUsers' \
    'full=id=79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be'
aws s3 cp 'mydoc.txt' 's3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mykey'

# Handle file streams.
# Useful for piping:
# - setting the source to '-' sends data from stdin
# - setting the destination to '-' sends data to stdout
aws s3 cp - 's3://my-bucket/stream.txt'
aws s3 cp - 's3://my-bucket/stream.txt' --expected-size '54760833024'
aws s3 cp 's3://my-bucket/stream.txt' -

# Directly print the contents of files to stdout.
aws s3 cp --quiet 's3://my-bucket/file.txt' '-'
aws s3 cp --quiet 's3://my-bucket/file.txt' '/dev/stdout'

# Remove objects.
aws s3 rm 's3://my-bucket/prefix-name' --recursive --dryrun

# Sync buckets.
aws s3 sync '.' 's3://my-bucket'
aws s3 sync 's3://my-bucket' '.' --delete
aws s3 sync 's3://my-bucket' 's3://my-other-bucket' --exclude "*.jpg"
aws s3 sync 's3://my-us-west-2-bucket' 's3://my-eu-east-1-bucket' \
  --source-region 'us-west-2' --region 'eu-east-1'
aws s3 sync '.' 's3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/'

# Delete buckets.
aws s3 rb 's3://my-bucket'
aws s3 rb 's3://my-bucket' --force

# Check permissions.
aws s3api get-bucket-acl --bucket 'my-bucket'

Lifecycle configurations

# Manage lifecycle configurations.
# Operations on lifecycle rules take a while.
aws s3api get-bucket-lifecycle-configuration --bucket 'bucketName'
aws s3api put-bucket-lifecycle-configuration --bucket 'bucketName' \
  --lifecycle-configuration 'file://lifecycle.definition.json'
aws s3api delete-bucket-lifecycle-configuration --bucket 'bucketName'

Real life use cases

# Get objects with their storage class.
aws s3api list-objects --bucket 'my-bucket' \
  --query 'Contents[].{Key: Key, StorageClass: StorageClass}'

# Show tags on objects.
aws s3api list-objects-v2 \
  --bucket 'my-bucket' --prefix 'someObjectsInHereAreTagged' \
  --query 'Contents[*].Key' --output text \
| xargs -n 1 \
    aws s3api get-object-tagging --bucket 'my-bucket' --query 'TagSet[*]' --key

Storage tiers

	Standard	Intelligent-Tiering	Express One Zone	Standard Infrequent Access	One Zone Infrequent Access	Glacier Instant Retrieval	Glacier Flexible Retrieval	Glacier Deep Archive
Retrieval charge	✗	✗	✗	per GB retrieved	per GB retrieved	per GB retrieved	per GB retrieved	per GB retrieved
Latency	milliseconds	milliseconds	single-digit milliseconds	milliseconds	milliseconds	milliseconds	minutes to hours	hours
Minimum storage charge	✗	✗	1 hour	30 days	30 days	90 days	90 days	180 days
Availability Zones	3+	3+	1	3+	1	3+	3+	3+

Lifecycle configuration

Adding, removing or changing lifecycle rules takes a while.
Wait a couple of minutes after the operation to make sure all the bucket's properties are synced.

When multiple rules are applied through S3 Lifecycle configurations, objects can become eligible for multiple S3 Lifecycle actions. In such cases:

1. Permanent deletion takes precedence over transitions.
2. Transitions takes precedence over creation of delete markers.
3. When objects are eligible for transition to both S3 Glacier Flexible Retrieval and S3 Standard-IA (or One Zone-IA), precedence is given to S3 Glacier Flexible Retrieval transition.

When adding S3 Lifecycle configurations to buckets, there is usually some lag before a new or updated Lifecycle configuration is fully propagated to all the S3's systems.
Expect a delay of a few minutes before any change in configuration fully takes effect. This includes configuration deletions.

Objects can only go down the tiers, not up.
Other constraints apply, like no transition done for objects smaller than 128KiB.
See General considerations for transitions.

Examples: 1, 2

Further readings

• Amazon Web Services
• Configure notification for lifecycle rules
• AWS' CLI
• Expiring Amazon S3 objects based on last accessed date to decrease costs

Sources

• Amazon S3 Storage Classes
• General considerations for transitions
• Lifecycle configuration examples
• CLI subcommand reference
• Find out the size of your Amazon S3 buckets
• How S3 Intelligent-Tiering works