mirror of
https://gitea.com/mcereda/oam.git
synced 2026-02-09 05:44:23 +00:00
158 lines
6.3 KiB
Markdown
158 lines
6.3 KiB
Markdown
# Simple Storage Service
|
|
|
|
1. [TL;DR](#tldr)
|
|
1. [Lifecycle configuration](#lifecycle-configuration)
|
|
1. [Further readings](#further-readings)
|
|
1. [Sources](#sources)
|
|
|
|
## TL;DR
|
|
|
|
<details>
|
|
<summary>Common usage</summary>
|
|
|
|
```sh
|
|
# List all buckets.
|
|
aws s3 ls
|
|
aws s3api list-buckets --output 'json' --query 'Buckets[].Name'
|
|
aws s3api list-buckets --output 'yaml-stream' | yq -r '.[].Buckets[].Name' -
|
|
|
|
# List prefixes and objects in buckets.
|
|
# Adding the trailing '/' or '--recurse' lists the content of prefixes.
|
|
aws s3 ls 's3://my-bucket'
|
|
aws s3 ls --recursive 's3://my-bucket/prefix/'
|
|
aws s3 ls 's3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/'
|
|
|
|
# Find the size of buckets or objects.
|
|
# It will list all the contents *and* give a total size at the end.
|
|
aws s3 ls --human-readable --recursive --summarize 's3://my-bucket'
|
|
aws s3 ls … 's3://my-bucket/prefix/'
|
|
|
|
# Create buckets.
|
|
aws s3 mb 's3://my-bucket'
|
|
|
|
# Copy files to or from buckets.
|
|
aws s3 cp 'test.txt' 's3://my-bucket/test4.txt'
|
|
aws s3 cp 'test.txt' 's3://my-bucket/test2.txt' --expires '2024-10-01T20:30:00Z'
|
|
aws s3 cp 's3://my-bucket/test.txt' 'test2.txt'
|
|
aws s3 cp 's3://my-bucket/test.txt' 's3://my-bucket/test5.txt'
|
|
aws s3 cp 's3://my-bucket/test.txt' 's3://my-other-bucket/'
|
|
aws s3 cp 's3://my-bucket' '.' --recursive
|
|
aws s3 cp 'myDir' 's3://my-bucket/' --recursive --exclude "*.jpg"
|
|
aws s3 cp 's3://my-bucket/logs/' 's3://my-bucket2/logs/' --recursive \
|
|
--exclude "*" --include "*.log"
|
|
aws s3 cp 's3://my-bucket/test.txt' 's3://my-bucket/test2.txt' \
|
|
--acl 'public-read-write'
|
|
aws s3 cp 'file.txt' 's3://my-bucket/' \
|
|
--grants read=uri='http://acs.amazonaws.com/groups/global/AllUsers' \
|
|
'full=id=79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be'
|
|
aws s3 cp 'mydoc.txt' 's3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/mykey'
|
|
|
|
# Handling file streams.
|
|
aws s3 cp - 's3://my-bucket/stream.txt'
|
|
aws s3 cp - 's3://my-bucket/stream.txt' --expected-size '54760833024'
|
|
aws s3 cp 's3://my-bucket/stream.txt' -
|
|
|
|
# Sync buckets.
|
|
aws s3 sync '.' 's3://my-bucket'
|
|
aws s3 sync 's3://my-bucket' '.' --delete
|
|
aws s3 sync 's3://my-bucket' 's3://my-other-bucket' --exclude "*.jpg"
|
|
aws s3 sync 's3://my-us-west-2-bucket' 's3://my-eu-east-1-bucket' \
|
|
--source-region 'us-west-2' --region 'eu-east-1'
|
|
aws s3 sync '.' 's3://arn:aws:s3:us-west-2:123456789012:accesspoint/myaccesspoint/'
|
|
|
|
# Delete buckets.
|
|
aws s3 rb 's3://my-bucket'
|
|
aws s3 rb 's3://my-bucket' --force
|
|
|
|
# Check permissions.
|
|
aws s3api get-bucket-acl --bucket 'my-bucket'
|
|
```
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary>Lifecycle configurations</summary>
|
|
|
|
```sh
|
|
# Manage lifecycle configurations.
|
|
# Operations on lifecycle rules take a while.
|
|
aws s3api get-bucket-lifecycle-configuration --bucket 'bucketName'
|
|
aws s3api put-bucket-lifecycle-configuration --bucket 'bucketName' \
|
|
--lifecycle-configuration 'file://lifecycle.definition.json'
|
|
aws s3api delete-bucket-lifecycle-configuration --bucket 'bucketName'
|
|
```
|
|
|
|
</details>
|
|
|
|
<details>
|
|
<summary>Real life use cases</summary>
|
|
|
|
```sh
|
|
# Get objects with their storage class.
|
|
aws s3api list-objects --bucket 'my-bucket' \
|
|
--query 'Contents[].{Key: Key, StorageClass: StorageClass}'
|
|
|
|
# Show tags on objects.
|
|
aws s3api list-objects-v2 \
|
|
--bucket 'my-bucket' --prefix 'someObjectsInHereAreTagged' \
|
|
--query 'Contents[*].Key' --output text \
|
|
| xargs -n 1 \
|
|
aws s3api get-object-tagging --bucket 'my-bucket' --query 'TagSet[*]' --key
|
|
```
|
|
|
|
</details>
|
|
|
|
## Lifecycle configuration
|
|
|
|
> Adding, removing or changing lifecycle rules takes a while.<br/>
|
|
> Wait a couple of minutes after the operation to make sure all the bucket's properties are synced.
|
|
|
|
When one has multiple rules in an S3 Lifecycle configuration, an object can become eligible for multiple S3 Lifecycle actions. In such cases, Amazon S3 follows these general rules:
|
|
|
|
1. Permanent deletion takes precedence over transition.
|
|
1. Transition takes precedence over creation of delete markers.
|
|
1. When an object is eligible for both a S3 Glacier Flexible Retrieval and S3 Standard-IA (or S3 One Zone-IA) transition, Amazon S3 chooses the S3 Glacier Flexible Retrieval transition.
|
|
|
|
Propagation delay: When you add an S3 Lifecycle configuration to a bucket, there is usually some lag before a new or updated Lifecycle configuration is fully propagated to all the Amazon S3 systems. Expect a delay of a few minutes before the configuration fully takes effect. This delay can also occur when you delete an S3 Lifecycle configuration.
|
|
|
|
Objects can only go down the tiers, not up. Many other constraints apply, like no transition done for objects <128KiB.<br/>
|
|
See [General considerations for transitions][lifecycle general considerations for transitions].
|
|
|
|
Examples: [1][lifecycle configuration examples], [2][s3 lifecycle rules examples]
|
|
|
|
## Further readings
|
|
|
|
- [Configure notification for lifecycle rules][lifecycle configure notification]
|
|
- AWS' [CLI]
|
|
- [Expiring Amazon S3 objects based on last accessed date to decrease costs]
|
|
|
|
### Sources
|
|
|
|
- [General considerations for transitions][lifecycle general considerations for transitions]
|
|
- [Lifecycle configuration examples][lifecycle configuration examples]
|
|
- [CLI subcommand reference]
|
|
- [Find out the size of your Amazon S3 buckets]
|
|
- [How S3 Intelligent-Tiering works]
|
|
|
|
<!--
|
|
References
|
|
-->
|
|
|
|
<!-- In-article sections -->
|
|
<!-- Knowledge base -->
|
|
[cli]: cli.md
|
|
|
|
<!-- Files -->
|
|
[s3 lifecycle rules examples]: ../../../examples/aws/s3.lifecycle-rules
|
|
|
|
<!-- Upstream -->
|
|
[cli subcommand reference]: https://docs.aws.amazon.com/cli/latest/reference/s3/
|
|
[expiring amazon s3 objects based on last accessed date to decrease costs]: https://aws.amazon.com/blogs/architecture/expiring-amazon-s3-objects-based-on-last-accessed-date-to-decrease-costs/
|
|
[find out the size of your amazon s3 buckets]: https://aws.amazon.com/blogs/storage/find-out-the-size-of-your-amazon-s3-buckets/
|
|
[how s3 intelligent-tiering works]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/intelligent-tiering-overview.html
|
|
[lifecycle configuration examples]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html
|
|
[lifecycle configure notification]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configure-notification.html
|
|
[lifecycle general considerations for transitions]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html
|
|
|
|
<!-- Others -->
|