brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
72a4db6e4514af4b27e9c28e66443d812a95673e
oam/knowledge base/cloud computing/aws/README.md
Michele Cereda 72a4db6e45 chore(kb/aws): add cloudfront notes
2025-02-21 21:12:26 +03:00

34 KiB
Raw Blame History

Amazon Web Services

  1. TL;DR
  2. Networking
    1. Elastic IP addresses
  3. Services
    1. Billing and Cost Management
    2. CloudFront
    3. CloudWatch
    4. Config
    5. Detective
    6. GuardDuty
    7. EventBridge
    8. ImageBuilder
    9. Inspector
    10. KMS
    11. Security Hub
  4. Resource constraints
  5. Access control
  6. Costs
  7. Savings plans
  8. Resource tagging
  9. API
    1. Python
  10. Further readings
    1. Sources

TL;DR

Regions are physical world locations where multiple Availability Zones exist.
They are physically isolated and independent from one another.
Regions come at no charge.

Availability Zones are sets of one or more data centers, each with their own resources, housed in separate facilities.

Resources created in one Region do not exist in any other Region, unless explicitly using replication features offered by AWS services.
Some services like IAM do not have Regional resources.

Recommended using regional STS endpoints instead of the global one to reduce latency.
Session tokens from regional STS endpoints are valid in all AWS Regions. However, tokens from the global endpoint are only valid in enabled Regions.

Session tokens valid in all Regions are larger. If storing session tokens, these might affect one's systems.

Regions introduced before 2019-03-20 are enabled by default. Newer regions are now disabled by default.
Regions enabled by default cannot be enabled or disabled.

Disabling Regions disables IAM access to resources in those Region. It will not delete resources in the disabled region, and they will continue to be charged at the standard rate.

Disabling a Region can takes a few minutes to several hours to take effect. Services and Console will be visible until the region is completely disabled.

Enabling Regions takes a few minutes to several hours. They cannot be used until the preparation process is complete.

Networking

VPCs define isolated virtual networking environments.
AWS accounts include one default VPC for each AWS Region. These allow for immediate launch and connection to EC2 instances.

Subnets are ranges of IP addresses in VPCs.
Each subnet resides in a single Availability Zone.
Public subnets have a direct route to an Internet gateway. Resources in public subnets can access the public Internet.
Private subnets do not have a direct route to an Internet gateway. Resources in private subnets require a NAT device to access the public internet.

Gateways connect VPCs to other networks.
Internet gateways connect VPCs to the Internet.
NAT gateways allow resources in private subnets to connect to the Internet, other VPCs, or on-premises networks. They can communicate with services outside the VPC, but cannot receive unsolicited connection requests.
VPC endpoints connect VPCs to AWS services privately, without the need of Internet gateways or NAT devices.

Elastic IP addresses

Refer Elastic IP addresses.

Static, public IPv4 addresses allocated to one's AWS account until one releases it.
One can can rapidly remapping addresses to other instances in one's account and use them as targets in DNS records.

Services

Service Description
Billing and Cost Management FIXME
CloudWatch Observability (logging, monitoring, alerting)
CloudFront Content delivery
Config Compliance
Detective FIXME
EC2 Managed virtual machines
ECR Container registry
ECS Run containers as a service
EFS Serverless file storage
EKS Managed Kubernetes clusters
EventBridge FIXME
GuardDuty Threat detection
IAM Access control
ImageBuilder Build custom AMIs
Inspector FIXME
KMS Key management
OpenSearch ELK, logging
RDS Databases
Route53 DNS
S3 Storage
Sagemaker Machine learning
Security Hub Aggregator for security findings
SNS Pub/sub message delivery
SQS Queues

Service icons are publicly available for diagrams and such. Public service IP address ranges are available in JSON form at https://ip-ranges.amazonaws.com/ip-ranges.json.

Billing and Cost Management

Costs can be grouped by Tags applied on resources.
Tags to use for this kind of grouping need to be activated in the Cost allocation tags section.
New tags might take 24 or 48 hours to appear there.

CloudFront

Web service speeding up distribution of static and dynamic web content such as .html, .css, .js, and image files.

Delivers content through edge locations.
When users request content served with CloudFront, the requests are routed to the edge location that provides the lowest latency in order to deliver with the best possible performance.

If the content is already in the edge location with the lowest latency, CloudFront delivers it immediately.
If the content is not in that edge location, CloudFront retrieves it from the origin defined for it.

Origins can be S3 buckets, MediaPackage channels, or HTTP servers.

CloudWatch

Observability service. with functions for logging, monitoring and alerting.

Metrics are whatever needs to be monitored (e.g. CPU usage).
Data points are the values of a metric over time.
Namespaces are containers for metrics.

Metrics only exist in the region in which they are created.

Many AWS services offer basic monitoring by publishing a default set of metrics to CloudWatch with no charge.
This feature is automatically enabled by default when one starts using one of these services.

API calls for CloudWatch are paid.

It's best practice to distribute the ListMetrics call to avoid throttling.
The default limit for ListMetrics is 25 transactions per second.

The CloudWatch console offers some default good queries.

Queries of interest
What Section Tab How to visualize
Top 10 log groups by written bytes All Metrics Graphed metrics Add Query > Logs > Top 10 log groups by written bytes
Get a dashboard of how much data a small set of log groups ingested in the last 30 days

This graph works only with the Absolute time period option.
Should you choose Relative, the graph returns incorrect data.

  1. CloudWatch console > All metrics (navigation pane on the left).
  2. Choose Logs, Log group metrics.
  3. Select the individual IncomingBytes metrics of each log group of interest.
  4. Choose the Graphed metrics tab.
  5. For each metric:
    • Change Statistic to Sum.
    • Change Period to 30 Days.
  6. Choose the Graph options tab.
  7. Choose the Number option group.
  8. At the top right of the graph, choose Custom as the time range.
  9. Choose Absolute.
  10. Select the last 30 days as start and end date.
Get a dashboard of how much data all log groups ingested in the last 30 days

This graph works only with the Absolute time period option.
Should you choose Relative, the graph returns incorrect data.

  1. CloudWatch console > All metrics (navigation pane on the left).

  2. Choose the Graphed metrics tab.

  3. From the Add math dropdown list, choose Start with an empty expression.

  4. Paste this as math expression:

    SORT(REMOVE_EMPTY(SEARCH('{AWS/Logs,LogGroupName} MetricName="IncomingBytes"', 'Sum', 2592000)),SUM, DESC)

  5. At the top right of the graph, choose Custom as the time range.

  6. Choose Absolute.

  7. Select the last 30 days as start and end date.

Config

Compliance service for assessing and auditing AWS resources.

Provides an inventory of resources.
Records and monitors resource configurations and their changes.
The data is stored in a bucket (default name config-bucket-{aws-account-number})
Changes can be streamed to 1 SNS topic for notification purposes.
Uses rules to evaluate whether the resources configurations comply.
Rule evaluation is done once every time a configuration changes, or periodically.
Resources are marked with the evaluation result (compliant, non-compliant).

Custom rules can be used to evaluate for uncommon requirements.
Custom rules leverage lambda functions.

Allows for automatic remediation for non-compliant resources by leveraging Systems Manager Automation documents.

Conformance packs are set of rules bundled together as a deployable single entity.
Defined as YAML templates.
Immutable: users cannot make changes without updating the whole rule package.
Sample templates for compliance standards and benchmarks are available.

Detective

Uses ML and graphs to try and identify the root cause of security issues.
Creates visualizations with details and context by leveraging events from VPC Flow Logs, CloudTrail and GuardDuty.

GuardDuty

Threat detection service.

It continuously monitors accounts and workloads for malicious activity and delivers security findings for visibility and remediation.
Done by pulling streams of data from CloudTrail, VPC Flow Logs or EKS.

Member accounts can administer GuardDuty by delegation if given the permissions to do so.

Findings are potential security issues for malicious events.
Those are also sent to EventBridge for notification (leveraging SNS).
Each is assigned a severity value (0.1 to 8+).

Trusted IP List is a whitelist of public IPs that will be ignored by the rules.
Threat IP List is a blacklist of public IPs and CIDRs that will be used by the rules.

EventBridge

TODO

ImageBuilder

Also refer Image baking in AWS using Packer and Image builder.

Inspector

TODO

KMS

Key material is the cryptographic secret of Keys that is used in encryption operations.

Enabling automatic key rotation for a KMS key makes the service generate new cryptographic material for the key every year by default.
Specify a custom rotation period to customize that time frame.

Perform on-demand rotation should you need to immediately initiate key material rotation.
This works regardless of whether the automatic key rotation is enabled or not. On-demand rotations do not change existing automatic rotation schedules.

KMS saves all previous versions of the cryptographic material in perpetuity to allow decryption of any data encrypted with keys.
Rotated key material is not deleted until the key itself is deleted.

Track the rotation of key material CloudWatch, CloudTrail, and the KMS console.
Alternatively, use the GetKeyRotationStatus operation to verify whether automatic rotation is enabled for a key and identify any in progress on-demand rotations. Use the ListKeyRotations operation to view the details of completed rotations.

When using a rotated KMS key to encrypt data, KMS uses the current key material.
When using the same rotated KMS key to decrypt ciphertext, KMS uses the version of the key material that was used for encryption.
One cannot select a particular version of key materials for decrypt operations. This automation allows to safely use rotated KMS keys in applications and AWS services without code changes.

Automatic key rotation has no effect on the data that KMS keys protect: it does not rotate the data generated by rotated keys, re-encrypts any data protected by the keys, nor it will mitigate the effect of compromised data keys.

KMS supports automatic and on-demand key rotation only for symmetric encryption keys with key material that KMS itself creates.
Automatic rotation is optional for customer managed KMS keys. KMS rotates the key material for AWS managed keys on an yearly basis. Rotation of AWS owned KMS keys is managed by the AWS service that owns the key.

Key rotation only changes the key material, not the key's properties.
The key is considered the same logical resource, regardless of whether or how many times its key material changes.

Creating a new key and using it in place of the original one has the same effect as rotating the key material in an existing key.
This is considered a manual key rotation and is a good choice to rotate keys that are not eligible for automatic key rotation.

AWS charges a monthly fee for the first and second rotation of key material maintained for each key.
This price increase is capped at the second rotation. Any subsequent rotations will not be billed.

Each key counts as one when calculating key resource quotas, regardless of the number of rotated key material versions.

Security Hub

Aggregator of findings for security auditing.

Uses Config to check resources' configuration by leveraging compliancy rules.

Security standards are offered as ret of rules for Config.

Data can be aggregated from different regions.
If the integration is enabled, findings from AWS services (GuardDuty) are used too within 5 minutes on average, while ones from 3rd parties can take longer.

Data can be imported from or exported to 3rd parties if the integration is enabled.
Kinda acts as a middle layer for AWS accounts.

Findings are consumed in AWS Security Finding Format (ASFF).
Those are automatically updated and deleted. Findings after 90 days are automatically deleted even if not resolved.

Can use custom insights.

Custom actions can be sent to EventBridge for automation.

Member accounts can administer Security Hub by delegation if given the permissions to do so.

Resource constraints

Data type Component Summary Description Type Length Pattern Required
Statement ID Value Optional identifier for a policy statement The element supports only ASCII uppercase letters (A-Z), lowercase letters (a-z), and numbers (0-9). String FIXME [A-Za-z0-9] No
Tag Key Required name of the tag The string value can be Unicode characters and cannot be prefixed with "aws:".
The string can contain only the set of Unicode letters, digits, white-space, _,' ., /, =, +, -, :, @ (Java regex: ^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$)		 String 1 to 128 ^([\p{L}\p{Z}\p{N}_.:/=+\-@]*)$ Yes
Tag Value The optional value of the tag The string value can be Unicode characters. The string can contain only the set of Unicode letters, digits, white-space, _, ., /, =, +, -, :, @ (Java regex: ^([\\p{L}\\p{Z}\\p{N}_.:/=+\\-]*)$", [\p{L}\p{Z}\p{N}_.:\/=+\-@]* on AWS) String 0 to 256 ^([\p{L}\p{Z}\p{N}_.:/=+\-@]*)$ Yes

Access control

Refer IAM.

Costs

One pays for data transfer between instances and services in the same region but different availability zone.
See Understanding data transfer charges.

One pays for sending logs to CloudWatch.
Refer Which log group is causing a sudden increase in my CloudWatch Logs bill? to get an idea of what changed in some time frame.

Savings plans

Refer Savings Plans user guide.

Pricing models offering lower prices compared to On-Demand prices. They require specific usage commitments ($/hour) for 1-year or 3-years terms.

Dedicated Instances, Spot Instances and Reserved Instances are not discounted by Savings Plans.

Savings Plan Included resources Up to
Compute EC2 instances regardless of family, size, AZ, region, OS or tenancy
Lambda
Fargate		 66%
EC2 Instance Individual EC2 instance families in a specific region (e.g. M5 usage in N. Virginia) regardless of AZ, size, OS or tenancy 72%
Amazon SageMaker Eligible SageMaker ML instances, including SageMaker Studio Notebook, SageMaker On-Demand Notebook, SageMaker Processing, SageMaker Data Wrangler, SageMaker Training, SageMaker Real-Time Inference, and SageMaker Batch Transform regardless of instance family, size, or region 64%

Both Compute and EC2 Instance plan types apply to EC2 instances that are a part of Amazon EMR, Amazon EKS, and Amazon ECS clusters. They do not apply to RDS instances.
Charges for the EKS service itself will not be covered by Savings Plans, but the underlying EC2 instances will be.

Savings Plans are available in the following payment options:

  • No Upfront: no upfront payments, commitment charged purely on a monthly basis.
  • Partial Upfront: lower prices, at least half of one's commitment upfront, remainder charged on a monthly basis.
  • All Upfront: lowest prices, entire commitment charged in one payment at the start.

Savings Plans can be purchased in any account within an AWS Organization/Consolidated Billing family.
By default, the benefits of the Plans are applicable to usage across all accounts. One can choose to restrict the benefit of the Plans to only the account that purchased them.

One account can have multiple Savings Plans active at the same time.

Plans cannot be cancelled during their term.
Plans can be returned only if:

  • They consist in an hourly commitment of $100 or less.
  • They have been purchased in the past 7 days and in the same calendar month.

Once returned, one will receive a 100% refund for any upfront charges for the Savings Plan.
Refunds will be reflected in one's bill within 24 hours of return.

Any usage covered by the plan will be charged at On-Demand rates, or get covered by a different Savings Plans if applicable.

Plans do not provide capacity reservations.
One can however reserve capacity with On Demand Capacity Reservations and pay lower prices on them with Savings Plans.

EC2 Instance Savings Plans are applied before Compute Savings Plans.

Savings Plans are applied to the highest savings percentage first. If there are multiple usages with equal savings percentages, Savings Plans are applied to the first usage with the lowest Savings Plans rate.
Savings Plans continue to apply until there are no more remaining usages, or one's commitment is exhausted. Any remaining usage is then charged at the On-Demand rates.

Resource tagging

Suggested:

Tag Purpose Example Notes
Name AWS UI GitlabRunner
Owner SecurityLead, SecOps, Workload-1-Development-team
BusinessUnitId Finance, Retail, API-1, DevOps
Environment Sandbox, Dev, PreProd, QA, Prod, Testing
CostCenter FIN123, Retail-123, Sales-248, HR-333
FinancialOwner HR, SecurityLead, DevOps-3, Workload-1-Development-team
ComplianceRequirement NIST, HIPAA, GDPR

Create tag policies to enforce values, and to prevent the creation of non-compliant resources.

API

Refer Tools to Build on AWS.

Python

Refer Boto3 documentation.
Also see Difference in Boto3 between resource, client, and session?.

Clients and Resources are different abstractions for service requests within the Boto3 SDK.
When making API calls to an AWS service with Boto3, one does so via a Client or a Resource.

Sessions are fundamental to both Clients and Resources and how both get access to AWS credentials.

Client

Provides low-level access to AWS services by exposing the botocore client to the developer.

Typically maps 1:1 with the related service's API and supports all operations for the called service.
Exposes Python-fashioned method names (e.g. ListBuckets API => list_buckets method).

Typically yields primitive, non-marshalled AWS data.
E.g. DynamoDB attributes are dictionaries representing primitive DynamoDB values.

Limited to listing at most 1000 objects, requiring the developer to deal with result pagination in code.
Use a paginator or implement one's own loop.

Example 
import boto3

client = boto3.client('s3')
response = client.list_objects_v2(Bucket='mybucket')
for content in response['Contents']:
    obj_dict = client.get_object(Bucket='mybucket', Key=content['Key'])
    print(content['Key'], obj_dict['LastModified'])
Resource

Refer Boto3 resources.

Provides high-level, object-oriented code.

Does not provide 100% API coverage of AWS services.

Uses identifiers and attributes, has actions (operations on resources), and exposes sub-resources and collections of AWS resources.

Typically yields marshalled data, not primitive AWS data.
E.g. DynamoDB attributes are native Python values representing primitive DynamoDB values.

Takes care of result pagination.
The resulting collections of sub-resources are lazily-loaded.

Resources are not thread safe and should not be shared across threads or processes.
Create a new Resource for each thread or process instead.

Since January 2023 the AWS Python SDK team stopped adding new features to the resources interface in Boto3.
Newer service features can be accessed through the Client interface.
Refer More info about resource deprecation? for more information.

Example 
import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('mybucket')
for obj in bucket.objects.all():
    print(obj.key, obj.last_modified)
Session

Refer Boto3 sessions.

Stores configuration information (primarily credentials and selected AWS Region).
Initiates the connectivity to AWS services.

Leveraged by service Clients and Resources.
boto3 creates a default session automatically when needed, using the default credential profile.
The default credentials profile uses the ~/.aws/credentials file if found, or tries assuming the role of the executing machine if not.

Further readings

Sources