diff --git a/knowledge base/cloud computing/aws/README.md b/knowledge base/cloud computing/aws/README.md index ac967aa..9c20da0 100644 --- a/knowledge base/cloud computing/aws/README.md +++ b/knowledge base/cloud computing/aws/README.md @@ -198,6 +198,8 @@ From [Using service-linked roles]: > Service-linked roles appear in your AWS account and are owned by the service. An IAM administrator can view, but not > edit the permissions for service-linked roles. +Check [aws.permissions.cloud] for a community-driven source of truth for AWS identity. + ### IAM policies IAM does not expose policies' `Sid` element in the IAM API, so it can't be used to retrieve statements. @@ -269,6 +271,7 @@ Examples: - [Working with DB instance read replicas] - AWS' [CLI] - [Configuring EC2 Disk alert using Amazon CloudWatch] +- [aws.permissions.cloud] ### Sources @@ -350,6 +353,7 @@ Examples: [automating dns-challenge based letsencrypt certificates with aws route 53]: https://johnrix.medium.com/automating-dns-challenge-based-letsencrypt-certificates-with-aws-route-53-8ba799dd207b [aws config tutorial by stephane maarek]: https://www.youtube.com/watch?v=qHdFoYSrUvk [aws icons]: https://aws-icons.com/ +[aws.permissions.cloud]: https://aws.permissions.cloud/ [configuring ec2 disk alert using amazon cloudwatch]: https://medium.com/@chandinims001/configuring-ec2-disk-alert-using-amazon-cloudwatch-793807e40d72 [date & time policy conditions at aws - 1-minute iam lesson]: https://www.youtube.com/watch?v=4wpKP1HLEXg [introduction to aws iam assumerole]: https://aws.plainenglish.io/introduction-to-aws-iam-assumerole-fbef3ce8e90b diff --git a/knowledge base/cloud computing/aws/opensearch.md b/knowledge base/cloud computing/aws/opensearch.md index 2e1cb5e..5e76693 100644 --- a/knowledge base/cloud computing/aws/opensearch.md +++ b/knowledge base/cloud computing/aws/opensearch.md @@ -17,20 +17,28 @@ Amazon offering for managed OpenSearch clusters. ## Storage -_Standard_ data nodes use _hot_ storage in the form of instance stores or EBS volumes attached to each node.
-Hot storage provides the fastest possible performance for indexing and searching new data. +Clusters can be set up to use the [hot-warm architecture]. -_UltraWarm_ nodes use S3 and caching.
-Useful for indexes that are **not** actively written to, queried less frequently, or don't need the hot storage's -performance. +_Hot_ storage provides the fastest possible performance for indexing and searching **new** data. -> Warm indexes are **read-only** unless returned to hot storage.
-> This makes UltraWarm storage best-suited for immutable data such as logs. +_Data_ nodes use **hot** storage in the form of instance stores or EBS volumes attached to each node. -Warm indexes behave like any other index. +Indexes that are **not** actively written to (e.g., immutable data like logs), that are queried less frequently, or that +don't need the hot storage's performance can be moved to _warm_ storage. -_Cold_ storage uses s3 too. It is meant for data accessed only occasionally or no longer in active use.
-One **can't** read from nor write to cold indexes. When one needs it, one can selectively attach it to UltraWarm nodes. +Warm indexes are **read-only** unless returned to hot storage.
+Aside that, they behave like any other hot index. + +_UltraWarm_ nodes use **warm** storage in the form of S3 and caching. + +AWS' managed OpenSearch service offers also _Cold_ storage.
+It is meant for data accessed only occasionally or no longer in active use.
+Cold indexes are normally detached from nodes and stored in S3, meaning one **can't** read from nor write to cold +indexes by default.
+Should one need to query them, one needs to selectively attach them to UltraWarm nodes. + +Use [Index State Management][index state management in amazon opensearch service] to automate indexes migration to +lower storage states after they meet specific conditions. ### UltraWarm storage @@ -54,8 +62,7 @@ Considerations: to the amount of storage each instance type can address and the maximum number of warm nodes supported by Domains. - Amazon recommends a maximum shard size of 50 GiB. - Upon enablement, UltraWarm might not be available to use for several hours even if the domain state is _Active_. -- Use [Index State Management][index state management in amazon opensearch service] to automate indexes migration to - UltraWarm after they meet specific conditions. +- The minimum amount of UltraWarm instances allowed by AWS is 2. > Before disabling UltraWarm, one **must** either delete **all** warm indexes or migrate them back to hot storage.
> After warm storage is empty, wait five minutes before attempting to disable UltraWarm. @@ -69,6 +76,10 @@ Requirements: - OpenSearch/ElasticSearch >= v7.9. - [UltraWarm storage] enabled for the same domain. +Considerations: + +- One **can't** read from nor write to cold indexes. + ## Operations ### Migrate indexes to UltraWarm storage @@ -203,6 +214,10 @@ can manage. - Choose appropriate [instance types and sizes][supported instance types in amazon opensearch service].
Leverage the ability to select them to tailor the service offering to one's needs. + + > [OR1 instances][or1 storage for amazon opensearch service] **cannot** (currently?) be selected as master nodes.
+ > They must also be selected **at domain creation**. + - Consider using reserved instances for long-term savings. - Enable index-level compression to save storage space and reduce I/O costs. - Use Index Lifecycle Management policies to move old data in lower storage tiers. @@ -216,17 +231,18 @@ can manage. - Optimize data ingestion. - Optimize indexes' mapping and settings. - Optimize the JVM heap size. -- Summarize and compress historical data using Rollups. +- Summarize and compress historical data using [index rollups]. - Check out caches. - Reduce the number of requests using throttling and rate limiting. -- Move to single-AZ deployments. -- Leverage Spot Instances for data ingestion and processing. -- Compress source data before sending it to OpenSearch to reduce the storage footprint and data transfer costs. +- Move to Single-AZ deployments. +- Filter out and compress source data before sending it to OpenSearch to reduce the storage footprint and data transfer + costs. - Share a single OpenSearch cluster with multiple accounts to reduce the overall number of instances and resources. ## Further readings - [OpenSearch] +- [Hot-warm architecture] - [Supported instance types in Amazon OpenSearch Service] ### Sources @@ -243,6 +259,7 @@ can manage. - [Dedicated master nodes in Amazon OpenSearch Service] - [Best practices for configuring your Amazon OpenSearch Service domain] - [Operational best practices for Amazon OpenSearch Service] +- [OR1 storage for Amazon OpenSearch Service] [dedicated master nodes]: #dedicated-master-nodes +[hot-warm architecture]: ../../opensearch.md#hot-warm-architecture [opensearch]: ../../opensearch.md [s3]: s3.md @@ -267,11 +285,13 @@ can manage. [index state management in amazon opensearch service]: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html [lower your amazon opensearch service storage cost with gp3 amazon ebs volumes]: https://aws.amazon.com/blogs/big-data/lower-your-amazon-opensearch-service-storage-cost-with-gp3-amazon-ebs-volumes/ [operational best practices for amazon opensearch service]: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html +[or1 storage for amazon opensearch service]: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/or1.html [supported instance types in amazon opensearch service]: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-instance-types.html [ultrawarm storage for amazon opensearch service]: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ultrawarm.html [cost-saving strategies for aws opensearch(finops): optimize performance without breaking the bank]: https://ramchandra-vadranam.medium.com/cost-saving-strategies-for-aws-opensearch-finops-optimize-performance-without-breaking-the-bank-f87f0bb2ce37 +[index rollups]: https://opensearch.org/docs/latest/im-plugin/index-rollups/index/ [opensearch cost optimization: 12 expert tips]: https://opster.com/guides/opensearch/opensearch-capacity-planning/how-to-reduce-opensearch-costs/ [reducing amazon opensearch service costs: our journey to over 60% savings]: https://medium.com/kreuzwerker-gmbh/how-we-accelerate-financial-and-operational-efficiency-with-amazon-opensearch-6b86b41d50a0 [right-size amazon opensearch instances to cut costs by 50% or more]: https://cloudfix.com/blog/right-size-amazon-opensearch-instances-cut-costs/ diff --git a/knowledge base/opensearch.md b/knowledge base/opensearch.md index e589756..da43829 100644 --- a/knowledge base/opensearch.md +++ b/knowledge base/opensearch.md @@ -18,6 +18,7 @@ Use cases: application search, log analytics, data observability, data ingestion 1. [Tuning](#tuning) 1. [The split brain problem](#the-split-brain-problem) 1. [APIs](#apis) +1. [Hot-warm architecture](#hot-warm-architecture) 1. [Further readings](#further-readings) 1. [Sources](#sources) @@ -152,6 +153,9 @@ Enormous documents should still be indexed individually. When indexing documents, the document's `_id` must be 512 bytes or less in size. +_Static_ index settings can only be updated on **closed** indexes.
+_Dynamic_ index settings can be updated at any time through the [APIs]. + ## Requirements | Port number | Component | @@ -193,13 +197,14 @@ Use docker compose. ## Tuning - Disable swapping.
- Swapping can dramatically decrease performance and stability. + If kept enabled, it can dramatically **decrease** performance and stability. - Avoid using network file systems for node storage in a production workflow.
Using those can cause performance issues due to network conditions (i.e.: latency, limited throughput) or read/write speeds. - Use solid-state drives (SSDs) on the hosts for node storage where possible. - Set the size of the Java heap.
- Recommend half of system RAM. + Recommended to use **half** of the system's RAM. +- Set up a [hot-warm architecture]. ## The split brain problem @@ -207,7 +212,40 @@ TODO ## APIs -TODO +FIXME: expand + +- Close indexes.
+ Disables read and write operations. + + ```plaintext + POST /prometheus-logs-20231205/_close + ``` + +- (Re)Open closed indexes.
+ Enables read and write operations. + + ```plaintext + POST /prometheus-logs-20231205/_open + ``` + +- Update indexes' settings.
+ _Static_ settings can only be updated on **closed** indexes. + + ```plaintext + PUT /prometheus-logs-20231205/_settings + { + "index": { + "codec": "zstd_no_dict", + "codec.compression_level": 3, + "refresh_interval": "2s" + } + } + ``` + +## Hot-warm architecture + +Refer +[Set up a hot-warm architecture](https://opensearch.org/docs/latest/tuning-your-cluster/#advanced-step-7-set-up-a-hot-warm-architecture). ## Further readings @@ -218,6 +256,7 @@ TODO - [Okapi BM25] - [`fsync`][fsync] - [AWS' managed OpenSearch] offering +- [Setting up Hot-Warm architecture for ISM in OpenSearch] ### Sources @@ -227,6 +266,7 @@ TODO - [Avoiding the Elasticsearch split brain problem, and how to recover] - [Index templates in OpenSearch - how to use composable templates] - [Index management] +- [Index settings] +[apis]: #apis +[hot-warm architecture]: #hot-warm-architecture [refresh operations]: #refresh-operations [translog]: #translog @@ -246,6 +288,7 @@ TODO [documentation]: https://opensearch.org/docs/latest/ [github]: https://github.com/opensearch-project [index management]: https://opensearch.org/docs/latest/dashboards/im-dashboards/index-management/ +[index settings]: https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/index-settings/ [website]: https://opensearch.org/ [what is opensearch?]: https://aws.amazon.com/what-is/opensearch/ @@ -256,3 +299,4 @@ TODO [index templates in opensearch - how to use composable templates]: https://opster.com/guides/opensearch/opensearch-data-architecture/index-templating-in-opensearch-how-to-use-composable-templates/ [lucene]: https://lucene.apache.org/ [okapi bm25]: https://en.wikipedia.org/wiki/Okapi_BM25 +[setting up hot-warm architecture for ism in opensearch]: https://opster.com/guides/opensearch/opensearch-data-architecture/setting-up-hot-warm-architecture-for-ism/