diff --git a/.vscode/settings.json b/.vscode/settings.json index 81d66d5..73eb419 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -189,6 +189,7 @@ "helmfile", "hoppscotch", "imager", + "ingester", "istio", "istioctl", "jdupes", diff --git a/knowledge base/cloud computing/aws/ecs.md b/knowledge base/cloud computing/aws/ecs.md index f30d26a..25a3d09 100644 --- a/knowledge base/cloud computing/aws/ecs.md +++ b/knowledge base/cloud computing/aws/ecs.md @@ -249,7 +249,7 @@ Such tasks support CPU values between 0.25 and 10 vCPUs. these fields are option Task definitions specifying `FARGATE` as value for the `requiresCompatibilities` attribute, **even if they also specify the `EC2` value**, **are required** to set both settings **and** to set them to one of the couples specified in the table.
-Fargate task definitions support **only** those specific values for tasks' CPU and memory. +Fargate task definitions support **only** those [specific values for tasks' CPU and memory][fargate tasks sizes]. | CPU units | vCPUs | Memory values | Supported OSes | Notes | | --------- | ----- | ------------------------------------------- | -------------- | -------------------------------- | @@ -440,6 +440,7 @@ Specify a supported value for the task CPU and memory in your task definition. [amazon ecs task definition differences for the fargate launch type]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-tasks-services.html [amazon ecs task lifecycle]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-lifecycle-explanation.html [amazon ecs task role]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html +[fargate tasks sizes]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-tasks-services.html#fargate-tasks-size [how amazon ecs manages cpu and memory resources]: https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/ [how amazon elastic container service works with iam]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security_iam_service-with-iam.html [identity and access management for amazon elastic container service]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam.html diff --git a/knowledge base/logstash.md b/knowledge base/logstash.md index 23f3ed9..8f5edac 100644 --- a/knowledge base/logstash.md +++ b/knowledge base/logstash.md @@ -1,6 +1,4 @@ -# Title - -TODO +# Logstash Server-side data processing pipeline that ingests data, transforms it, and then sends the results to any collector. @@ -14,7 +12,7 @@ Part of the Elastic Stack along with Beats, [ElasticSearch] and [Kibana]. - -[main repository]: https://github.com/elastic/logstash +[codebase]: https://github.com/elastic/logstash [website]: https://website/ +[how to debug your logstash configuration file]: https://logz.io/blog/debug-logstash/ diff --git a/knowledge base/loki.md b/knowledge base/loki.md index 01f7b6b..8869d9e 100644 --- a/knowledge base/loki.md +++ b/knowledge base/loki.md @@ -14,12 +14,16 @@ very cost-effective and easy to operate. 1. [Compactor](#compactor) 1. [Ruler](#ruler) 1. [Clients](#clients) + 1. [Logstash](#logstash) + 1. [OpenTelemetry](#opentelemetry) 1. [Labels](#labels) 1. [Labelling best practices](#labelling-best-practices) 1. [Deployment](#deployment) 1. [Monolithic mode](#monolithic-mode) 1. [Simple scalable mode](#simple-scalable-mode) 1. [Microservices mode](#microservices-mode) +1. [Object storage](#object-storage) +1. [Analytics](#analytics) 1. [Further readings](#further-readings) 1. [Sources](#sources) @@ -247,6 +251,28 @@ Multiple rulers will use a consistent hash ring to distribute rule groups amongs Refer [Send log data to Loki]. +### Logstash + +Loki provides the `logstash-output-loki` Logstash output plugin to enable shipping logs to a Loki or Grafana Cloud +instance.
+Refer [Logstash plugin]. + +```sh +logstash-plugin install 'logstash-output-loki' +``` + +```rb +output { + loki { + url => "http://loki.example.org:3100/loki/api/v1/push" + } +} +``` + +### OpenTelemetry + +See also [OpenTelemetry / OTLP]. + ## Labels The content of each log line is **not** indexed. Instead, log entries are grouped into streams.
@@ -262,7 +288,8 @@ namespace = grafana-server Sets of log messages that share all the labels above would be called a _log stream_. -Loki has a default limit of 15 index labels. +Loki has a default limit of 15 index labels.
+I can't seem to find ways to set this up as of 2025-01-21. When Loki performs searches, it: @@ -396,6 +423,79 @@ Designed for Kubernetes deployments and available as the [loki-distributed] comm Only recommended for very large Loki clusters, or when needing more precise control over them. +## Object storage + +Refer [Storage] and [Loki S3 Storage: A Guide for Efficient Log Management]. + +
+ AWS example + +Refer also [AWS deployment (S3 Single Store)]. + +
+ Permissions + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:DeleteObject", + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::", + "arn:aws:s3:::/*", + "arn:aws:s3:::", + "arn:aws:s3:::/*" + ] + } + ] +} +``` + +
+
+ Loki settings + +```yaml +storage_config: + aws: + region: + bucketnames: # comma-separated list + +schema_config: + configs: + - store: tsdb + object_store: aws +compactor: + delete_request_store: aws +ruler: + storage: + type: s3 + s3: + region: + bucketnames: # comma-separated list + +``` + +
+
+ +## Analytics + +By default, Loki will send anonymous but uniquely-identifiable usage and configuration analytics to Grafana Labs.
+Explicitly disable reporting if wanted: + +```yaml +analytics: + reporting_enabled: false +``` + ## Further readings - [Website] @@ -406,6 +506,10 @@ Only recommended for very large Loki clusters, or when needing more precise cont - [Grafana Loki store log data on S3 bucket on AWS Fargate] - [How to install Loki on (AWS) EKS using Terraform with S3] - [Deploy the Loki Helm chart on AWS] +- [Loki S3 Storage: A Guide for Efficient Log Management] +- [Grafana Loki Configuration Nuances] +- [OpenTelemetry / OTLP] +- [Loki-Operator] ### Sources @@ -413,7 +517,7 @@ Only recommended for very large Loki clusters, or when needing more precise cont - [HTTP API reference] - [How to Set Up Grafana, Loki, and Prometheus Locally with Docker Compose: Part 1 of 3] - [Deploying Grafana, Loki, and Prometheus on AWS ECS with EFS and Cloud Formation (Part 3 of 3)] -- [Storage - AWS deployment (S3 Single Store)] +- [AWS deployment (S3 Single Store)] +[aws deployment (s3 single store)]: https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store [codebase]: https://github.com/grafana/loki [deploy the loki helm chart on aws]: https://grafana.com/docs/loki/latest/setup/install/helm/deployment-guides/aws/ [documentation]: https://grafana.com/docs/loki/latest/ @@ -447,9 +552,14 @@ Only recommended for very large Loki clusters, or when needing more precise cont [http api reference]: https://grafana.com/docs/loki/latest/reference/loki-http-api/ [loki-distributed]: https://github.com/grafana/helm-charts/tree/main/charts/loki-distributed [send log data to loki]: https://grafana.com/docs/loki/latest/send-data/ -[storage - aws deployment (s3 single store)]: https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store +[storage]: https://grafana.com/docs/loki/latest/configure/storage/ [website]: https://grafana.com/oss/loki/ +[logstash plugin]: https://grafana.com/docs/loki/latest/send-data/logstash/ [deploying grafana, loki, and prometheus on aws ecs with efs and cloud formation (part 3 of 3)]: https://medium.com/@ahmadbilalch891/deploying-grafana-loki-and-prometheus-on-aws-ecs-with-efs-and-cloud-formation-part-3-of-3-24140ea8ccfb +[grafana loki configuration nuances]: https://medium.com/lonto-digital-services-integrator/grafana-loki-configuration-nuances-2e9b94da4ac1 [how to set up grafana, loki, and prometheus locally with docker compose: part 1 of 3]: https://medium.com/@ahmadbilalch891/how-to-set-up-grafana-loki-and-prometheus-locally-with-docker-compose-part-1-of-3-62fb25e51d92 +[loki s3 storage: a guide for efficient log management]: https://last9.io/blog/loki-s3-storage-guide/ +[loki-operator]: https://loki-operator.dev/ +[opentelemetry / otlp]: https://loki-operator.dev/docs/open-telemetry.md/