brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(logstash): send logs to loki

Browse Source
This commit is contained in:
Michele Cereda
2025-01-21 21:03:32 +01:00
parent 31150325dd
commit ff833002e6
4 changed files with 157 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.vscode/settings.json vendored
View File

@@ -189,6 +189,7 @@
"helmfile",
"hoppscotch",
"imager",
"ingester",
"istio",
"istioctl",
"jdupes",

3
knowledge base/cloud computing/aws/ecs.md
View File

@@ -249,7 +249,7 @@ Such tasks support CPU values between 0.25 and 10 vCPUs. these fields are option
Task definitions specifying `FARGATE` as value for the `requiresCompatibilities` attribute, **even if they also specify
the `EC2` value**, **are required** to set both settings **and** to set them to one of the couples specified in the
table.<br/>
Fargate task definitions support **only** those specific values for tasks' CPU and memory.
Fargate task definitions support **only** those [specific values for tasks' CPU and memory][fargate tasks sizes].
| CPU units | vCPUs | Memory values | Supported OSes | Notes |
| --------- | ----- | ------------------------------------------- | -------------- | -------------------------------- |
@@ -440,6 +440,7 @@ Specify a supported value for the task CPU and memory in your task definition.
[amazon ecs task definition differences for the fargate launch type]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-tasks-services.html
[amazon ecs task lifecycle]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-lifecycle-explanation.html
[amazon ecs task role]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
[fargate tasks sizes]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-tasks-services.html#fargate-tasks-size
[how amazon ecs manages cpu and memory resources]: https://aws.amazon.com/blogs/containers/how-amazon-ecs-manages-cpu-and-memory-resources/
[how amazon elastic container service works with iam]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security_iam_service-with-iam.html
[identity and access management for amazon elastic container service]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam.html

49
knowledge base/logstash.md
View File

@@ -1,6 +1,4 @@
# Title
TODO
# Logstash
Server-side data processing pipeline that ingests data, transforms it, and then sends the results to any collector.
@@ -14,7 +12,7 @@ Part of the Elastic Stack along with Beats, [ElasticSearch] and [Kibana].
<!-- Uncomment if used
<details>
<summary>Installation and configuration</summary>
<summary>Setup</summary>
```sh
```
@@ -22,15 +20,47 @@ Part of the Elastic Stack along with Beats, [ElasticSearch] and [Kibana].
</details>
-->
<!-- Uncomment if used
<details>
<summary>Usage</summary>
```sh
# Validate configuration files.
logstash -tf 'config.conf'
logstash --config.test_and_exit --path.config 'config.conf'
# Install plugins.
logstash-plugin install 'logstash-output-loki'
# List installed plugins.
logstash-plugin list
logstash-plugin list --verbose
logstash-plugin list '*namefragment*'
logstash-plugin list --group 'output'
```
```rb
input { … }
filter {
mutate {
add_field => {
"cluster" => "us-central-1"
"job" => "logstash"
}
replace => { "type" => "stream"}
remove_field => [ "src" ]
}
}
output {
loki {
url => "http://loki.example.org:3100/loki/api/v1/push"
}
}
```
</details>
-->
<!-- Uncomment if used
<details>
@@ -45,11 +75,13 @@ Part of the Elastic Stack along with Beats, [ElasticSearch] and [Kibana].
## Further readings
- [Website]
- [Main repository]
- [Codebase]
- [Beats], [ElasticSearch] and [Kibana]: the rest of the Elastic stack
### Sources
- [How to debug your Logstash configuration file]
<!--
Reference
═╬═Time══
@@ -63,7 +95,8 @@ Part of the Elastic Stack along with Beats, [ElasticSearch] and [Kibana].
<!-- Files -->
<!-- Upstream -->
[main repository]: https://github.com/elastic/logstash
[codebase]: https://github.com/elastic/logstash
[website]: https://website/
<!-- Others -->
[how to debug your logstash configuration file]: https://logz.io/blog/debug-logstash/

116
knowledge base/loki.md
View File

@@ -14,12 +14,16 @@ very cost-effective and easy to operate.
1. [Compactor](#compactor)
1. [Ruler](#ruler)
1. [Clients](#clients)
1. [Logstash](#logstash)
1. [OpenTelemetry](#opentelemetry)
1. [Labels](#labels)
1. [Labelling best practices](#labelling-best-practices)
1. [Deployment](#deployment)
1. [Monolithic mode](#monolithic-mode)
1. [Simple scalable mode](#simple-scalable-mode)
1. [Microservices mode](#microservices-mode)
1. [Object storage](#object-storage)
1. [Analytics](#analytics)
1. [Further readings](#further-readings)
1. [Sources](#sources)
@@ -247,6 +251,28 @@ Multiple rulers will use a consistent hash ring to distribute rule groups amongs
Refer [Send log data to Loki].
### Logstash
Loki provides the `logstash-output-loki` Logstash output plugin to enable shipping logs to a Loki or Grafana Cloud
instance.<br/>
Refer [Logstash plugin].
```sh
logstash-plugin install 'logstash-output-loki'
```
```rb
output {
loki {
url => "http://loki.example.org:3100/loki/api/v1/push"
}
}
```
### OpenTelemetry
See also [OpenTelemetry / OTLP].
## Labels
The content of each log line is **not** indexed. Instead, log entries are grouped into streams.<br/>
@@ -262,7 +288,8 @@ namespace = grafana-server
Sets of log messages that share all the labels above would be called a _log stream_.
Loki has a default limit of 15 index labels.
Loki has a default limit of 15 index labels.<br/>
I can't seem to find ways to set this up as of 2025-01-21.
When Loki performs searches, it:
@@ -396,6 +423,79 @@ Designed for Kubernetes deployments and available as the [loki-distributed] comm
Only recommended for very large Loki clusters, or when needing more precise control over them.
## Object storage
Refer [Storage] and [Loki S3 Storage: A Guide for Efficient Log Management].
<details>
<summary>AWS example</summary>
Refer also [AWS deployment (S3 Single Store)].
<details style="padding-left: 1em;">
<summary>Permissions</summary>
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::<chunks-bucket-name>",
"arn:aws:s3:::<chunks-bucket-name>/*",
"arn:aws:s3:::<ruler-bucket-name>",
"arn:aws:s3:::<ruler-bucket-name>/*"
]
}
]
}
```
</details>
<details style="padding-left: 1em;">
<summary>Loki settings</summary>
```yaml
storage_config:
aws:
region: <aws-region>
bucketnames: # comma-separated list
<chunks-bucket-name>
schema_config:
configs:
- store: tsdb
object_store: aws
compactor:
delete_request_store: aws
ruler:
storage:
type: s3
s3:
region: <aws-region>
bucketnames: # comma-separated list
<ruler-bucket-name>
```
</details>
</details>
## Analytics
By default, Loki will send anonymous but uniquely-identifiable usage and configuration analytics to Grafana Labs.<br/>
Explicitly disable reporting if wanted:
```yaml
analytics:
reporting_enabled: false
```
## Further readings
- [Website]
@@ -406,6 +506,10 @@ Only recommended for very large Loki clusters, or when needing more precise cont
- [Grafana Loki store log data on S3 bucket on AWS Fargate]
- [How to install Loki on (AWS) EKS using Terraform with S3]
- [Deploy the Loki Helm chart on AWS]
- [Loki S3 Storage: A Guide for Efficient Log Management]
- [Grafana Loki Configuration Nuances]
- [OpenTelemetry / OTLP]
- [Loki-Operator]
### Sources
@@ -413,7 +517,7 @@ Only recommended for very large Loki clusters, or when needing more precise cont
- [HTTP API reference]
- [How to Set Up Grafana, Loki, and Prometheus Locally with Docker Compose: Part 1 of 3]
- [Deploying Grafana, Loki, and Prometheus on AWS ECS with EFS and Cloud Formation (Part 3 of 3)]
- [Storage - AWS deployment (S3 Single Store)]
- [AWS deployment (S3 Single Store)]
<!--
Reference
@@ -439,6 +543,7 @@ Only recommended for very large Loki clusters, or when needing more precise cont
<!-- Files -->
<!-- Upstream -->
[aws deployment (s3 single store)]: https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store
[codebase]: https://github.com/grafana/loki
[deploy the loki helm chart on aws]: https://grafana.com/docs/loki/latest/setup/install/helm/deployment-guides/aws/
[documentation]: https://grafana.com/docs/loki/latest/
@@ -447,9 +552,14 @@ Only recommended for very large Loki clusters, or when needing more precise cont
[http api reference]: https://grafana.com/docs/loki/latest/reference/loki-http-api/
[loki-distributed]: https://github.com/grafana/helm-charts/tree/main/charts/loki-distributed
[send log data to loki]: https://grafana.com/docs/loki/latest/send-data/
[storage - aws deployment (s3 single store)]: https://grafana.com/docs/loki/latest/configure/storage/#aws-deployment-s3-single-store
[storage]: https://grafana.com/docs/loki/latest/configure/storage/
[website]: https://grafana.com/oss/loki/
[logstash plugin]: https://grafana.com/docs/loki/latest/send-data/logstash/
<!-- Others -->
[deploying grafana, loki, and prometheus on aws ecs with efs and cloud formation (part 3 of 3)]: https://medium.com/@ahmadbilalch891/deploying-grafana-loki-and-prometheus-on-aws-ecs-with-efs-and-cloud-formation-part-3-of-3-24140ea8ccfb
[grafana loki configuration nuances]: https://medium.com/lonto-digital-services-integrator/grafana-loki-configuration-nuances-2e9b94da4ac1
[how to set up grafana, loki, and prometheus locally with docker compose: part 1 of 3]: https://medium.com/@ahmadbilalch891/how-to-set-up-grafana-loki-and-prometheus-locally-with-docker-compose-part-1-of-3-62fb25e51d92
[loki s3 storage: a guide for efficient log management]: https://last9.io/blog/loki-s3-storage-guide/
[loki-operator]: https://loki-operator.dev/
[opentelemetry / otlp]: https://loki-operator.dev/docs/open-telemetry.md/