diff --git a/knowledge base/flog.md b/knowledge base/flog.md new file mode 100644 index 0000000..3820db7 --- /dev/null +++ b/knowledge base/flog.md @@ -0,0 +1,88 @@ +# Flog + +Fake log generator for common log formats. + +1. [TL;DR](#tldr) +1. [Further readings](#further-readings) + 1. [Sources](#sources) + +## TL;DR + +
+ Setup + +```sh +go install 'github.com/mingrammer/flog' +docker run --rm -it 'mingrammer/flog' +brew install 'mingrammer/flog/flog' +``` + +
+ +
+ Usage + +```sh +# Generate 1000 lines of logs to stdout. +flog + +# Generate 200 lines of logs to stdout. +# Wait 1s for each line. +flog -n '200' -d '1' +flog --number '200' --delay '1s' + +# Generate a single log file with 1000 lines of logs. +# Overwrite existing log files. +flog -t 'log' -w +flog --type 'log' --overwrite + +# Generate a single gzipped log file with 3000 lines. +flog -t 'gz' -o 'log.gz' -n '3000' +flog --type 'gz' --output 'log.gz' --number '3000' + +# Generate up to 10MB of logs. +# Split logs in files every 1MB. +# Use the 'apache combined' format. +flog -t 'log' -f 'apache_combined' -o 'web/log/apache.log' -b '10485760' -p '1048576' +flog --type 'log' --format 'apache_combined' --output 'web/log/apache.log' --bytes '10485760' --split-by '1048576' + +# Generate logs in the rfc3164 format until killed. +flog -f 'rfc3164' -l +flog --format 'rfc3164' --loop +``` + +
+ + + +## Further readings + +- [Website] +- [Codebase] + +### Sources + +- [Documentation] + + + + + + + +[codebase]: https://github.com/mingrammer/flog +[documentation]: https://website/docs/ +[website]: https://website/ + + diff --git a/knowledge base/fluent bit.md b/knowledge base/fluent bit.md index 708cebb..73118a1 100644 --- a/knowledge base/fluent bit.md +++ b/knowledge base/fluent bit.md @@ -2,7 +2,8 @@ TODO -Lightweight and highly scalable logging and metrics processor and forwarder. +Lightweight and highly scalable logging and metrics processor and forwarder.
+Technically a stripped-down version of [Fluentd]. 1. [TL;DR](#tldr) 1. [Further readings](#further-readings) @@ -48,6 +49,8 @@ Lightweight and highly scalable logging and metrics processor and forwarder. ### Sources +- [Fluentd vs. Fluent Bit: side by side comparison] + +[fluentd vs. fluent bit: side by side comparison]: https://logz.io/blog/fluentd-vs-fluent-bit/ diff --git a/knowledge base/fluentd.md b/knowledge base/fluentd.md index dea1aa1..338940c 100644 --- a/knowledge base/fluentd.md +++ b/knowledge base/fluentd.md @@ -48,6 +48,8 @@ Open-Source log collector. ### Sources +- [Fluentd vs. Fluent Bit: side by side comparison] + [fluent bit]: fluent%20bit.md + [main repository]: https://github.com/fluent/fluentd [website]: https://www.fluentd.org/ +[fluentd vs. fluent bit: side by side comparison]: https://logz.io/blog/fluentd-vs-fluent-bit/ diff --git a/knowledge base/grafana.md b/knowledge base/grafana.md index bd2a7fb..9c67ffe 100644 --- a/knowledge base/grafana.md +++ b/knowledge base/grafana.md @@ -13,6 +13,9 @@ Open-source platform for monitoring and observability. ## TL;DR +
+ Usage + ```sh # Export all existing dashboards by ID. curl -sS \ @@ -27,6 +30,8 @@ curl -sS \ " ``` +
+ ## Installation
@@ -195,8 +200,9 @@ $ curl -sS \ - [Github] - [HTTP API reference] - [Prometheus] -- [`containers/monitoring`][containers/monitoring] +- [docker compositions/monitoring] - Official [helm chart] +- [Loki] ## Sources @@ -218,10 +224,11 @@ All the references in the [further readings] section, plus the following: [further readings]: #further-readings +[loki]: loki.md [prometheus]: prometheus.md -[containers/monitoring]: ../containers/monitoring/README.md +[docker compositions/monitoring]: ../docker%20compositions/monitoring/README.md [data source on startup]: https://community.grafana.com/t/data-source-on-startup/8618/2 diff --git a/knowledge base/logging.md b/knowledge base/logging.md new file mode 100644 index 0000000..a76c187 --- /dev/null +++ b/knowledge base/logging.md @@ -0,0 +1,71 @@ +# Logging + +> TODO + +Intro + + + +1. [TL;DR](#tldr) +1. [Further readings](#further-readings) + 1. [Sources](#sources) + +## TL;DR + + + + + + + +## Further readings + +- Grafana's [Loki] +- [Fluentd] / [Fluent Bit] +- [LogStash] + +### Sources + +- [Distributed logging for Microservices] + + + + + +[fluent bit]: fluent%20bit.md +[fluentd]: fluentd.md +[logstash]: logstash.md +[loki]: loki.md + + + + +[distributed logging for microservices]: https://www.geeksforgeeks.org/distributed-logging-for-microservices/ diff --git a/knowledge base/loki.md b/knowledge base/loki.md new file mode 100644 index 0000000..6722557 --- /dev/null +++ b/knowledge base/loki.md @@ -0,0 +1,76 @@ +# Grafana Loki + +Horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus and designed to be +very cost-effective and easy to operate. + +1. [TL;DR](#tldr) +1. [Further readings](#further-readings) + 1. [Sources](#sources) + +## TL;DR + +It indexes **a set of labels** for each log stream instead of the full logs' contents. + +Needs agents or other clients to push logs to the server. + +Supports object storage solutions. + +
+ Setup + +```sh +# Install via package repository. +apt install 'loki' +dnf install 'loki' + +# Run via Docker. +docker run --name loki -d \ + -p '3100:3100' -v "$(pwd)/config/loki.yml:/etc/loki/config.yml:ro" \ + 'grafana/loki:3.3.2' -config.file='/etc/loki/config.yml' +``` + +Default configuration file for package-based installations is `/etc/loki/config.yml`. + +
+ +
+ Usage + +```sh +# Check the server is working +curl 'http://loki.fqdn:3100/ready' +curl 'http://loki.fqdn:3100/metrics' +``` + +
+ +## Further readings + +- [Website] +- [Codebase] +- [Grafana] +- [Promtail] + +### Sources + +- [Documentation] +- [HTTP API reference] + + + + + +[grafana]: grafana.md +[promtail]: promtail.md + + + +[codebase]: https://github.com/grafana/loki +[documentation]: https://grafana.com/docs/loki/latest/ +[http api reference]: https://grafana.com/docs/loki/latest/reference/loki-http-api/ +[website]: https://grafana.com/oss/loki/ + + diff --git a/knowledge base/promtail.md b/knowledge base/promtail.md new file mode 100644 index 0000000..184ea71 --- /dev/null +++ b/knowledge base/promtail.md @@ -0,0 +1,75 @@ +# Promtail + +> TODO + +Agent shipping the contents of **local** logs (e.g. files, systemd's journal, k8s pods) to some Grafana Loki instance. + +1. [TL;DR](#tldr) +1. [Further readings](#further-readings) + 1. [Sources](#sources) + +## TL;DR + +Comes packaged with [Loki] releases. + +
+ Setup + +```sh +# Install via package repository. +apt install 'promtail' +dnf install 'promtail' + +# Run via Docker. +docker run --name 'promtail' -d \ + -p '9080:9080' -v "$(pwd)/config/promtail.yml:/etc/promtail/config.yml:ro" \ + 'grafana/promtail:3.3.2' -config.file='/etc/promtail/config.yml' +``` + +Default configuration file for package-based installations is `/etc/promtail/config.yml`. + +
+ +
+ Usage + +```sh +# Do a test run +promtail -dry-run -config.file '/etc/promtail/config.yml' + +# Check the server is working +curl 'http://promtail.fqdn:9080/ready' +curl 'http://promtail.fqdn:9080/metrics' + +# Connect to the web server +open 'http://promtail.fqdn:9080/' +``` + +
+ +## Further readings + +- [Codebase] +- [Grafana] +- [Loki] + +### Sources + +- [Documentation] + + + + + +[grafana]: grafana.md +[loki]: loki.md + + + +[codebase]: https://github.com/grafana/loki +[documentation]: https://grafana.com/docs/loki/latest/send-data/promtail/ + + diff --git a/snippets/grafana.sh b/snippets/grafana.sh new file mode 100644 index 0000000..4e15f17 --- /dev/null +++ b/snippets/grafana.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env sh + +# Export all existing dashboards by ID +curl -sS \ + -H 'Authorization: Basic YWRtaW46YWRtaW4=' \ + 'http://grafana:3000/api/search' \ +| jq -r '.[].uid' - \ +| parallel " \ + curl -sS \ + -H 'Authorization: Basic YWRtaW46YWRtaW4=' \ + 'http://grafana:3000/api/dashboards/uid/{}' \ + > '{}.json' \ +" diff --git a/snippets/loki.sh b/snippets/loki.sh new file mode 100644 index 0000000..22d1a1e --- /dev/null +++ b/snippets/loki.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env sh + +# Check the server is working +curl 'http://loki.fqdn:3100/ready' +curl 'http://loki.fqdn:3100/metrics' diff --git a/snippets/promtail.sh b/snippets/promtail.sh new file mode 100644 index 0000000..6df6396 --- /dev/null +++ b/snippets/promtail.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env sh + +# Install +apt install 'promtail' +brew install 'promtail' +dnf install 'promtail' +docker run \ + -v "${PWD}/config.yml:/etc/promtail/config.yml" -v '/var/log:/var/log' \ + 'grafana/promtail:3.2.1' --config.file='/etc/promtail/config.yml' +helm upgrade --install 'promtail' \ + --repo 'https://grafana.github.io/helm-charts' 'grafana/promtail' \ + --values 'values.yaml' + +# Validate config files. +# Seems to be quite useless, it does not find stupid configuration errors. +promtail -check-syntax -config.file '/etc/promtail/config.yml' + +# Do a test run +promtail -dry-run -config.file '/etc/promtail/config.yml' + +# Check the server is working +curl 'http://promtail.fqdn:9080/ready' +curl 'http://promtail.fqdn:9080/metrics' + +# Connect to the web server +open 'http://promtail.fqdn:9080/'