chore: try loki as logging solution

knowledge base/flog.md

@@ -0,0 +1,88 @@
+# Flog
+
+Fake log generator for common log formats.
+
+1. [TL;DR](#tldr)
+1. [Further readings](#further-readings)
+   1. [Sources](#sources)
+
+## TL;DR
+
+<details>
+  <summary>Setup</summary>
+
+```sh
+go install 'github.com/mingrammer/flog'
+docker run --rm -it 'mingrammer/flog'
+brew install 'mingrammer/flog/flog'
+```
+
+</details>
+
+<details>
+  <summary>Usage</summary>
+
+```sh
+# Generate 1000 lines of logs to stdout.
+flog
+
+# Generate 200 lines of logs to stdout.
+# Wait 1s for each line.
+flog -n '200' -d '1'
+flog --number '200' --delay '1s'
+
+# Generate a single log file with 1000 lines of logs.
+# Overwrite existing log files.
+flog -t 'log' -w
+flog --type 'log' --overwrite
+
+# Generate a single gzipped log file with 3000 lines.
+flog -t 'gz' -o 'log.gz' -n '3000'
+flog --type 'gz' --output 'log.gz' --number '3000'
+
+# Generate up to 10MB of logs.
+# Split logs in files every 1MB.
+# Use the 'apache combined' format.
+flog -t 'log' -f 'apache_combined' -o 'web/log/apache.log' -b '10485760' -p '1048576'
+flog --type 'log' --format 'apache_combined' --output 'web/log/apache.log' --bytes '10485760' --split-by '1048576'
+
+# Generate logs in the rfc3164 format until killed.
+flog -f 'rfc3164' -l
+flog --format 'rfc3164' --loop
+```
+
+</details>
+
+<!-- Uncomment if used
+<details>
+  <summary>Real world use cases</summary>
+
+```sh
+```
+
+</details>
+-->
+
+## Further readings
+
+- [Website]
+- [Codebase]
+
+### Sources
+
+- [Documentation]
+
+<!--
+  Reference
+  ═╬═Time══
+  -->
+
+<!-- In-article sections -->
+<!-- Knowledge base -->
+<!-- Files -->
+<!-- Upstream -->
+[codebase]: https://github.com/mingrammer/flog
+[documentation]: https://website/docs/
+[website]: https://website/
+
+<!-- Others -->

knowledge base/fluent bit.md

@@ -2,7 +2,8 @@
 
 TODO
 
-Lightweight and highly scalable logging and metrics processor and forwarder.
+Lightweight and highly scalable logging and metrics processor and forwarder.<br/>
+Technically a stripped-down version of [Fluentd].
 
 1. [TL;DR](#tldr)
 1. [Further readings](#further-readings)
@@ -48,6 +49,8 @@ Lightweight and highly scalable logging and metrics processor and forwarder.
 
 ### Sources
 
+- [Fluentd vs. Fluent Bit: side by side comparison]
+
 <!--
   Reference
   ═╬═Time══
@@ -63,3 +66,4 @@ Lightweight and highly scalable logging and metrics processor and forwarder.
 [website]: https://www.fluentbit.io/
 
 <!-- Others -->
+[fluentd vs. fluent bit: side by side comparison]: https://logz.io/blog/fluentd-vs-fluent-bit/

knowledge base/fluentd.md

@@ -48,6 +48,8 @@ Open-Source log collector.
 
 ### Sources
 
+- [Fluentd vs. Fluent Bit: side by side comparison]
+
 <!--
   Reference
   ═╬═Time══
@@ -56,9 +58,11 @@ Open-Source log collector.
 <!-- In-article sections -->
 <!-- Knowledge base -->
 [fluent bit]: fluent%20bit.md
+
 <!-- Files -->
 <!-- Upstream -->
 [main repository]: https://github.com/fluent/fluentd
 [website]: https://www.fluentd.org/
 
 <!-- Others -->
+[fluentd vs. fluent bit: side by side comparison]: https://logz.io/blog/fluentd-vs-fluent-bit/

knowledge base/grafana.md

@@ -13,6 +13,9 @@ Open-source platform for monitoring and observability.
 
 ## TL;DR
 
+<details>
+  <summary>Usage</summary>
+
 ```sh
 # Export all existing dashboards by ID.
 curl -sS \
@@ -27,6 +30,8 @@ curl -sS \
   "
 ```
 
+</details>
+
 ## Installation
 
 <details>
@@ -195,8 +200,9 @@ $ curl -sS \
 - [Github]
 - [HTTP API reference]
 - [Prometheus]
-- [`containers/monitoring`][containers/monitoring]
+- [docker compositions/monitoring]
 - Official [helm chart]
+- [Loki]
 
 ## Sources
 
@@ -218,10 +224,11 @@ All the references in the [further readings] section, plus the following:
 [further readings]: #further-readings
 
 <!-- Knowledge base -->
+[loki]: loki.md
 [prometheus]: prometheus.md
 
 <!-- Files -->
-[containers/monitoring]: ../containers/monitoring/README.md
+[docker compositions/monitoring]: ../docker%20compositions/monitoring/README.md
 
 <!-- Upstream -->
 [data source on startup]: https://community.grafana.com/t/data-source-on-startup/8618/2

knowledge base/logging.md

@@ -0,0 +1,71 @@
+# Logging
+
+> TODO
+
+Intro
+
+<!-- Remove this line to uncomment if used
+## Table of contents <!-- omit in toc -->
+
+1. [TL;DR](#tldr)
+1. [Further readings](#further-readings)
+   1. [Sources](#sources)
+
+## TL;DR
+
+<!-- Uncomment if used
+<details>
+  <summary>Setup</summary>
+
+```sh
+```
+
+</details>
+-->
+
+<!-- Uncomment if used
+<details>
+  <summary>Usage</summary>
+
+```sh
+```
+
+</details>
+-->
+
+<!-- Uncomment if used
+<details>
+  <summary>Real world use cases</summary>
+
+```sh
+```
+
+</details>
+-->
+
+## Further readings
+
+- Grafana's [Loki]
+- [Fluentd] / [Fluent Bit]
+- [LogStash]
+
+### Sources
+
+- [Distributed logging for Microservices]
+
+<!--
+  Reference
+  ═╬═Time══
+  -->
+
+<!-- In-article sections -->
+<!-- Knowledge base -->
+[fluent bit]: fluent%20bit.md
+[fluentd]: fluentd.md
+[logstash]: logstash.md
+[loki]: loki.md
+
+<!-- Files -->
+<!-- Upstream -->
+<!-- Others -->
+[distributed logging for microservices]: https://www.geeksforgeeks.org/distributed-logging-for-microservices/

knowledge base/loki.md

@@ -0,0 +1,76 @@
+# Grafana Loki
+
+Horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus and designed to be
+very cost-effective and easy to operate.
+
+1. [TL;DR](#tldr)
+1. [Further readings](#further-readings)
+   1. [Sources](#sources)
+
+## TL;DR
+
+It indexes **a set of labels** for each log stream instead of the full logs' contents.
+
+Needs agents or other clients to push logs to the server.
+
+Supports object storage solutions.
+
+<details>
+  <summary>Setup</summary>
+
+```sh
+# Install via package repository.
+apt install 'loki'
+dnf install 'loki'
+
+# Run via Docker.
+docker run --name loki -d \
+  -p '3100:3100' -v "$(pwd)/config/loki.yml:/etc/loki/config.yml:ro" \
+  'grafana/loki:3.3.2' -config.file='/etc/loki/config.yml'
+```
+
+Default configuration file for package-based installations is `/etc/loki/config.yml`.
+
+</details>
+
+<details>
+  <summary>Usage</summary>
+
+```sh
+# Check the server is working
+curl 'http://loki.fqdn:3100/ready'
+curl 'http://loki.fqdn:3100/metrics'
+```
+
+</details>
+
+## Further readings
+
+- [Website]
+- [Codebase]
+- [Grafana]
+- [Promtail]
+
+### Sources
+
+- [Documentation]
+- [HTTP API reference]
+
+<!--
+  Reference
+  ═╬═Time══
+  -->
+
+<!-- In-article sections -->
+<!-- Knowledge base -->
+[grafana]: grafana.md
+[promtail]: promtail.md
+
+<!-- Files -->
+<!-- Upstream -->
+[codebase]: https://github.com/grafana/loki
+[documentation]: https://grafana.com/docs/loki/latest/
+[http api reference]: https://grafana.com/docs/loki/latest/reference/loki-http-api/
+[website]: https://grafana.com/oss/loki/
+
+<!-- Others -->

knowledge base/promtail.md

@@ -0,0 +1,75 @@
+# Promtail
+
+> TODO
+
+Agent shipping the contents of **local** logs (e.g. files, systemd's journal, k8s pods) to some Grafana Loki instance.
+
+1. [TL;DR](#tldr)
+1. [Further readings](#further-readings)
+   1. [Sources](#sources)
+
+## TL;DR
+
+Comes packaged with [Loki] releases.
+
+<details>
+  <summary>Setup</summary>
+
+```sh
+# Install via package repository.
+apt install 'promtail'
+dnf install 'promtail'
+
+# Run via Docker.
+docker run --name 'promtail' -d \
+  -p '9080:9080' -v "$(pwd)/config/promtail.yml:/etc/promtail/config.yml:ro" \
+  'grafana/promtail:3.3.2' -config.file='/etc/promtail/config.yml'
+```
+
+Default configuration file for package-based installations is `/etc/promtail/config.yml`.
+
+</details>
+
+<details>
+  <summary>Usage</summary>
+
+```sh
+# Do a test run
+promtail -dry-run -config.file '/etc/promtail/config.yml'
+
+# Check the server is working
+curl 'http://promtail.fqdn:9080/ready'
+curl 'http://promtail.fqdn:9080/metrics'
+
+# Connect to the web server
+open 'http://promtail.fqdn:9080/'
+```
+
+</details>
+
+## Further readings
+
+- [Codebase]
+- [Grafana]
+- [Loki]
+
+### Sources
+
+- [Documentation]
+
+<!--
+  Reference
+  ═╬═Time══
+  -->
+
+<!-- In-article sections -->
+<!-- Knowledge base -->
+[grafana]: grafana.md
+[loki]: loki.md
+
+<!-- Files -->
+<!-- Upstream -->
+[codebase]: https://github.com/grafana/loki
+[documentation]: https://grafana.com/docs/loki/latest/send-data/promtail/
+
+<!-- Others -->

snippets/grafana.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env sh
+
+# Export all existing dashboards by ID
+curl -sS \
+	-H 'Authorization: Basic YWRtaW46YWRtaW4=' \
+	'http://grafana:3000/api/search' \
+| jq -r '.[].uid' - \
+| parallel " \
+	curl -sS \
+		-H 'Authorization: Basic YWRtaW46YWRtaW4=' \
+		'http://grafana:3000/api/dashboards/uid/{}' \
+	> '{}.json' \
+"

snippets/loki.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env sh
+
+# Check the server is working
+curl 'http://loki.fqdn:3100/ready'
+curl 'http://loki.fqdn:3100/metrics'

snippets/promtail.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env sh
+
+# Install
+apt install 'promtail'
+brew install 'promtail'
+dnf install 'promtail'
+docker run \
+	-v "${PWD}/config.yml:/etc/promtail/config.yml" -v '/var/log:/var/log' \
+	'grafana/promtail:3.2.1' --config.file='/etc/promtail/config.yml'
+helm upgrade --install 'promtail' \
+	--repo 'https://grafana.github.io/helm-charts' 'grafana/promtail' \
+	--values 'values.yaml'
+
+# Validate config files.
+# Seems to be quite useless, it does not find stupid configuration errors.
+promtail -check-syntax -config.file '/etc/promtail/config.yml'
+
+# Do a test run
+promtail -dry-run -config.file '/etc/promtail/config.yml'
+
+# Check the server is working
+curl 'http://promtail.fqdn:9080/ready'
+curl 'http://promtail.fqdn:9080/metrics'
+
+# Connect to the web server
+open 'http://promtail.fqdn:9080/'