diff --git a/knowledge base/prometheus/README.md b/knowledge base/prometheus/README.md index 95564e2..e23e474 100644 --- a/knowledge base/prometheus/README.md +++ b/knowledge base/prometheus/README.md @@ -43,6 +43,8 @@ Prometheus to scrape from.
Exporters are small and purpose-built applications that collect their objects' metrics in different ways, then expose them in an HTTP endpoint in their place. +Prometheus requires a configuration file for scraping settings. +
Setup @@ -70,6 +72,11 @@ helm --namespace 'prometheus' uninstall 'prometheus' prometheus prometheus --web.enable-admin-api +# Validate the configuration file. +promtool check config /etc/prometheus/prometheus.yml +docker run --rm -v "$PWD/config.yaml:/etc/prometheus/prometheus.yml:ro" --entrypoint 'promtool' 'prom/prometheus' \ + check config /etc/prometheus/prometheus.yml + # Reload the configuration file *without* restarting the process. kill -s 'SIGHUP' '3969' pkill --signal 'HUP' 'prometheus' @@ -78,6 +85,10 @@ curl -i -X 'POST' 'localhost:9090/-/reload' # if admin APIs are enabled # Shut down the process *gracefully*. kill -s 'SIGTERM' '3969' pkill --signal 'TERM' 'prometheus' + +# Push test metrics to a remote. +promtool push metrics 'http://mimir.example.org:8080/api/v1/push' +docker run --rm --entrypoint 'promtool' 'prom/prometheus' push metrics 'http://mimir.example.org:8080/api/v1/push' ```
@@ -147,8 +158,22 @@ kubectl -n 'monitoring' get pods -l -l "app=prometheus-pushgateway,component=pus ## Configuration +Refer [Configuration]. + +Prometheus is configured via **both** command-line flags **and** a configuration file. + +The command-line flags configure **immutable system parameters** (e.g. storage locations, amount of data to keep on disk +and in memory).
+The configuration file is a YAML file that defines everything related to: + +- Scraping jobs and their instances. +- Which rule files to load. + The default configuration file is at `/etc/prometheus/prometheus.yml`. +
+ Configuration file example + ```yml global: scrape_interval: 15s @@ -170,12 +195,22 @@ scrape_configs: regex: '(node_cpu)' ``` -Reload the configuration with**out** restarting Prometheus's process by using the `SIGHUP` signal: +
-```sh -kill -s 'SIGHUP' '3969' -pkill --signal 'HUP' 'prometheus' -``` +Prometheus can reload the configuration file with**out** restarting its process by + +- Sending the `SIGHUP` signal to the process: + + ```sh + kill -s 'SIGHUP' '3969' + pkill --signal 'HUP' 'prometheus' + ``` + +- Sending a `POST` HTTP request to the `/-/reload` endpoint.
+ Requires the process to start with the `--web.enable-lifecycle` flag enabled. + +If the new configuration is **not** well-formed, changes will **not** be applied.
+This will also reload any configured rule files. ### Filter metrics @@ -518,6 +553,7 @@ Typically achieved by: [codebase]: https://github.com/prometheus/prometheus +[Configuration]: https://prometheus.io/docs/prometheus/latest/configuration/configuration/ [documentation]: https://prometheus.io/docs/ [Exporters and integrations]: https://prometheus.io/docs/instrumenting/exporters/ [functions]: https://prometheus.io/docs/prometheus/latest/querying/functions/