mirror of
https://gitea.com/mcereda/oam.git
synced 2026-02-09 05:44:23 +00:00
133 lines
4.9 KiB
Markdown
133 lines
4.9 KiB
Markdown
# Prometheus
|
|
|
|
Monitoring and alerting system that collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.<br/>
|
|
Metrics can also be pushed using plugins, in the event hosts are behind a firewall or prohibited from opening ports by security policy.
|
|
|
|
## Table of contents <!-- omit in toc -->
|
|
|
|
1. [Components](#components)
|
|
1. [Extras](#extras)
|
|
1. [Configuration](#configuration)
|
|
1. [Queries](#queries)
|
|
1. [Further readings](#further-readings)
|
|
1. [Sources](#sources)
|
|
|
|
## Components
|
|
|
|
Prometheus is composed by its **server**, the **Alertmanager** and its **exporters**.
|
|
|
|
Alerting rules can be created within Prometheus, and configured to send custom alerts to _Alertmanager_.<br/>
|
|
Alertmanager then processes and handles the alerts, including sending notifications through different mechanisms or third-party services.
|
|
|
|
The _exporters_ can be libraries, processes, devices, or anything else exposing metrics so that they can be scraped by Prometheus.<br/>
|
|
Such metrics are usually made available at the `/metrics` endpoint, which allows them to be scraped directly from Prometheus without the need of an agent.
|
|
|
|
### Extras
|
|
|
|
As welcomed addition, [Grafana] can be configured to use Prometheus as a backend of its in order to provide data visualization and dashboarding functions on the data it provides.
|
|
|
|
## Configuration
|
|
|
|
The default configuration file is at `/etc/prometheus/prometheus.yml`.
|
|
|
|
```yml
|
|
global:
|
|
scrape_interval: 15s
|
|
scrape_configs:
|
|
- job_name: prometheus
|
|
static_configs:
|
|
- targets: [ 'localhost:9090' ]
|
|
- job_name: nodes
|
|
static_configs:
|
|
- targets:
|
|
- fqdn:9100
|
|
- host.local:9100
|
|
- job_name: router
|
|
static_configs:
|
|
- targets: [ 'openwrt.local:9100' ]
|
|
```
|
|
|
|
## Queries
|
|
|
|
Prometheus' query syntax is [PromQL].
|
|
|
|
All data is stored as time series, each one identified by a metric name, e.g. `node_filesystem_avail_bytes` for available filesystem space.<br/>
|
|
Metrics' names can be used in the expressions to select all of the time series with this name and produce an **instant vector**.
|
|
|
|
Time series can be filtered using selectors and labels (sets of key-value pairs):
|
|
|
|
```promql
|
|
node_filesystem_avail_bytes{fstype="ext4"}
|
|
node_filesystem_avail_bytes{fstype!="xfs"}
|
|
```
|
|
|
|
Square brackets allow to select a range of samples from the current time backwards:
|
|
|
|
```promql
|
|
node_memory_MemAvailable_bytes[5m]
|
|
```
|
|
|
|
When using time ranges, the vector returned will be a **range vector**.
|
|
|
|
[Functions] can be used to build advanced queries:
|
|
|
|
```promql
|
|
100 * (1 - avg by(instance)(irate(node_cpu_seconds_total{job='node_exporter',mode='idle'}[5m])))
|
|
```
|
|
|
|
|
|
|
|
Labels are used to filter the job and the mode. `node_cpu_seconds_total` returns a **counter**, and the irate() function calculates the **per-second rate of change** based on the last two data points of the range interval.<br/>
|
|
To calculate the overall CPU usage, the idle mode of the metric is used. Since idle percent of a processor is the opposite of a busy processor, the irate value is subtracted from 1. To make it a percentage, it is multiplied by 100.
|
|
|
|
## Further readings
|
|
|
|
- [Website]
|
|
- [Github]
|
|
- [`docker/monitoring`][docker/monitoring]
|
|
- [Node exporter]
|
|
- [SMNP exporter]
|
|
- [`ordaa/boinc_exporter`][ordaa/boinc_exporter]
|
|
- [Grafana]
|
|
|
|
## Sources
|
|
|
|
All the references in the [further readings] section, plus the following:
|
|
|
|
- [Getting started with Prometheus]
|
|
- [Node exporter guide]
|
|
- [SNMP monitoring and easing it with Prometheus]
|
|
- [`prometheus/node_exporter`][prometheus/node_exporter]
|
|
- [`prometheus/snmp_exporter`][prometheus/snmp_exporter]
|
|
- [How I monitor my OpenWrt router with Grafana Cloud and Prometheus]
|
|
|
|
<!--
|
|
References
|
|
-->
|
|
|
|
<!-- Upstream -->
|
|
[functions]: https://prometheus.io/docs/prometheus/latest/querying/functions/
|
|
[github]: https://github.com/prometheus/prometheus
|
|
[node exporter guide]: https://prometheus.io/docs/guides/node-exporter/
|
|
[prometheus/node_exporter]: https://github.com/prometheus/node_exporter
|
|
[prometheus/snmp_exporter]: https://github.com/prometheus/snmp_exporter
|
|
[promql]: https://prometheus.io/docs/prometheus/latest/querying/basics/
|
|
[website]: https://prometheus.io/
|
|
|
|
<!-- In-article sections -->
|
|
[further readings]: #further-readings
|
|
|
|
<!-- Knowledge base -->
|
|
[grafana]: grafana.md
|
|
[node exporter]: node%20exporter.md
|
|
[snmp exporter]: snmp%20exporter.md
|
|
|
|
<!-- Files -->
|
|
[docker/monitoring]: ../docker/monitoring/README.md
|
|
|
|
<!-- Others -->
|
|
[getting started with prometheus]: https://opensource.com/article/18/12/introduction-prometheus
|
|
[how i monitor my openwrt router with grafana cloud and prometheus]: https://grafana.com/blog/2021/02/09/how-i-monitor-my-openwrt-router-with-grafana-cloud-and-prometheus/
|
|
[ordaa/boinc_exporter]: https://gitlab.com/ordaa/boinc_exporter
|
|
[snmp monitoring and easing it with prometheus]: https://medium.com/@openmohan/snmp-monitoring-and-easing-it-with-prometheus-b157c0a42c0c
|