brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
c0bea63d8a45dc8a679446993fb4427480e62b4c
oam/knowledge base/grafana.md
Michele Cereda c0bea63d8a refactor(kb/prometheus): give articles their own folder
2025-04-20 14:12:01 +02:00

8.8 KiB
Raw Blame History

Grafana

Open-source platform for monitoring and observability.

  1. TL;DR
  2. Setup
  3. Provisioning
    1. Datasources
    2. Dashboards
  4. Dashboards of interest
  5. Alerting
  6. Further readings
  7. Sources

TL;DR

Usage 
# Export all existing dashboards by ID.
curl -sS \
  -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
  'http://grafana:3000/api/search' \
| jq -r '.[].uid' - \
| parallel " \
    curl -sS \
      -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
      'http://grafana:3000/api/dashboards/uid/{}' \
    > '{}.json' \
  "

Setup

Kubernetes 
helm repo add 'grafana' 'https://grafana.github.io/helm-charts'
helm -n 'monitoring' upgrade -i --create-namespace --set adminPassword='abc0123' 'grafana' 'grafana/grafana'

helm -n 'monitoring' upgrade -i --create-namespace --repo 'https://grafana.github.io/helm-charts' 'grafana' 'grafana'

Access components:

Component From within the cluster
Server grafana.monitoring.svc.cluster.local:80 
# Access the server
kubectl -n 'monitoring' get secret 'grafana' -o jsonpath='{.data.admin-password}' | base64 --decode
kubectl -n 'monitoring' get pods -l 'app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana' \
  -o jsonpath='{.items[0].metadata.name}' \
| xargs -I {} kubectl -n 'monitoring' port-forward {} 3000

Clean up:

helm -n 'monitoring' delete 'grafana'
kubectl delete namespace --ignore-not-found 'monitoring'

Access Prometheus instances in the same namespace using http://prometheus-server

Provisioning

See provision dashboards and data sources for details.

Datasources

Data sources can be managed automatically at provisioning by adding YAML configuration files in the provisioning/datasources directory.

Each configuration file can contain a list of datasources to add or update during startup.
If the data source already exists, Grafana reconfigures it to match the provisioned configuration file.

Grafana also deletes the data sources listed in deleteDatasources before adding or updating those in the datasources list.

---
apiVersion: 1
datasources:
  - id: 1
    name: Prometheus
    orgId: 1
    uid: a17feb01-a0c1-432e-8ef5-7b277cb0b32b
    type: prometheus
    typeName: Prometheus
    typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg
    access: proxy
    url: http://prometheus:9090
    user: ''
    database: ''
    basicAuth: false
    isDefault: true
    jsonData:
      httpMethod: POST
    readOnly: false

The easiest way to write datasources definitions in the configuration file is to:

  1. Login to Grafana as admin

  2. Manually setup the datasource

  3. Issue a GET /api/datasources request to Grafana's API to get the datasource configuration

    curl -sS 'http://grafana:3000/api/datasources' -H 'Authorization: Basic YWRtaW46YWRtaW4='

  4. Edit it as YAML

  5. Drop the YAML definition into the provisioning/datasources directory

$ curl -sS 'http://grafana:3000/api/datasources' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
| yq -y '{apiVersion: 1, datasources: .}' - \
| tee '/etc/grafana/provisioning/datasources/default.yml'
apiVersion: 1
datasources:
  - id: 1
    uid: a17feb01-a0c1-432e-8ef5-7b277cb0b32b
    orgId: 1
    name: Prometheus
    type: prometheus
    typeName: Prometheus
    typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg
    access: proxy
    url: http://rpi4b.lan:9090
    user: ''
    database: ''
    basicAuth: false
    isDefault: true
    jsonData:
      httpMethod: POST
    readOnly: true

Dashboards

Dashboards can be automatically managed by adding one or more YAML config files in the provisioning/dashboards directory.
Each config file can contain a list of dashboards providers that load dashboards into Grafana from the local filesystem.

When Grafana starts, it will insert all dashboards available in the configured path, or update them if they are already present.
Later on it will poll that path every updateIntervalSeconds, look for updated json files and update/insert those into the database.

apiVersion: 1
providers:
  - name: dashboards
    folder: ''
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Save existing dashboards like you would for the datasources.
Save the dashboard definitions in JSON files in the path searched by the provider (e.g. /var/lib/grafana/dashboards).

$ curl -sS \
  -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
  'http://grafana:3000/api/search' \
| jq -r '.[].uid' - \
| parallel " \
    curl -sS \
      -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
      'http://grafana:3000/api/dashboards/uid/{}' \
    > '/var/lib/grafana/dashboards/{}.json' \
  "

Dashboards of interest

Name Grafana ID URLs
Node exporter full 1860 grafana, github raw
OpenWRT 11147 grafana
prometheus 2.0 overview 3662 FIXME
kubernetes cluster (prometheus) 6417 FIXME
Nextcloud 9632 FIXME

Alerting

Refer alerting and Get started with Grafana Alerting.

  1. Create a contact point if not existing already.
  2. Create an alert rule.

Further readings

Sources

All the references in the further readings section, plus the following: