brandon/oam

Fork 0

mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00

Files

Michele Cereda 3147e2df49 fix(grafana): refer the executable instead of the product

2025-07-10 20:18:23 +02:00

12 KiB

Raw Blame History

Grafana

Open-source platform for monitoring and observability.

TL;DR
Setup
Configuration
Provisioning
1. Datasources
2. Dashboards
Dashboards of interest
Alerting
APIs
Further readings
Sources

TL;DR

Grafana needs a database to store users, dashboards, and other data. It supports mysql, postgres or sqlite3.
By default it uses the sqlite3 embedded database included in the main grafana binary.

Grafana creates a folder for each installed plugin, containing its associated files and data.
Plugin files are located by default in $PWD/data/plugins (/var/lib/grafana/plugins for deb or rpm packages).

Usage

docker run -d --rm --name 'grafana-oss' -p '3000:3000' 'grafana/grafana-oss'
docker run -d --rm --name 'grafana-enterprise' -p '3000:3000' -ti -entrypoint 'bash' 'grafana/grafana-enterprise'

# Return health information
GET /api/health

Real world use cases

# Export all existing dashboards by ID.
curl -sS 'http://grafana:3000/api/search' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
| jq -r '.[].uid' - \
| parallel " \
    curl -sS 'http://grafana:3000/api/dashboards/uid/{}' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
    > '{}.json' \
  "

Setup

Kubernetes

helm repo add 'grafana' 'https://grafana.github.io/helm-charts'
helm -n 'monitoring' upgrade -i --create-namespace --set adminPassword='abc0123' 'grafana' 'grafana/grafana'

helm -n 'monitoring' upgrade -i --create-namespace --repo 'https://grafana.github.io/helm-charts' 'grafana' 'grafana'

Access components:

Component	From within the cluster
Server	`grafana.monitoring.svc.cluster.local:80`

# Access the server
kubectl -n 'monitoring' get secret 'grafana' -o jsonpath='{.data.admin-password}' | base64 --decode
kubectl -n 'monitoring' get pods -l 'app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana' \
  -o jsonpath='{.items[0].metadata.name}' \
| xargs -I '%%' kubectl -n 'monitoring' port-forward "%%" '3000'

Clean up:

helm -n 'monitoring' delete 'grafana'
kubectl delete namespace --ignore-not-found 'monitoring'

Access Prometheus instances in the same namespace using http://prometheus-server.

Configuration

Refer Configuration file location.

Grafana searches for default settings in the ${PWD}/conf/defaults.ini file. Do not change this file.
Depending on the executing OS, Grafana searches for custom configuration in the ${PWD}/conf/custom.ini or in the /usr/local/etc/grafana/grafana.ini files. Specify a custom file path with the --config option.

deb and rpm packages put the custom configuration file at /etc/grafana/grafana.ini, and do not use a separate custom.ini file. That path is specified in Grafana's init script using the --config option.

Prefer using environmental variables to override existing options (and not to add them).

All letters must be uppercase.
Replace periods (.) and dashes (-) with underscores (_).

# default section
instance_name = ${HOSTNAME}

[security]
admin_user = admin

[auth.google]
client_secret = 0ldS3cretKey

[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true

[feature_toggles]
enable = newNavigation

# export GF_${SECTION NAME}_${KEY}='value'
export \
  GF_DEFAULT_INSTANCE_NAME='some-instance' \
  GF_SECURITY_ADMIN_USER='owner' \
  GF_AUTH_GOOGLE_CLIENT_SECRET='newS3cretKey' \
  GF_PLUGIN_GRAFANA_IMAGE_RENDERER_RENDERING_IGNORE_HTTPS_ERRORS=true \
  GF_FEATURE_TOGGLES_ENABLE='newNavigation'

Grafana evaluates options containing the expression $__<PROVIDER>{<ARGUMENT>}or ${<ENVIRONMENT VARIABLE>}.
The evaluation runs the specified provider with the provided argument to get the final value of the option.

Available providers are env, file, and vault.

The env provider expands environment variables.
One can also use the short-hand syntax ${PORT}.

instance_name = ${HOSTNAME}

[paths]
logs = $__env{LOGDIR}/grafana

The file provider reads a value from a file in the filesystem.
It trims whitespace from the beginning and the end of files.

[database]
password = $__file{/etc/secrets/gf_sql_password}

The vault provider integrates with Hashicorp Vault. It is only available in Grafana Enterprise.

Provisioning

See provision dashboards and data sources for details.

Datasources

Data sources can be managed automatically at provisioning by adding YAML configuration files in the provisioning/datasources directory.

Each configuration file can contain a list of datasources to add or update during startup.
If the data source already exists, Grafana reconfigures it to match the provisioned configuration file.

Grafana also deletes the data sources listed in deleteDatasources before adding or updating those in the datasources list.

---
apiVersion: 1
datasources:
  - id: 1
    name: Prometheus
    orgId: 1
    uid: a17feb01-a0c1-432e-8ef5-7b277cb0b32b
    type: prometheus
    typeName: Prometheus
    typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg
    access: proxy
    url: http://prometheus:9090
    user: ''
    database: ''
    basicAuth: false
    isDefault: true
    jsonData:
      httpMethod: POST
    readOnly: false

The easiest way to write datasources definitions in the configuration file is to:

Login to Grafana as admin
Manually setup the datasource

Issue a GET /api/datasources request to Grafana's API to get the datasource configuration

curl -sS 'http://grafana:3000/api/datasources' -H 'Authorization: Basic YWRtaW46YWRtaW4='

Edit it as YAML
Drop the YAML definition into the provisioning/datasources directory

$ curl -sS 'http://grafana:3000/api/datasources' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
| yq -y '{apiVersion: 1, datasources: .}' - \
| tee '/etc/grafana/provisioning/datasources/default.yml'
apiVersion: 1
datasources:
  - id: 1
    uid: a17feb01-a0c1-432e-8ef5-7b277cb0b32b
    orgId: 1
    name: Prometheus
    type: prometheus
    typeName: Prometheus
    typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg
    access: proxy
    url: http://rpi4b.lan:9090
    user: ''
    database: ''
    basicAuth: false
    isDefault: true
    jsonData:
      httpMethod: POST
    readOnly: true

Dashboards

Dashboards can be automatically managed by adding one or more YAML config files in the provisioning/dashboards directory.
Each config file can contain a list of dashboards providers that load dashboards into Grafana from the local filesystem.

When Grafana starts, it will insert all dashboards available in the configured path, or update them if they are already present.
Later on it will poll that path every updateIntervalSeconds, look for updated json files and update/insert those into the database.

apiVersion: 1
providers:
  - name: dashboards
    folder: ''
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Save existing dashboards like you would for the datasources.
Save the dashboard definitions in JSON files in the path searched by the provider (e.g. /var/lib/grafana/dashboards).

curl -sS 'http://grafana:3000/api/search' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
| jq -r '.[].uid' - \
| parallel " \
    curl -sS 'http://grafana:3000/api/dashboards/uid/{}' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
    > '/var/lib/grafana/dashboards/{}.json' \
  "

Dashboards of interest

Name	Grafana ID	URLs
Node exporter full	1860	summary code
OpenWRT	11147	summary code
Prometheus	19105	summary code
Kubernetes Cluster (Prometheus)	6417	summary
Kubernetes cluster monitoring (via Prometheus)	315	summary
Nextcloud	20716	summary

Alerting

Refer alerting and Get started with Grafana Alerting.

Create a contact point if not existing already.
Create an alert rule.

APIs

Refer HTTP API reference.

Sources

All the references in the further readings section, plus the following:

12 KiB Raw Blame History