# Grafana Open-source platform for monitoring and observability. 1. [TL;DR](#tldr) 1. [Setup](#setup) 1. [Configuration](#configuration) 1. [Provisioning](#provisioning) 1. [Datasources](#datasources) 1. [Dashboards](#dashboards) 1. [Dashboards of interest](#dashboards-of-interest) 1. [Alerting](#alerting) 1. [APIs](#apis) 1. [Further readings](#further-readings) 1. [Sources](#sources) ## TL;DR Grafana needs a database to store users, dashboards, and other data. It supports `mysql`, `postgres` or `sqlite3`.
By default it uses the `sqlite3` embedded database included in the main `grafana` binary. Grafana creates a folder for each installed plugin, containing its associated files and data.
Plugin files are located by default in `$PWD/data/plugins` (`/var/lib/grafana/plugins` for `deb` or `rpm` packages).
Usage ```sh docker run -d --rm --name 'grafana-oss' -p '3000:3000' 'grafana/grafana-oss' docker run -d --rm --name 'grafana-enterprise' -p '3000:3000' -ti -entrypoint 'bash' 'grafana/grafana-enterprise' ``` ```plaintext # Return health information GET /api/health ```
Real world use cases ```sh # Export all existing dashboards by ID. curl -sS 'http://grafana:3000/api/search' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \ | jq -r '.[].uid' - \ | parallel " \ curl -sS 'http://grafana:3000/api/dashboards/uid/{}' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \ > '{}.json' \ " ```
## Setup
Kubernetes ```sh helm repo add 'grafana' 'https://grafana.github.io/helm-charts' helm -n 'monitoring' upgrade -i --create-namespace --set adminPassword='abc0123' 'grafana' 'grafana/grafana' helm -n 'monitoring' upgrade -i --create-namespace --repo 'https://grafana.github.io/helm-charts' 'grafana' 'grafana' ``` Access components: | Component | From within the cluster | | --------- | ----------------------------------------- | | Server | `grafana.monitoring.svc.cluster.local:80` | ```sh # Access the server kubectl -n 'monitoring' get secret 'grafana' -o jsonpath='{.data.admin-password}' | base64 --decode kubectl -n 'monitoring' get pods -l 'app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana' \ -o jsonpath='{.items[0].metadata.name}' \ | xargs -I '%%' kubectl -n 'monitoring' port-forward "%%" '3000' ``` Clean up: ```sh helm -n 'monitoring' delete 'grafana' kubectl delete namespace --ignore-not-found 'monitoring' ``` Access Prometheus instances in the same namespace using `http://prometheus-server`.
## Configuration Refer [Configuration file location]. Grafana searches for default settings in the `${PWD}/conf/defaults.ini` file. Do **not** change this file.
Depending on the executing OS, Grafana searches for custom configuration in the `${PWD}/conf/custom.ini` or in the `/usr/local/etc/grafana/grafana.ini` files. Specify a custom file path with the `--config` option. `deb` and `rpm` packages put the custom configuration file at `/etc/grafana/grafana.ini`, and do **not** use a separate `custom.ini` file. That path is specified in Grafana's init script using the `--config` option. Prefer using environmental variables to **override** existing options (and not to _add_ them). All letters must be **uppercase**.
Replace periods (`.`) and dashes (`-`) with underscores (_).
```ini # default section instance_name = ${HOSTNAME} [security] admin_user = admin [auth.google] client_secret = 0ldS3cretKey [plugin.grafana-image-renderer] rendering_ignore_https_errors = true [feature_toggles] enable = newNavigation ``` ```sh # export GF_${SECTION NAME}_${KEY}='value' export \ GF_DEFAULT_INSTANCE_NAME='some-instance' \ GF_SECURITY_ADMIN_USER='owner' \ GF_AUTH_GOOGLE_CLIENT_SECRET='newS3cretKey' \ GF_PLUGIN_GRAFANA_IMAGE_RENDERER_RENDERING_IGNORE_HTTPS_ERRORS=true \ GF_FEATURE_TOGGLES_ENABLE='newNavigation' ```
Grafana evaluates options containing the expression `$__{}or ${}`.
The evaluation runs the specified provider with the provided argument to get the final value of the option. Available providers are `env`, `file`, and `vault`. The `env` provider expands environment variables.
One can also use the short-hand syntax `${PORT}`.
```ini instance_name = ${HOSTNAME} [paths] logs = $__env{LOGDIR}/grafana ```
The `file` provider reads a value from a file in the filesystem.
It trims whitespace from the beginning and the end of files.
```ini [database] password = $__file{/etc/secrets/gf_sql_password} ```
The `vault` provider integrates with [Hashicorp Vault]. It is only available in Grafana Enterprise. ## Provisioning See [provision dashboards and data sources] for details. ### Datasources Data sources can be managed automatically at provisioning by adding YAML configuration files in the `provisioning/datasources` directory. Each configuration file can contain a list of `datasources` to add or update during startup.
If the data source already exists, Grafana reconfigures it to match the provisioned configuration file. Grafana also deletes the data sources listed in `deleteDatasources` before adding or updating those in the `datasources` list. ```yml --- apiVersion: 1 datasources: - id: 1 name: Prometheus orgId: 1 uid: a17feb01-a0c1-432e-8ef5-7b277cb0b32b type: prometheus typeName: Prometheus typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg access: proxy url: http://prometheus:9090 user: '' database: '' basicAuth: false isDefault: true jsonData: httpMethod: POST readOnly: false ``` The easiest way to write datasources definitions in the configuration file is to: 1. Login to Grafana as `admin` 1. Manually setup the datasource 1. Issue a `GET /api/datasources` request to Grafana's API to get the datasource configuration ```sh curl -sS 'http://grafana:3000/api/datasources' -H 'Authorization: Basic YWRtaW46YWRtaW4=' ``` 1. Edit it as YAML 1. Drop the YAML definition into the `provisioning/datasources` directory ```sh $ curl -sS 'http://grafana:3000/api/datasources' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \ | yq -y '{apiVersion: 1, datasources: .}' - \ | tee '/etc/grafana/provisioning/datasources/default.yml' apiVersion: 1 datasources: - id: 1 uid: a17feb01-a0c1-432e-8ef5-7b277cb0b32b orgId: 1 name: Prometheus type: prometheus typeName: Prometheus typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg access: proxy url: http://rpi4b.lan:9090 user: '' database: '' basicAuth: false isDefault: true jsonData: httpMethod: POST readOnly: true ``` ### Dashboards Dashboards can be automatically managed by adding one or more YAML config files in the `provisioning/dashboards` directory.
Each config file can contain a list of dashboards `providers` that load dashboards into Grafana from the local filesystem. When Grafana starts, it will insert all dashboards available in the configured path, or update them if they are already present.
Later on it will poll that path every `updateIntervalSeconds`, look for updated json files and update/insert those into the database. ```yml apiVersion: 1 providers: - name: dashboards folder: '' disableDeletion: false updateIntervalSeconds: 10 allowUiUpdates: false options: path: /var/lib/grafana/dashboards foldersFromFilesStructure: true ``` Save existing dashboards like [you would for the datasources][datasources provisioning].
Save the dashboard definitions in JSON files in the path searched by the provider (e.g. `/var/lib/grafana/dashboards`). ```sh curl -sS 'http://grafana:3000/api/search' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \ | jq -r '.[].uid' - \ | parallel " \ curl -sS 'http://grafana:3000/api/dashboards/uid/{}' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \ > '/var/lib/grafana/dashboards/{}.json' \ " ``` ## Dashboards of interest | Name | Grafana ID | URLs | | ---------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Node exporter full | 1860 | [summary](https://grafana.com/grafana/dashboards/1860-node-exporter-full/)
[code](https://raw.githubusercontent.com/rfmoz/grafana-dashboards/master/prometheus/node-exporter-full.json) | | OpenWRT | 11147 | [summary](https://grafana.com/grafana/dashboards/11147-openwrt/)
[code](https://raw.githubusercontent.com/try2codesecure/grafana_dashboards/refs/heads/master/OpenWRT/openwrt.json) | | Prometheus | 19105 | [summary](https://grafana.com/grafana/dashboards/19105-prometheus/)
[code](https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/refs/heads/master/dashboards/k8s-addons-prometheus.json) | | Kubernetes Cluster (Prometheus) | 6417 | [summary](https://grafana.com/grafana/dashboards/6417-kubernetes-cluster-prometheus/) | | Kubernetes cluster monitoring (via Prometheus) | 315 | [summary](https://grafana.com/grafana/dashboards/315-kubernetes-cluster-monitoring-via-prometheus/) | | Nextcloud | 20716 | [summary](https://grafana.com/grafana/dashboards/20716-nextcloud/) | ## Alerting Refer [alerting] and [Get started with Grafana Alerting]. 1. Create a contact point if not existing already. 1. Create an alert rule. ## APIs Refer [HTTP API reference]. ## Further readings - [Website] - [Github] - [Documentation] - [HTTP API reference] - [Prometheus] - [docker compositions/monitoring] - Official [helm chart] - [Loki] - [Get started with Grafana Alerting] ## Sources All the references in the [further readings] section, plus the following: - [Provisioning] - [Provision dashboards and data sources] - [Data source on startup] - [Set up prometheus and ingress on kubernetes] - [How to integrate Prometheus and Grafana on Kubernetes using Helm] - [Alerting] [datasources provisioning]: #datasources [further readings]: #further-readings [HashiCorp Vault]: hashicorp%20vault.md [loki]: loki.md [prometheus]: prometheus/README.md [docker compositions/monitoring]: ../docker%20compositions/monitoring/README.md [alerting]: https://grafana.com/docs/grafana/latest/alerting/ [data source on startup]: https://community.grafana.com/t/data-source-on-startup/8618/2 [documentation]: https://grafana.com/docs/grafana/latest/ [get started with grafana alerting]: https://grafana.com/tutorials/alerting-get-started/ [github]: https://github.com/grafana/grafana [helm chart]: https://github.com/grafana/helm-charts/tree/main/charts/grafana [http api reference]: https://grafana.com/docs/grafana/latest/developers/http_api/ [provision dashboards and data sources]: https://grafana.com/tutorials/provision-dashboards-and-data-sources/ [provisioning]: https://grafana.com/docs/grafana/latest/administration/provisioning/ [website]: https://grafana.com [Configuration file location]: https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#configuration-file-location [how to integrate prometheus and grafana on kubernetes using helm]: https://semaphoreci.com/blog/prometheus-grafana-kubernetes-helm [set up prometheus and ingress on kubernetes]: https://blog.gojekengineering.com/diy-how-to-set-up-prometheus-and-ingress-on-kubernetes-d395248e2ba