brandon/oam
1
0
Fork 0
mirror of https://gitea.com/mcereda/oam.git synced 2026-02-09 05:44:23 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
cba9d9193e64fb9e2c4aef864bf19a220dc140c0
oam/knowledge base/grafana.md
Michele Cereda 1f2259ad91 feat(kb): introduce secrets management
2025-08-02 22:47:51 +02:00

363 lines
12 KiB
Markdown
Raw Blame History

# Grafana
Open-source platform for monitoring and observability.
1. [TL;DR](#tldr)
1. [Setup](#setup)
1. [Configuration](#configuration)
1. [Provisioning](#provisioning)
1. [Datasources](#datasources)
1. [Dashboards](#dashboards)
1. [Dashboards of interest](#dashboards-of-interest)
1. [Alerting](#alerting)
1. [APIs](#apis)
1. [Further readings](#further-readings)
1. [Sources](#sources)
## TL;DR
Grafana needs a database to store users, dashboards, and other data. It supports `mysql`, `postgres` or `sqlite3`.<br/>
By default it uses the `sqlite3` embedded database included in the main `grafana` binary.
Grafana creates a folder for each installed plugin, containing its associated files and data.<br/>
Plugin files are located by default in `$PWD/data/plugins` (`/var/lib/grafana/plugins` for `deb` or `rpm`
packages).
<details>
<summary>Usage</summary>
```sh
docker run -d --rm --name 'grafana-oss' -p '3000:3000' 'grafana/grafana-oss'
docker run -d --rm --name 'grafana-enterprise' -p '3000:3000' -ti -entrypoint 'bash' 'grafana/grafana-enterprise'
```
```plaintext
# Return health information
GET /api/health
```
</details>
<details>
<summary>Real world use cases</summary>
```sh
# Export all existing dashboards by ID.
curl -sS 'http://grafana:3000/api/search' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
| jq -r '.[].uid' - \
| parallel " \
curl -sS 'http://grafana:3000/api/dashboards/uid/{}' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
> '{}.json' \
"
```
</details>
## Setup
<details>
<summary>Kubernetes</summary>
```sh
helm repo add 'grafana' 'https://grafana.github.io/helm-charts'
helm -n 'monitoring' upgrade -i --create-namespace --set adminPassword='abc0123' 'grafana' 'grafana/grafana'
helm -n 'monitoring' upgrade -i --create-namespace --repo 'https://grafana.github.io/helm-charts' 'grafana' 'grafana'
```
Access components:
| Component | From within the cluster |
| --------- | ----------------------------------------- |
| Server | `grafana.monitoring.svc.cluster.local:80` |
```sh
# Access the server
kubectl -n 'monitoring' get secret 'grafana' -o jsonpath='{.data.admin-password}' | base64 --decode
kubectl -n 'monitoring' get pods -l 'app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana' \
-o jsonpath='{.items[0].metadata.name}' \
| xargs -I '%%' kubectl -n 'monitoring' port-forward "%%" '3000'
```
Clean up:
```sh
helm -n 'monitoring' delete 'grafana'
kubectl delete namespace --ignore-not-found 'monitoring'
```
Access Prometheus instances in the same namespace using `http://prometheus-server`.
</details>
## Configuration
Refer [Configuration file location].
Grafana searches for default settings in the `${PWD}/conf/defaults.ini` file. Do **not** change this file.<br/>
Depending on the executing OS, Grafana searches for custom configuration in the `${PWD}/conf/custom.ini` or in the
`/usr/local/etc/grafana/grafana.ini` files. Specify a custom file path with the `--config` option.
`deb` and `rpm` packages put the custom configuration file at `/etc/grafana/grafana.ini`, and do **not** use a separate
`custom.ini` file. That path is specified in Grafana's init script using the `--config` option.
Prefer using environmental variables to **override** existing options (and not to _add_ them).
All letters must be **uppercase**.<br/>
Replace periods (`.`) and dashes (`-`) with underscores (_).
<details style='padding: 0 0 1rem 1rem'>
```ini
# default section
instance_name = ${HOSTNAME}
[security]
admin_user = admin
[auth.google]
client_secret = 0ldS3cretKey
[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true
[feature_toggles]
enable = newNavigation
```
```sh
# export GF_${SECTION NAME}_${KEY}='value'
export \
GF_DEFAULT_INSTANCE_NAME='some-instance' \
GF_SECURITY_ADMIN_USER='owner' \
GF_AUTH_GOOGLE_CLIENT_SECRET='newS3cretKey' \
GF_PLUGIN_GRAFANA_IMAGE_RENDERER_RENDERING_IGNORE_HTTPS_ERRORS=true \
GF_FEATURE_TOGGLES_ENABLE='newNavigation'
```
</details>
Grafana evaluates options containing the expression `$__<PROVIDER>{<ARGUMENT>}or ${<ENVIRONMENT VARIABLE>}`.<br/>
The evaluation runs the specified provider with the provided argument to get the final value of the option.
Available providers are `env`, `file`, and `vault`.
The `env` provider expands environment variables.<br/>
One can also use the short-hand syntax `${PORT}`.
<details style='padding: 0 0 1rem 1rem'>
```ini
instance_name = ${HOSTNAME}
[paths]
logs = $__env{LOGDIR}/grafana
```
</details>
The `file` provider reads a value from a file in the filesystem.<br/>
It trims whitespace from the beginning and the end of files.
<details style='padding: 0 0 1rem 1rem'>
```ini
[database]
password = $__file{/etc/secrets/gf_sql_password}
```
</details>
The `vault` provider integrates with [Hashicorp Vault].
It is only available in Grafana Enterprise.
## Provisioning
See [provision dashboards and data sources] for details.
### Datasources
Data sources can be managed automatically at provisioning by adding YAML configuration files in the
`provisioning/datasources` directory.
Each configuration file can contain a list of `datasources` to add or update during startup.<br/>
If the data source already exists, Grafana reconfigures it to match the provisioned configuration file.
Grafana also deletes the data sources listed in `deleteDatasources` before adding or updating those in the `datasources`
list.
```yml
---
apiVersion: 1
datasources:
- id: 1
name: Prometheus
orgId: 1
uid: a17feb01-a0c1-432e-8ef5-7b277cb0b32b
type: prometheus
typeName: Prometheus
typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg
access: proxy
url: http://prometheus:9090
user: ''
database: ''
basicAuth: false
isDefault: true
jsonData:
httpMethod: POST
readOnly: false
```
The easiest way to write datasources definitions in the configuration file is to:
1. Login to Grafana as `admin`
1. Manually setup the datasource
1. Issue a `GET /api/datasources` request to Grafana's API to get the datasource configuration
```sh
curl -sS 'http://grafana:3000/api/datasources' -H 'Authorization: Basic YWRtaW46YWRtaW4='
```
1. Edit it as YAML
1. Drop the YAML definition into the `provisioning/datasources` directory
```sh
$ curl -sS 'http://grafana:3000/api/datasources' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
| yq -y '{apiVersion: 1, datasources: .}' - \
| tee '/etc/grafana/provisioning/datasources/default.yml'
apiVersion: 1
datasources:
- id: 1
uid: a17feb01-a0c1-432e-8ef5-7b277cb0b32b
orgId: 1
name: Prometheus
type: prometheus
typeName: Prometheus
typeLogoUrl: public/app/plugins/datasource/prometheus/img/prometheus_logo.svg
access: proxy
url: http://rpi4b.lan:9090
user: ''
database: ''
basicAuth: false
isDefault: true
jsonData:
httpMethod: POST
readOnly: true
```
### Dashboards
Dashboards can be automatically managed by adding one or more YAML config files in the `provisioning/dashboards`
directory.<br/>
Each config file can contain a list of dashboards `providers` that load dashboards into Grafana from the local
filesystem.
When Grafana starts, it will insert all dashboards available in the configured path, or update them if they are already
present.<br/>
Later on it will poll that path every `updateIntervalSeconds`, look for updated json files and update/insert those into
the database.
```yml
apiVersion: 1
providers:
- name: dashboards
folder: ''
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: false
options:
path: /var/lib/grafana/dashboards
foldersFromFilesStructure: true
```
Save existing dashboards like [you would for the datasources][datasources provisioning].<br/>
Save the dashboard definitions in JSON files in the path searched by the provider (e.g. `/var/lib/grafana/dashboards`).
```sh
curl -sS 'http://grafana:3000/api/search' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
| jq -r '.[].uid' - \
| parallel " \
curl -sS 'http://grafana:3000/api/dashboards/uid/{}' -H 'Authorization: Basic YWRtaW46YWRtaW4=' \
> '/var/lib/grafana/dashboards/{}.json' \
"
```
## Dashboards of interest
| Name | Grafana ID | URLs |
| ---------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Node exporter full | 1860 | [summary](https://grafana.com/grafana/dashboards/1860-node-exporter-full/)<br/>[code](https://raw.githubusercontent.com/rfmoz/grafana-dashboards/master/prometheus/node-exporter-full.json) |
| OpenWRT | 11147 | [summary](https://grafana.com/grafana/dashboards/11147-openwrt/)<br/>[code](https://raw.githubusercontent.com/try2codesecure/grafana_dashboards/refs/heads/master/OpenWRT/openwrt.json) |
| Prometheus | 19105 | [summary](https://grafana.com/grafana/dashboards/19105-prometheus/)<br/>[code](https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/refs/heads/master/dashboards/k8s-addons-prometheus.json) |
| Kubernetes Cluster (Prometheus) | 6417 | [summary](https://grafana.com/grafana/dashboards/6417-kubernetes-cluster-prometheus/) |
| Kubernetes cluster monitoring (via Prometheus) | 315 | [summary](https://grafana.com/grafana/dashboards/315-kubernetes-cluster-monitoring-via-prometheus/) |
| Nextcloud | 20716 | [summary](https://grafana.com/grafana/dashboards/20716-nextcloud/) |
## Alerting
Refer [alerting] and [Get started with Grafana Alerting].
1. Create a contact point if not existing already.
1. Create an alert rule.
## APIs
Refer [HTTP API reference].
## Further readings
- [Website]
- [Github]
- [Documentation]
- [HTTP API reference]
- [Prometheus]
- [docker compositions/monitoring]
- Official [helm chart]
- [Loki]
- [Get started with Grafana Alerting]
## Sources
All the references in the [further readings] section, plus the following:
- [Provisioning]
- [Provision dashboards and data sources]
- [Data source on startup]
- [Set up prometheus and ingress on kubernetes]
- [How to integrate Prometheus and Grafana on Kubernetes using Helm]
- [Alerting]
<!--
Reference
═╬═Time══
-->
<!-- In-article sections -->
[datasources provisioning]: #datasources
[further readings]: #further-readings
<!-- Knowledge base -->
[HashiCorp Vault]: hashicorp%20vault.md
[loki]: loki.md
[prometheus]: prometheus/README.md
<!-- Files -->
[docker compositions/monitoring]: ../docker%20compositions/monitoring/README.md
<!-- Upstream -->
[alerting]: https://grafana.com/docs/grafana/latest/alerting/
[data source on startup]: https://community.grafana.com/t/data-source-on-startup/8618/2
[documentation]: https://grafana.com/docs/grafana/latest/
[get started with grafana alerting]: https://grafana.com/tutorials/alerting-get-started/
[github]: https://github.com/grafana/grafana
[helm chart]: https://github.com/grafana/helm-charts/tree/main/charts/grafana
[http api reference]: https://grafana.com/docs/grafana/latest/developers/http_api/
[provision dashboards and data sources]: https://grafana.com/tutorials/provision-dashboards-and-data-sources/
[provisioning]: https://grafana.com/docs/grafana/latest/administration/provisioning/
[website]: https://grafana.com
[Configuration file location]: https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#configuration-file-location
<!-- Others -->
[how to integrate prometheus and grafana on kubernetes using helm]: https://semaphoreci.com/blog/prometheus-grafana-kubernetes-helm
[set up prometheus and ingress on kubernetes]: https://blog.gojekengineering.com/diy-how-to-set-up-prometheus-and-ingress-on-kubernetes-d395248e2ba
Reference in New Issue View Git Blame Copy Permalink