Better file provisioning for Grafana dashboards

Extract panels from Grafana dashboards into separate smaller JSON files and have a nice history of changes in git + bonus: add ‘Show YAML’ button for alert rules

Dec 18, 2022 by Mikolaj Gasior

Intro

Grafana is a great tool for monitoring and observability. It’s very easy to install, configure and use it. You can get essential monitoring and alerting quite quickly.

However, there are small things from infrastructure-as-a-code perspective that make maintaining Grafana configuration slightly less comfortable. One of these things are dashboards which are represented by a JSON model.

In our case here, we wanted to store whole Grafana configuration (dashboards, contact points, alerts etc.) in GitHub repository. And one of the things we spotted is the fact that the difference between two versions of dashboard JSON model is completely unreadable. Dashboard contains panels, loads of them. Having them in one big JSON file is really difficult to maintain. A simple line comparison, like the one there is in git between commits, doesn’t work with structure like JSON. It is easy to write a simple script to compare two JSON files but what we were really looking for is to have a nice history of git commits.

Split the dashboard JSON

So, we came up with an idea of splitting the big dashboard JSON file into:

1) JSON file that contains dashboard details without the panels (so just like original but with panels: [])
2) separate JSON files for each of the panels in a panels directory, eg. panels/008-title-of-the-panel.yaml.

See the listing of files below to get a picture of that.

nameless@computer dashboard-name % find . -type f   
./panels/001-title-of-some-panel.json
./panels/002-another-panel.json
./panels/003-row-and-its-title.json
./panels/004-yet-another-panel.json
./dashboard.json

With a structure like that, it is so much easier to read the history of changes. Instead of going through one big file (where the diff does not make sense when you change order of items in the panels list), you can go through specific smaller files.

Implementation

In terms of implementing it there are two parts of it. First one is to merge all these files into one dashboard JSON file when it is imported by Grafana. And the second part is to modify the UI to show us the files we should put in the repository. When JSON Model link is clicked in dashboard settings you get the whole JSON file.

Merge new structure into one JSON file

In our case, Grafana is deployed on Kubernetes cluster with Helm chart and we have a dashboard sidecar running, which deals with the provisioning. Dashboard JSON files are stored in ConfigMap and the sidecar keeps them in-sync with Grafana database. There is more information about that in Grafana docs or in the charts repository.

See below configuration in the values file of the chart. That’s how the sidecar can be added. Again, check the chart docs to get information on some of the options listed below.

  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      folderAnnotation: grafana_dashboard_folder
      env_directory: testenv
      provider:
        disableDelete: false
        allowUiUpdates: false
        foldersFromFilesStructure: true

Above only adds an additional container to the pod that will seek for ConfigMap resources that are labeled grafana_dashboard. Now, we need to add these by creating a directory dashboards with JSON files in it and adding a template in the templates directory.

Our templates/configmap-dashboards.yaml has a the following block. It will read each directory, get dashboard.json, get JSON files in panels sub-directory and merge it (actually replace panels: [] string) all together into one JSON.

{{- $files := .Files.Glob (print "dashboards/" .Values.grafana.sidecar.dashboards.env_directory "/**" | replace "//" "/") }}
{{- if $files }}
{{- range $path, $bytes := $files }}
{{- if and (regexMatch "dashboard\\.json$" $path) (not (regexMatch "general\\-alerting" $path)) }}
{{- $dashboardName := base (dir $path)}}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ printf "grafana-dash-%s" $dashboardName | trunc 63 | trimSuffix "-" }}
  namespace: {{ $.Release.Namespace }}
  labels:
    {{- if $.Values.grafana.sidecar.dashboards.label }}
    {{ $.Values.grafana.sidecar.dashboards.label }}: "1"
    {{- end }}
    app: {{ $.Chart.Name }}-grafana
data:
  {{- $panelFiles := $.Files.Glob (print "dashboards/" $.Values.grafana.sidecar.dashboards.env_directory "/" $dashboardName "/panels/*.json" | replace "//" "/") }}
  {{- $panelsContent := "" }}
  {{- if $panelFiles }}
    {{- range $panelPath, $panelBytes := $panelFiles }}
      {{- if ne $panelsContent "" }}
        {{- $panelsContent = (print $panelsContent "," ($.Files.Get $panelPath)) }}
      {{- else }}
        {{- $panelsContent = (print $panelsContent ($.Files.Get $panelPath)) }}
      {{- end }}
    {{- end }}
  {{- end }}
  {{ $dashboardName }}.json: {{ $.Files.Get $path | replace "\"panels\": []" (print "\"panels\": [ " $panelsContent " ]") | toJson }}
{{- end }}
{{- end }}
{{- end }}

Export dashboards

Unfortunately, Grafana UI provided JSON model that contains everything. Hence, all the panels (in the panels element) must be extracted to separate files, and - as seen above in the template snippet - the panels element must be changed to an empty array for the string replace to take place.

"panels": []

Modified UI

Frontend is written in React and it can be modified to display each panel in a separate <textarea> element, on the JSON Model page. We had a patch with instructions doing just that but it was made for an old version so it seems better to not share it.

See below screenshot on how the JSON Model setting of the dashboard can look like.

Grafana JSON Model setting for dashboards

Result

We have few dashboards with plenty of panels in it, some of them being shared between environments, and some being specific for an environment (needs tweaking the configmap-dashboards.yaml chart template mentioned earlier + see the env_directory value). Everything is kept in git repository, we have transparent history of changes and it is quite simple to manage things. Also, with the big warning showing up next to any dashboard (or alert) not provisioned from the file, users do not forget to update JSON (and YAML) files in the repository.