Grafana 8.* plugin installation failure

23 Jan 2022

Grafana is one of the best metrics visualization tools so far that I have come across. Plug and play are what I like more about it.
Migrations are always hard. Grafana migration from 7.5.* to 8.* is not a big surprise as well. The team though it will be just version update and plugins setup. But wait surprises are always there. Based on the doc, we just set the special variable GF_INSTALL_PLUGINS that's it. And then we were getting

Error: ✗ failed to download plugin archive: <?xml version='1.0' encoding='UTF-8'?><Error><Code>SecurityPolicyViolated</Code><Message>Request violates VPC Service Controls.</Message><Details>Request is prohibited by organization's policy. vpcServiceControlsUniqueIdentifier: xXLILqaJM0zPkKkvM8J23tg9Q8BZBEcsy6wEddm9Lwvf3SdPoYpCfQ</Details></Error> (Grafana v8.3.3 linux-amd64)
Seems like they have moved the plugins to GCP bucket with restricted policies So I come up with a manual installation approach in Dockerfile as below:
# Dockerfile 
FROM grafana/grafana:8.3.3
RUN grafana-cli plugins install fetzerch-sunandmoon-datasource 
RUN grafana-cli plugins install grafana-piechart-panel 
RUN grafana-cli plugins install mtanda-heatmap-epoch-panel 
RUN grafana-cli plugins install briangann-datatable-panel 
RUN grafana-cli plugins install briangann-gauge-panel 
RUN grafana-cli plugins install fzakaria-simple-annotations-datasource 
RUN grafana-cli plugins install grafana-simple-json-datasource 
RUN grafana-cli plugins install mtanda-histogram-panel 
RUN grafana-cli plugins install michaeldmoore-annunciator-panel 
RUN grafana-cli plugins install flant-statusmap-panel 
RUN grafana-cli plugins install simpod-json-datasource 
RUN grafana-cli plugins install grafana-kubernetes-app

Everything went smoother but still, plugins were not shown. No wonder there were missing plugins in the default grafana plugin directory
✗ kubect  exec -it grafana-cbdb58d99-ktpxf sh 
/usr/share/grafana # ls -al /var/lib/grafana/plugins 
/usr/share/grafana #
Looked at the deployment and it was volume mount which was overriding the /var/lib/pluginSo now plugin directory must be moved to custom one and set to /var/lib/grafana-plugins
# Dockerfile 
FROM grafana/grafana:8.3.3
RUN mkdir -p /var/lib/grafana-plugins
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install fetzerch-sunandmoon-datasource 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install grafana-piechart-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install mtanda-heatmap-epoch-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install briangann-datatable-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install briangann-gauge-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install fzakaria-simple-annotations-datasource 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install grafana-simple-json-datasource 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install mtanda-histogram-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install michaeldmoore-annunciator-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install flant-statusmap-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install simpod-json-datasource 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install grafana-kubernetes-app

But still, the plugin wasn't shown in Grafana UI. It was a permission issue on the plugin directory. Also, make sure you have allow_loading_unsigned_plugins variable updated in /etc/grafana/grafana.iniBelow configuration in Dockerfile finally worked.
# Dockerfile 
FROM grafana/grafana:8.3.3 
USER root
RUN mkdir -p /var/lib/grafana-plugins
RUN chown 472:0 /var/lib/grafana-plugins
USER grafana RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install fetzerch-sunandmoon-datasource 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install grafana-piechart-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install mtanda-heatmap-epoch-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install briangann-datatable-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install briangann-gauge-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install fzakaria-simple-annotations-datasource 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install grafana-simple-json-datasource 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install mtanda-histogram-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install michaeldmoore-annunciator-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install flant-statusmap-panel 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install simpod-json-datasource 
RUN grafana-cli --pluginsDir=/var/lib/grafana-plugins plugins install grafana-kubernetes-app

Just want to mention here, whenever you are stuck with some solution try to find out alternatives. Don't waste more time thinking of the same solution. It will widespread your views.
Feel free to tag on twitter if you have queries or comments.