Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Grafana setup ignoring defined certificates which cause errors in dashboard

See original GitHub issue

Bug Report

What happened: ceph-ansible is skipping the copy of defined certificates which causes errors in performance dashboards. The http_addr setting is also an issue, since it picks the IP of the server, but certificates are issued for a domain. So if I head to performance dashboards with generated Grafana certs, I get the error: Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT. If i replace the certificates manually, most of dashboards are working, but

Dashboard continuously pops the red error “500 - Internal Server Error”
Pools performance dashboard is broken and I additionally get “TypeError: l.c[t.type] is undefined” error
ceph-mgr log still reports an error:

SSLError: HTTPSConnectionPool(host='10.1.40.10', port=3000): Max retries exceeded with url: /api/dashboards/uid/z99hzWtmk (Caused by SSLError(CertificateError("hostname '10.1.40.10' doesn't match '*.dc1.redacted.com'",),))

Setting the http_addr manually to domain also doesn’t help =/

How to reproduce it (minimal and precise): Just run the rolling update playbook with dashboard settings and certificates defined.

inventory

[mons]
ceph1-40-10.dc1.redacted.com
ceph1-40-11.dc1.redacted.com
ceph1-40-12.dc1.redacted.com

[osds]
ceph1-40-10.dc1.redacted.com
ceph1-40-11.dc1.redacted.com
ceph1-40-12.dc1.redacted.com

[mgrs]
ceph1-40-10.dc1.redacted.com
ceph1-40-11.dc1.redacted.com
ceph1-40-12.dc1.redacted.com

[rgws]
ceph1-40-10.dc1.redacted.com
ceph1-40-11.dc1.redacted.com
ceph1-40-12.dc1.redacted.com

[nfss]
ceph1-40-10.dc1.redacted.com
ceph1-40-11.dc1.redacted.com
ceph1-40-12.dc1.redacted.com

[grafana-server]
ceph1-40-10.dc1.redacted.com
ceph1-40-11.dc1.redacted.com
ceph1-40-12.dc1.redacted.com

[ceph:children]
mons
osds
mgrs
rgws
nfss
grafana-server

snippet from dashboard config in group vars (everything else is default)

dashboard_enabled: True
dashboard_protocol: https
dashboard_port: 8443
dashboard_admin_user: admin
dashboard_admin_password: *****************************
dashboard_crt: '/etc/ssl/certs/companycert.pem'
dashboard_key: '/etc/ssl/private/companycert-key.pem'
node_exporter_port: 9100
grafana_admin_user: admin
grafana_admin_password: **********************
grafana_crt: '/etc/ssl/certs/companycert.pem'
grafana_key: '/etc/ssl/private/companycert-key.pem'

How the grafana.ini looks like after deployment

[users]
default_theme = light

[auth.anonymous]
# enable anonymous access
enabled = true

# specify organization name that should be used for unauthenticated users
org_name = Main Org.

# specify role for unauthenticated users
org_role = Viewer

[server]
cert_file = /etc/grafana/ceph-dashboard.crt      <--- those were generated by ansible-run
cert_key = /etc/grafana/ceph-dashboard.key    <--- those were generated by ansible-run
domain = ceph1-40-10.dc1.redacted.com
protocol = https
http_port = 3000
http_addr = 10.1.40.10

[security]
admin_user = admin
admin_password = *********************
allow_embedding = True

grafana snippet from ansible log

2020-02-12 12:17:36,082 p=24266 u=root |  TASK [ceph-grafana : ship systemd service] *************************************************************************************************
2020-02-12 12:17:36,083 p=24266 u=root |  Wednesday 12 February 2020  12:17:36 +0100 (0:00:00.731)       0:30:21.922 **** 
2020-02-12 12:17:36,558 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:36,704 p=24266 u=root |  TASK [ceph-grafana : start the grafana-server service] *************************************************************************************
2020-02-12 12:17:36,704 p=24266 u=root |  Wednesday 12 February 2020  12:17:36 +0100 (0:00:00.621)       0:30:22.543 **** 
2020-02-12 12:17:37,201 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:37,352 p=24266 u=root |  TASK [ceph-grafana : include configure_grafana.yml] ****************************************************************************************
2020-02-12 12:17:37,352 p=24266 u=root |  Wednesday 12 February 2020  12:17:37 +0100 (0:00:00.647)       0:30:23.191 **** 
2020-02-12 12:17:37,649 p=24266 u=root |  included: /etc/ansible/ceph-ansible/roles/ceph-grafana/tasks/configure_grafana.yml for ceph1-40-10.dc1.redacted.com
2020-02-12 12:17:37,802 p=24266 u=root |  TASK [ceph-grafana : install ceph-grafana-dashboards package on RedHat or SUSE] ************************************************************
2020-02-12 12:17:37,803 p=24266 u=root |  Wednesday 12 February 2020  12:17:37 +0100 (0:00:00.450)       0:30:23.642 **** 
2020-02-12 12:17:37,909 p=24266 u=root |  skipping: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:38,058 p=24266 u=root |  TASK [ceph-grafana : make sure grafana is down] ********************************************************************************************
2020-02-12 12:17:38,059 p=24266 u=root |  Wednesday 12 February 2020  12:17:38 +0100 (0:00:00.255)       0:30:23.898 **** 
2020-02-12 12:17:38,756 p=24266 u=root |  changed: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:38,914 p=24266 u=root |  TASK [ceph-grafana : wait for grafana to be stopped] ***************************************************************************************
2020-02-12 12:17:38,914 p=24266 u=root |  Wednesday 12 February 2020  12:17:38 +0100 (0:00:00.855)       0:30:24.753 **** 
2020-02-12 12:17:39,612 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:39,703 p=24266 u=root |  TASK [ceph-grafana : make sure grafana configuration directories exist] ********************************************************************
2020-02-12 12:17:39,703 p=24266 u=root |  Wednesday 12 February 2020  12:17:39 +0100 (0:00:00.789)       0:30:25.542 **** 
2020-02-12 12:17:39,923 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=/etc/grafana/dashboards/ceph-dashboard)
2020-02-12 12:17:40,154 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=/etc/grafana/provisioning/datasources)
2020-02-12 12:17:40,406 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=/etc/grafana/provisioning/dashboards)
2020-02-12 12:17:40,665 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=/etc/grafana/provisioning/notifiers)
2020-02-12 12:17:40,823 p=24266 u=root |  TASK [ceph-grafana : download ceph grafana dashboards] *************************************************************************************
2020-02-12 12:17:40,823 p=24266 u=root |  Wednesday 12 February 2020  12:17:40 +0100 (0:00:01.119)       0:30:26.662 **** 
2020-02-12 12:17:42,240 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=ceph-cluster.json)
2020-02-12 12:17:42,992 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=cephfs-overview.json)
2020-02-12 12:17:43,830 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=host-details.json)
2020-02-12 12:17:44,611 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=hosts-overview.json)
2020-02-12 12:17:45,384 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=osd-device-details.json)
2020-02-12 12:17:46,332 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=osds-overview.json)
2020-02-12 12:17:47,209 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=pool-detail.json)
2020-02-12 12:17:47,988 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=pool-overview.json)
2020-02-12 12:17:48,784 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=radosgw-detail.json)
2020-02-12 12:17:49,598 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=radosgw-overview.json)
2020-02-12 12:17:50,381 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com] => (item=rbd-overview.json)
2020-02-12 12:17:50,562 p=24266 u=root |  TASK [ceph-grafana : write grafana.ini] ****************************************************************************************************
2020-02-12 12:17:50,563 p=24266 u=root |  Wednesday 12 February 2020  12:17:50 +0100 (0:00:09.739)       0:30:36.402 **** 
2020-02-12 12:17:51,136 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:51,290 p=24266 u=root |  TASK [ceph-grafana : write datasources provisioning config file] ***************************************************************************
2020-02-12 12:17:51,290 p=24266 u=root |  Wednesday 12 February 2020  12:17:51 +0100 (0:00:00.727)       0:30:37.129 **** 
2020-02-12 12:17:51,780 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:51,932 p=24266 u=root |  TASK [ceph-grafana : Write dashboards provisioning config file] ****************************************************************************
2020-02-12 12:17:51,932 p=24266 u=root |  Wednesday 12 February 2020  12:17:51 +0100 (0:00:00.641)       0:30:37.771 **** 
2020-02-12 12:17:52,424 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:52,579 p=24266 u=root |  TASK [ceph-grafana : copy grafana SSL certificate file] ************************************************************************************
2020-02-12 12:17:52,580 p=24266 u=root |  Wednesday 12 February 2020  12:17:52 +0100 (0:00:00.647)       0:30:38.419 **** 
2020-02-12 12:17:52,672 p=24266 u=root |  skipping: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:52,822 p=24266 u=root |  TASK [ceph-grafana : copy grafana SSL certificate key] *************************************************************************************
2020-02-12 12:17:52,822 p=24266 u=root |  Wednesday 12 February 2020  12:17:52 +0100 (0:00:00.242)       0:30:38.661 **** 
2020-02-12 12:17:52,915 p=24266 u=root |  skipping: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:53,061 p=24266 u=root |  TASK [ceph-grafana : generate a Self Signed OpenSSL certificate for dashboard] *************************************************************
2020-02-12 12:17:53,062 p=24266 u=root |  Wednesday 12 February 2020  12:17:53 +0100 (0:00:00.239)       0:30:38.901 **** 
2020-02-12 12:17:53,302 p=24266 u=root |  changed: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:53,452 p=24266 u=root |  TASK [ceph-grafana : enable and start grafana] *********************************************************************************************
2020-02-12 12:17:53,452 p=24266 u=root |  Wednesday 12 February 2020  12:17:53 +0100 (0:00:00.390)       0:30:39.291 **** 
2020-02-12 12:17:53,946 p=24266 u=root |  changed: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:54,095 p=24266 u=root |  TASK [ceph-grafana : wait for grafana to start] ********************************************************************************************
2020-02-12 12:17:54,095 p=24266 u=root |  Wednesday 12 February 2020  12:17:54 +0100 (0:00:00.642)       0:30:39.934 **** 
2020-02-12 12:17:58,350 p=24266 u=root |  ok: [ceph1-40-10.dc1.redacted.com]
2020-02-12 12:17:58,509 p=24266 u=root |  TASK [set ceph grafana install 'Complete'] *************************************************************************************************

Additional informations:

Copying the same certificate for dashboard works fine, so it’s not an issue with certificates itself.

Environment:

OS (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS
Kernel (e.g. uname -a): Linux ceph1-40-10 5.0.0-16-generic
Docker version if applicable (e.g. docker version): 18.09.7
Ansible version (e.g. ansible-playbook --version): ansible-playbook 2.8.6
ceph-ansible version (e.g. git head or tag or stable branch): stable-4.0
Ceph version (e.g. ceph -v): ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)

Issue Analytics

State:
Created 4 years ago
Comments:10 (6 by maintainers)

Top GitHub Comments

1reaction

dsavineaucommented, Feb 12, 2020

On Centos 7 and Nautilus 14.2.7, the command is present

@styleart Ok I’ve probably messed up with my different ceph environments because my previous comment is wrong

In fact the set-grafana-api-ssl-verify has been added to nautilus but after 14.2.7 [1] That’s why we can see it in [2] but not in [3]

So the command isn’t present in 14.2.7 (it will be present probably in 14.2.8)

And I was looking at the ceph dashboard documentation from master [4]

[1] https://github.com/ceph/ceph/commit/a83e839b215f3ee89dfc6b249d4bd1db3798068d [2] https://github.com/ceph/ceph/blob/nautilus/src/pybind/mgr/dashboard/settings.py#L42 [3] https://github.com/ceph/ceph/blob/v14.2.7/src/pybind/mgr/dashboard/settings.py#L38-L42 [4] https://docs.ceph.com/docs/master/mgr/dashboard/#enabling-the-embedding-of-grafana-dashboards

0reactions

styleartcommented, Feb 13, 2020

@dsavineau aaah, alright, that makes sense! Well, then I’ll wait until the 14.2.8 release and try out if it works. Btw, the command is also present in current nautilus dashboard docs: https://docs.ceph.com/docs/nautilus/mgr/dashboard/

Thank you for the awesome feedback! 😃

Top Results From Across the Web

Configure Grafana | Grafana documentation

Otherwise, add a configuration file named custom.ini to the conf folder to override the settings defined in conf/defaults.ini .

Provision Grafana | Grafana documentation

Describes provisioning settings for Grafana using configuration files. ... the dashboard's definition files just the dashboard provisioning configuration.

Error handling | Grafana documentation

Error handling This guide explains how to handle errors in plugins. Provide usable defaults Allow the user to learn your plugin in small...

X509: certificate signed by unknown authority

I try to use grafana-cli to install a plugin. I try it with --insecure flag, error changed, but still can't install it. Without...

Certificate error with renderer docker image and https

There is nothing wrong with the certificate. The ip is the ip of the grafana-image-renderer docker . Setting the environment variable should ignore...