Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

No metrics on authorization failure

See original GitHub issue

I observed some random outages of our metrics for all proxmox nodes during incidents with only a single affected node. The log is showing this authorization failure triggered by a request done in the collector.py to get the VM metrics. So I think it happens if a node is reported as online but isn’t correctly reachable or has other problems. This should be handled by a try/except logic to allow all other metrics to get correctly collected. I’ll add a pull request for that.

Dec 21 00:00:14 node5 pve_exporter[476752]: Exception thrown while rendering view
Dec 21 00:00:14 node5 pve_exporter[476752]: Traceback (most recent call last):
Dec 21 00:00:14 node5 pve_exporter[476752]:   File "/usr/local/lib/python3.7/dist-packages/pve_exporter/http.py", line 101, in view
Dec 21 00:00:14 node5 pve_exporter[476752]:     return self._views[endpoint](**params)
Dec 21 00:00:14 node5 pve_exporter[476752]:   File "/usr/local/lib/python3.7/dist-packages/pve_exporter/http.py", line 53, in on_pve
Dec 21 00:00:14 node5 pve_exporter[476752]:     output = collect_pve(self._config[module], target)
Dec 21 00:00:14 node5 pve_exporter[476752]:   File "/usr/local/lib/python3.7/dist-packages/pve_exporter/collector.py", line 301, in collect_pve
Dec 21 00:00:14 node5 pve_exporter[476752]:     return generate_latest(registry)
Dec 21 00:00:14 node5 pve_exporter[476752]:   File "/usr/local/lib/python3.7/dist-packages/prometheus_client/exposition.py", line 90, in generate_latest
Dec 21 00:00:14 node5 pve_exporter[476752]:     for metric in registry.collect():
Dec 21 00:00:14 node5 pve_exporter[476752]:   File "/usr/local/lib/python3.7/dist-packages/prometheus_client/registry.py", line 75, in collect
Dec 21 00:00:14 node5 pve_exporter[476752]:     for metric in collector.collect():
Dec 21 00:00:14 node5 pve_exporter[476752]:   File "/usr/local/lib/python3.7/dist-packages/pve_exporter/collector.py", line 272, in collect
Dec 21 00:00:14 node5 pve_exporter[476752]:     for vmdata in self._pve.nodes(node['node']).qemu.get():
Dec 21 00:00:14 node5 pve_exporter[476752]:   File "/usr/local/lib/python3.7/dist-packages/proxmoxer/core.py", line 84, in get
Dec 21 00:00:14 node5 pve_exporter[476752]:     return self(args)._request("GET", params=params)
Dec 21 00:00:14 node5 pve_exporter[476752]:   File "/usr/local/lib/python3.7/dist-packages/proxmoxer/core.py", line 79, in _request
Dec 21 00:00:14 node5 pve_exporter[476752]:     resp.content))
Dec 21 00:00:14 node5 pve_exporter[476752]: proxmoxer.core.ResourceException: 401 Unauthorized: b''