I will share a little workaround in order to re-establish Prometheus service back in the game.
- Origin of issue:
I’ve had an issue with my local network at home by the ISP provider, I had have shut down my RaspberryPi and other devices.
Then when I could solved the issue and all my devices were up and running I checked up my services and I found that Prometheus got stuck by a “mmap: invalid argument” message.
- Possible troubleshooting:
Have a look at 1)journalctl as well as service by 2)systemd and the full message it was:
level=error ts=2021-02-22T09:37:48.824Z caller=main.go:798 err="opening storage failed: mmap files, file: /usr/local/bin/prometheus/data/chunks_head/000534: mmap: invalid argument"
The service did not get start because when it was trying to grab that “/usr/local/bin/prometheus/data/chunks_head/000534” file stopped automatically.
I checked the folder from that pointed path:
root@raspbi:/usr/local/bin/prometheus# ls -l /usr/local/bin/prometheus/data/chunks_head/
-rw-r--r-- 1 pi pi 1996469 Feb 21 13:00 000533
-rw-r--r-- 1 pi pi 0 Feb 21 13:00 000534
That was the error, the service it was trying to start with a empty file, so I deleted the “000534” and restart the service.
systemctl restart prometheus.service
And all it was back just right.