I will share a little workaround in order to re-establish Prometheus service back in the game.

  • Origin of issue:

I’ve had an issue with my local network at home by the ISP provider, I had have shut down my RaspberryPi and other devices.

Then when I could solved the issue and all my devices were up and running I checked up my services and I found that Prometheus got stuck by a “mmap: invalid argument” message.

  • Possible troubleshooting:

Have a look at 1)journalctl as well as service by 2)systemd and the full message it was:

level=error ts=2021-02-22T09:37:48.824Z caller=main.go:798 err="opening storage failed: mmap files, file: /usr/local/bin/prometheus/data/chunks_head/000534: mmap: invalid argument"

The service did not get start because when it was trying to grab that “/usr/local/bin/prometheus/data/chunks_head/000534” file stopped automatically.

I checked the folder from that pointed path:

root@raspbi:/usr/local/bin/prometheus# ls -l /usr/local/bin/prometheus/data/chunks_head/
-rw-r--r-- 1 pi pi 1996469 Feb 21 13:00 000533
-rw-r--r-- 1 pi pi 0 Feb 21 13:00 000534

That was the error, the service it was trying to start with a empty file, so I deleted the “000534” and restart the service.

systemctl restart prometheus.service

And all it was back just right.