背景
由于内部服务器安装GitLab，但是发型GitLab运行一段时间后就无法访问，重启后又可以
终极分析原因
由于之前升级python的原因，导致安装GitLab的时候安装prometheus失败
解决方案
查看GitLab状态
gitlab-ctl status
发现prometheus启动失败
run: gitaly: (pid 1508) 19980339s; run: log: (pid 1500) 19980339s
run: gitlab-monitor: (pid 1505) 19980339s; run: log: (pid 1496) 19980339s
run: gitlab-workhorse: (pid 1513) 19980339s; run: log: (pid 1510) 19980339s
run: logrotate: (pid 15567) 1913s; run: log: (pid 1502) 19980339s
run: nginx: (pid 1509) 19980339s; run: log: (pid 1498) 19980339s
run: node-exporter: (pid 1504) 19980339s; run: log: (pid 1494) 19980339s
run: postgres-exporter: (pid 1506) 19980339s; run: log: (pid 1497) 19980339s
run: postgresql: (pid 1532) 19980339s; run: log: (pid 1514) 19980339s
down: prometheus: 6s, normally up; run: log: (pid 17174) 1076s
run: sidekiq: (pid 1823) 19980328s; run: log: (pid 1499) 19980339s
run: unicorn: (pid 1512) 19980339s; run: log: (pid 1501) 19980339s
查看日志
tail -f -n 100 /var/log/gitlab/prometheus/current
发现LevelDB存在问题，提示需要修复
2018-09-26_13:23:42.79129 time="2018-09-26T21:23:42+08:00" level=info msg="Listening on localhost:11002" source="web.go:341"
2018-09-26_13:23:42.79331 time="2018-09-26T21:23:42+08:00" level=error msg="Could not open the fingerprint-to-metric index for archived series. Please try a 3rd party tool to repair LevelDB in directory "/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric". If unsuccessful or undesired, delete the whole directory and restart Prometheus for crash recovery. You will lose all archived time series." source="persistence.go:213"
2018-09-26_13:23:42.79333 time="2018-09-26T21:23:42+08:00" level=error msg="Error opening memory series storage: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000785]" source="main.go:192"
网上查找相关资料，说运行以下命令修复
sudo -u gitlab-prometheus python -c "import leveldb; leveldb.RepairDB('/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric')"
但是运行提示错误：
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named leveldb
需要先安装leveldb
pip install leveldb
然后再运行命令... 这时发现pip也出现错误
如提示pip相关错误，那可能是因为之前python升级问题，导致错误
这个时候你需要重新安装setuptools和pip，参考升级地址：
https://blog.csdn.net/uisoul/article/details/90216021
升级完成后，重新执行修复命令，就可以了
pip install leveldb
sudo -u gitlab-prometheus python -c "import leveldb; leveldb.RepairDB('/var/opt/gitlab/prometheus/data/archived_fingerprint_to_metric')"
