This repository was archived by the owner on Feb 13, 2025. It is now read-only.
This repository was archived by the owner on Feb 13, 2025. It is now read-only.
scollector: too many open files #1557
Description
scollector seems to leak socket connections when left running for several days (>60). The issue becomes apparent when looking at syslog, and noticing that the process stops recording metrics:
$ tail /var/log/syslog
...
scollector[3666]: error: queue.go:84: Post http://localhost:9107/api/put: dial tcp 127.0.0.1:9107: socket: too many open files
The output of lsof shows sockets open:
$ sudo lsof -p 3666
...
scollecto 3666 scollector 1018u sock 0,7 0t0 9458743 can't identify protocol
scollecto 3666 scollector 1019u sock 0,7 0t0 9814562 can't identify protocol
scollecto 3666 scollector 1020u sock 0,7 0t0 9824249 can't identify protocol
scollecto 3666 scollector 1021u sock 0,7 0t0 9832600 can't identify protocol
scollecto 3666 scollector 1022u sock 0,7 0t0 9976212 can't identify protocol
scollecto 3666 scollector 1023u sock 0,7 0t0 9849449 can't identify protocol
This points to similar issues, e.g. http://stackoverflow.com/questions/7911840/seeing-too-many-lsof-cant-identify-protocol, where sockets were left half-open.
My setup uses scollector in conjunction with prometheus-scollector to export metrics to a Prometheus server.