Discussion: https://nsidc.slack.com/archives/C4UCJ1NAF/p1666721015728609

Some things to look at:

• ELK Stack (ElasticSearch, Logstash, Kibana): Maybe too heavy and brittle for our use case?
• Graylog: Uses ElasticSearch under-the-hood for searching.
• Grafana Loki: "Like prometheus, but for logs" (We already run Prometheus and Grafana on NSIDC hardware for monitoring VMs). Requires special labels on log messages to effectively do its indexing.

Questions

• How can we make our logging system resilient to changes in log storage backends? E.g. using Vector (https://github.com/vectordotdev/vector) would allow us to log to Vector and broadcast those logs to multiple back-ends, or switch backends in-flight.
• Can our tool of choice ingest from our existing text log files to populate a history?
• Can it ingest from live text files being written to by existing apps not configured for network logging? (e.g. ELK Stack's "filebeat" forwarder)
• What protocol(s) is/are used to pass logs to the service?
• How good/easy is the LDAP auth integration setup?

References/Links

• https://github.com/SigNoz/logs-benchmark