Datapoints overwhelm the metrics queue and blow up ram usage. #8272
Conversation
mvines
left a comment
There was a problem hiding this comment.
We previously had coupled datapoint levels to log levels like this, and decoupled because we wanted to allow the metrics databoard-based on debug/trace datapoints s to continue to function while not spamming the log with their values. I don't have a strong opinion on this but I believe @pgarg66 and @sagar-solana did.
Also it seems like when Sagar's work to fix up metrics.rs's run() comes in then this PR won't matter
that wont prevent the constructors from running prior to run. If you want the data, it should be info and produced no more than once per slot. or use a counter which emit once per 1k samples. |
Codecov Report
@@ Coverage Diff @@
## master #8272 +/- ##
========================================
- Coverage 80.5% 80.5% -0.1%
========================================
Files 254 254
Lines 55350 55350
========================================
- Hits 44607 44605 -2
- Misses 10743 10745 +2 |
automerge (cherry picked from commit 17fb825) # Conflicts: # core/src/cluster_info.rs # ledger/src/blockstore_processor.rs
automerge (cherry picked from commit 17fb825)
automerge (cherry picked from commit 17fb825)
Problem
Datapoints are submitted into the metrics queue regardless of what log level is enabled.
Summary of Changes
Don't submit datapoints unless log level is enabled.
Fixes #