-
Notifications
You must be signed in to change notification settings - Fork 580
Description
**Free memory of tile38 leader goes down slowly throughout the month, eventually coming down to ~10% **
I am running tile38 with 2 followers and 1 leader, my ec2 instance goes out of memory after 1 to 1.5 months, eventually I have to restart the sentinal and tile38 which resolves the issue.
To Reproduce
My system has 70K+ geofences (on leader) register with kafka callback hooks.
Expected behavior
System should run without any issues, and free memory should not exhaust for leader
Logs
If applicable, provide logs, panics, system messages to help explain your problem.
Operating System (please complete the following information):
OS: Linux - ec2 instance
Version: latest
Additional context
Running with redis-sentinel and leader/follower
Grafana / prometheus query for free memory -
(node_memory_MemFree_bytes {Service="TILE38"} + node_memory_Cached_bytes + node_memory_Buffers_bytes) / node_memory_MemTotal_bytes * 100