这是indexloc提供的服务,不要输入任何密码
Skip to content

Free memory of leader instance comes down to 10% every 1-1.5 months #769

@Mukund2900

Description

@Mukund2900

**Free memory of tile38 leader goes down slowly throughout the month, eventually coming down to ~10% **
I am running tile38 with 2 followers and 1 leader, my ec2 instance goes out of memory after 1 to 1.5 months, eventually I have to restart the sentinal and tile38 which resolves the issue.

Image

Image

To Reproduce
My system has 70K+ geofences (on leader) register with kafka callback hooks.

Image I am running GC, AOFSHRINK periodically through a cron on all instances.

Expected behavior
System should run without any issues, and free memory should not exhaust for leader

Logs
If applicable, provide logs, panics, system messages to help explain your problem.

Operating System (please complete the following information):
OS: Linux - ec2 instance
Version: latest

Additional context
Running with redis-sentinel and leader/follower

Grafana / prometheus query for free memory -
(node_memory_MemFree_bytes {Service="TILE38"} + node_memory_Cached_bytes + node_memory_Buffers_bytes) / node_memory_MemTotal_bytes * 100

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions