-
Notifications
You must be signed in to change notification settings - Fork 581
Description
Hi @tidwall ,
First of all love your projects, it is helping me so much to learn about Go. Just wanted to report about a strange behaviour in tile38 followers.
Describe the bug
AOF shrink at leader causes memory spike in followers and increase in latency. The memory is not going down even after 6 hours now.
To Reproduce
Have some data in AOF file, then execute aofshrink
command on leader. This will result in increase in memory in followers as leader sends either command by command update or the whole AOF file. I even tried manually running gc
command on followers but it doesn't work. Our AOF file size is around 27 MB.
Expected behavior
Ideally this should not result in increase in memory on followers, even if there is increase it should go down after sometime.
Operating System (please complete the following information):
- OS: Linux
- CPU: Intel
- Version: 1.30.0
- Container: Kubernetes via this Helm Chart (link)
Additional context
I am attaching screenshots to show the issue.
Increase in memory in follower pods:
Increase in network activity due to syncing with leader:
Increase in P99 of our services which directly rely on tile38 followers