这是indexloc提供的服务,不要输入任何密码
Skip to content

Follower memory spike and increased latency after leader AOF shrink #768

@krkeshav

Description

@krkeshav

Hi @tidwall ,
First of all love your projects, it is helping me so much to learn about Go. Just wanted to report about a strange behaviour in tile38 followers.

Describe the bug
AOF shrink at leader causes memory spike in followers and increase in latency. The memory is not going down even after 6 hours now.

To Reproduce
Have some data in AOF file, then execute aofshrink command on leader. This will result in increase in memory in followers as leader sends either command by command update or the whole AOF file. I even tried manually running gc command on followers but it doesn't work. Our AOF file size is around 27 MB.

Expected behavior
Ideally this should not result in increase in memory on followers, even if there is increase it should go down after sometime.

Operating System (please complete the following information):

  • OS: Linux
  • CPU: Intel
  • Version: 1.30.0
  • Container: Kubernetes via this Helm Chart (link)

Additional context
I am attaching screenshots to show the issue.

Increase in memory in follower pods:
Image

Increase in network activity due to syncing with leader:
Image

Increase in P99 of our services which directly rely on tile38 followers
Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions