这是indexloc提供的服务,不要输入任何密码
Skip to content

Bad performance when many keys in redis #646

@xzycn

Description

@xzycn

Hi, we found our service (built with souin and go-redis) has bad performance, many high latency responces that duration great than 100ms, it's unusual for redis and service in one LAN.

Image
(the responses that duration great than 100ms)

Meanwhile, we can see stats in redis:
(info commandstats)
cmdstat_mget:calls=4610,usec=18398490975,usec_per_call=3990995.75,rejected_calls=0,failed_calls=0 cmdstat_get:calls=20580149,usec=391883145,usec_per_call=19.04,rejected_calls=0,failed_calls=0
We can see usec_per_call=3990995.75 for mget, it's huge amount time for a action in redis, all mget calls block all other get calls.

And get some slow logs, almost from mget:

32) "... (2898426 more arguments)

Millions keys are sent and values recieved,resulting in high CPU usage and throughput.

It looks like the events occur almost a minute apart, most likely due to EvictMapping, which has an interval of 1 minute, and the MapKeys called is the mget used.

When we run our service in multiple pods, the problem will be more painful with more EvictMapping check and more mget to redis.

Please check this.

Souin version: v1.7.7

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions