-
Notifications
You must be signed in to change notification settings - Fork 64
Description
Hi, we found our service (built with souin and go-redis) has bad performance, many high latency responces that duration great than 100ms, it's unusual for redis and service in one LAN.
(the responses that duration great than 100ms)
Meanwhile, we can see stats in redis:
(info commandstats)
cmdstat_mget:calls=4610,usec=18398490975,usec_per_call=3990995.75,rejected_calls=0,failed_calls=0 cmdstat_get:calls=20580149,usec=391883145,usec_per_call=19.04,rejected_calls=0,failed_calls=0
We can see usec_per_call=3990995.75 for mget, it's huge amount time for a action in redis, all mget calls block all other get calls.
And get some slow logs, almost from mget:
32) "... (2898426 more arguments)
Millions keys are sent and values recieved,resulting in high CPU usage and throughput.
It looks like the events occur almost a minute apart, most likely due to EvictMapping, which has an interval of 1 minute, and the MapKeys called is the mget used.
When we run our service in multiple pods, the problem will be more painful with more EvictMapping check and more mget to redis.
Please check this.
Souin version: v1.7.7