Fix memory leak #130
Fix memory leak #130
Conversation
|
What can cause a Session to stop receiving internal messages? |
|
The likely case this would trigger is, for example, if a bot/script/match manager were to stop reading from a socket without closing the connection. Previously, RLBotServer's memory usage would explode as it saved every missed message. Now, we store up to 60 and discard any further missed messages. |
|
beta 12 of the GUI currently has this exact bug which causes the memory leak (although the bug has been fixed on master) |
|
I see. And the reason the messages pile up in the internal channel is that we want to make sure the client receives all messages - even old ones? Maybe we should consider changing that philosophy instead. The typical interfaces discards all non-latest game packets anyway. |
|
More just, that's just what happens when you use an unbounded channel. Also, we cant directly control the contents of the channel. Hence, we just stop storing the messages after a little bit instead of letting the unbounded channel grow forever. |
|
A properly functioning client should never encounter this case |
|
Good point. So it's a user error and we need a warning. |
When a session stops receiving messages, we would keep saving all the missed messages would would cause memory usage to balloon. This PR protects core from these misbehaving sessions.
This PR adds back pressure on
FlatbufferServerby limitingsessionChannelto storing a max of 60 items. I think this is a fair number, but if there's crazy amounts ofMatchCommmessages being sent around it might be be enough? The limit can be increased later if needed.If a session misses a message, a warning is printed. In order to not spam the console, exponential backoff has been implemented to a power of 10 (do we want to do the power of 2?) so it will print on the 1st missed message, then the 10th, 100th, 1000th, etc. Example of this warning:
This warning has only been implemented for when
BallPrediction/GamePacketare added tosessionChannelbecause of the high frequency with which they are consistently sent, adding it to these alone is likely enough to flag the problem.