-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Description
libpq grows the buffers it uses for input and output, as needed,
exponentially, but it never shrinks them. This means that if we try to keep
connections open in a thread pool for as long as possible we should expect
RES to grow and grow as e.g. a new biggest result query comes in.
Keep in mind this is for each connection in a pool of maybe hundreds, so a few megabytes here and there can add up. And it only takes one outlier huge result set to grow the buffer.
Here we can see libpq doubling the buffer as we expect:
One proper solution is to simply shrink the buffers back down to some max
acceptable size after each transaction. User Ilya confirms they've been
using essentially this approach successfully in production since reporting
the first ticket below.
Unfortunately it seems we can't simply create a binding and do this without vendoring postgres itself. So we'll first have a workaround and then see if we can do something more efficient.
EDIT: actually it looks like we can, so long as we copy what postgresql-libpq does in their Setup.hs, and we depend on e.g. postgresql-server-dev-9.6 (the version doesn't matter AFAICT)
See e.g. pqCheckInBufferSpace which is the source of the leak that we noticed:
https://docs.huihoo.com/doxygen/postgresql/fe-misc_8c.html#ab9525b356ad08b90e9beb497203848f8
References to others who have encountered this issue
https://www.postgresql.org/message-id/15693-0df90da151425ff5@postgresql.org
https://groups.google.com/forum/#!topic/pgsql.general/DIA4g8mMzNM