这是indexloc提供的服务,不要输入任何密码
Skip to content

PG pool can cause memory to never be freed on an active server #5087

@jberryman

Description

@jberryman

libpq grows the buffers it uses for input and output, as needed,
exponentially, but it never shrinks them. This means that if we try to keep
connections open in a thread pool for as long as possible we should expect
RES to grow and grow as e.g. a new biggest result query comes in.

Keep in mind this is for each connection in a pool of maybe hundreds, so a few megabytes here and there can add up. And it only takes one outlier huge result set to grow the buffer.

Here we can see libpq doubling the buffer as we expect:

hasura-heaptrack

One proper solution is to simply shrink the buffers back down to some max
acceptable size after each transaction. User Ilya confirms they've been
using essentially this approach successfully in production since reporting
the first ticket below.

Unfortunately it seems we can't simply create a binding and do this without vendoring postgres itself. So we'll first have a workaround and then see if we can do something more efficient.
EDIT: actually it looks like we can, so long as we copy what postgresql-libpq does in their Setup.hs, and we depend on e.g. postgresql-server-dev-9.6 (the version doesn't matter AFAICT)

See e.g. pqCheckInBufferSpace which is the source of the leak that we noticed:

https://docs.huihoo.com/doxygen/postgresql/fe-misc_8c.html#ab9525b356ad08b90e9beb497203848f8

References to others who have encountered this issue

https://www.postgresql.org/message-id/15693-0df90da151425ff5@postgresql.org
https://groups.google.com/forum/#!topic/pgsql.general/DIA4g8mMzNM

Linking:
#3388
#4077

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions