这是indexloc提供的服务,不要输入任何密码
Skip to content

Hasura consuming massive amounts of memory #4077

@mrmurphy

Description

@mrmurphy

We recently ended up shutting down our production Hasura deployment, because we were only using it for data ingestion, and we ended up in a situation where Hasura was constantly using all available memory, and then crashing.

I don't have any logs to offer, unfortunately. But I was asked on Discord to create an issue, so here's the information I can provide about our setup:

  • Deployed on Heroku
  • Initially ran on a $7/mo box on v1alpha1 build 28 (We didn't watch it closely, and at some point realized that it was crashing regularly from running out of memory)
  • Scaled the box to larger standard dynos, even up to 3 boxes with 2.5Gb of ram each. Even in that situation, Hasura used all and more than the available ram, and was killed by Heroku.
  • Heroku's router measurements said we were getting about 10 reqs / second from our client devices.
  • The client devices were doing mutations only. No queries. They were writing in telemetry events.
  • Sometimes queries could be multiple megabytes in size (a lot of events were in the write backlog)
  • We used an auth hook to our application server that responded usually within a few hundred milliseconds.
  • Sometimes the auth hook would throw a 500 error.
  • Right before crashing the Hasura logs showed mutation requests timing out (taking longer than 30 seconds)
  • Rebooting the box helped for a few seconds, but the slowness almost immediately returned.
  • We truncated our events table to make sure the database wasn't the bottleneck, but the memory issues on Hasura continued.

Sorry I can't offer more. Thank you!

Metadata

Metadata

Labels

a/performancec/serverRelated to serverp/highcandidate for being included in the upcoming sprintsupport/needs-actionsupport ticket that requires action by team

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions