-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Closed
Labels
a/performancec/serverRelated to serverRelated to serverp/highcandidate for being included in the upcoming sprintcandidate for being included in the upcoming sprintsupport/needs-actionsupport ticket that requires action by teamsupport ticket that requires action by team
Description
We recently ended up shutting down our production Hasura deployment, because we were only using it for data ingestion, and we ended up in a situation where Hasura was constantly using all available memory, and then crashing.
I don't have any logs to offer, unfortunately. But I was asked on Discord to create an issue, so here's the information I can provide about our setup:
- Deployed on Heroku
- Initially ran on a $7/mo box on v1alpha1 build 28 (We didn't watch it closely, and at some point realized that it was crashing regularly from running out of memory)
- Scaled the box to larger standard dynos, even up to 3 boxes with 2.5Gb of ram each. Even in that situation, Hasura used all and more than the available ram, and was killed by Heroku.
- Heroku's router measurements said we were getting about 10 reqs / second from our client devices.
- The client devices were doing mutations only. No queries. They were writing in telemetry events.
- Sometimes queries could be multiple megabytes in size (a lot of events were in the write backlog)
- We used an auth hook to our application server that responded usually within a few hundred milliseconds.
- Sometimes the auth hook would throw a 500 error.
- Right before crashing the Hasura logs showed mutation requests timing out (taking longer than 30 seconds)
- Rebooting the box helped for a few seconds, but the slowness almost immediately returned.
- We truncated our events table to make sure the database wasn't the bottleneck, but the memory issues on Hasura continued.
Sorry I can't offer more. Thank you!
yasinarik, jflambert, luke-j, evancorl, nkostic and 9 more24601
Metadata
Metadata
Labels
a/performancec/serverRelated to serverRelated to serverp/highcandidate for being included in the upcoming sprintcandidate for being included in the upcoming sprintsupport/needs-actionsupport ticket that requires action by teamsupport ticket that requires action by team