manage schema cache when horizontally scaled (closes #1182) #1574

rakeshkky · 2019-02-08T09:00:35Z

Description

What component does this PR affect?

Requires changes from other components? If yes, please mark the components:

Related Issue

close #1182

Solution and Design

Haskel library pg-client-hs has been updated to expose a function that helps listen to postgres notifications over a channel in this PR
The server records an event in a table hdb_catalog.hdb_cache_update_event whenever any /v1/query (that changes metadata) is requested. A trigger notifies a cache update event via hasura_cache_update channel
The server runs two concurrent threads namely listener and processor. The listener thread listens to events on hasura_cache_update channel and pushed into a Queue. The processor thread fetches events from that Queue and processes it. Thus server rebuilds schema cache from database and updates.

Type

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Docs update
Community content

Checklist:

I have read the contributing guide and my code conforms to the guidelines.
This change requires a change in the documentation.
I have updated the documentation accordingly.
I have added required tests.

hasura-bot · 2019-02-08T09:43:17Z

Review app for commit 428f70d deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-428f70d

coco98 · 2019-02-08T13:30:38Z

We should add tests for this? Will require some interesting black magic from @shahidhk @arvi3411301 I think ;)

server/src-exec/Main.hs

server/src-lib/Hasura/Server/App.hs

server/src-lib/Hasura/Server/Query.hs

0x777 · 2019-02-11T06:13:28Z

server/src-lib/Hasura/Server/Query.hs

  -> UserInfo -> SchemaCache -> HTTP.Manager
  -> RQLQuery -> m (BL.ByteString, SchemaCache)
-runQuery pool isoL userInfo sc hMgr query = do
+runQuery pool isoL serverId userInfo sc hMgr query = do


Should return (BL.ByteString, Maybe (SchemaCache, Timestamp). Just (SC, TS) implies that the schemacache has to be replaced. After this we don't have to use queryNeedsReload in the locations where runQuery is being called.

0x777 · 2019-02-11T06:14:32Z

server/src-rsr/initialise.sql

  comment TEXT
 );
+
+CREATE TABLE hdb_catalog.hdb_cache_update_event (


We are recording schema updates. So let's call it hdb_schema_update_event (cascade the changes through out the PR).

0x777 · 2019-02-11T08:20:06Z

server/src-lib/Hasura/Server/CacheUpdate.hs

+    -- Postgres notification handler
+    notifyHandler notif = do
+      let eventChannel = PGChannel $ bsToTxt $ PQ.notifyRelname notif
+      when (eventChannel == pgChannel) $


Why do this check again?

0x777 · 2019-02-11T08:34:17Z

server/src-lib/Hasura/Server/CacheUpdate.hs

+    onError e = do
+      logError logger threadType $ TEQueryError e
+      -- Push a listen failed event to queue
+      STM.atomically $ STM.writeTQueue eventsQueue CUEListenFail


Don't emit a 'listen error' event. Emit an event once the connection is re-established after an error. You'll probably need to expose this functionality from pg-client-hs, probably listen can take something like this as an argument:

data NotifyHandler = NotifyHandler { _ehOnInit :: IO () , _ehOnMessage :: Notify -> IO () }

0x777 · 2019-02-11T08:48:25Z

server/src-lib/Hasura/Server/CacheUpdate.hs

+cacheUpdateEventProcessor pool logger httpManager eventsQueue
+                       cacheRef lk serverId cacheInit = do
+  -- Initiate previous event IO reference
+  prevEventRef <- IORef.newIORef Nothing


You don't have to store the previous event ref after the above suggested change (emit event on reconnection).

Resolve Conflicts: server/src-exec/Main.hs server/src-lib/Hasura/Server/Init.hs

Resolve Conflicts: server/src-exec/Main.hs server/src-exec/Ops.hs server/src-lib/Hasura/Server/App.hs server/src-lib/Hasura/Server/Init.hs server/src-lib/Hasura/Server/Query.hs server/src-rsr/migrate_from_9_to_10.sql

hasura-bot · 2019-03-06T11:41:44Z

Review app for commit 933b734 deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-933b734

hasura-bot · 2019-03-07T07:08:25Z

Review app for commit e9c7c2b deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-e9c7c2b

hasura-bot · 2019-03-07T08:48:32Z

Review app for commit 1632dd1 deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-1632dd1

hasura-bot · 2019-03-07T09:51:12Z

Review app for commit bf30119 deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-bf30119

…l-engine into issue-1182-cache-update

hasura-bot · 2019-03-07T11:11:39Z

Review app for commit 054b26a deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-054b26a

hasura-bot · 2019-03-07T15:46:51Z

Review app for commit 4472a21 deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-4472a21

hasura-bot · 2019-03-08T14:48:18Z

Review app for commit 6338341 deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-6338341

hasura-bot · 2019-03-11T07:52:35Z

Review app for commit 4a74839 deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-4a74839

hasura-bot · 2019-03-11T13:55:50Z

Review app for commit 524bf91 deployed to Heroku: https://hge-ci-pull-1574.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1574-524bf91

shahidhk

LGTM

hasura-bot · 2019-03-12T05:46:31Z

Review app https://hge-ci-pull-1574.herokuapp.com is deleted

### What  Noticed in #1574 , pulled into own PR as it's a no-op. --------- Co-authored-by: Daniel Chambers <daniel@hasura.io> V3_GIT_ORIGIN_REV_ID: 45bc4c2ff27444502758e32248f40eb61c5f9db0

manage schema cache when horizontally scaled, closes hasura#1182

a79a6cc

rakeshkky added the c/server Related to server label Feb 8, 2019

rakeshkky self-assigned this Feb 8, 2019

rakeshkky requested a review from 0x777 February 8, 2019 09:00

Merge branch 'master' into issue-1182-cache-update

428f70d

shahidhk mentioned this pull request Feb 8, 2019

add horizontal scaling guide #940

Closed

0x777 reviewed Feb 11, 2019

View reviewed changes

rakeshkky added 3 commits February 21, 2019 13:20

support pg-client-hs library with retries & improve naming

4539bdc

Merge branch 'master' into issue-1182-cache-update

2c4fa62

Resolve Conflicts: server/src-exec/Main.hs server/src-lib/Hasura/Server/Init.hs

Merge branch 'master' into issue-1182-cache-update

933b734

Resolve Conflicts: server/src-exec/Main.hs server/src-exec/Ops.hs server/src-lib/Hasura/Server/App.hs server/src-lib/Hasura/Server/Init.hs server/src-lib/Hasura/Server/Query.hs server/src-rsr/migrate_from_9_to_10.sql

added basic tests

e9c7c2b

rakeshkky requested a review from shahidhk as a code owner March 7, 2019 06:21

arvi3411301 added 3 commits March 7, 2019 12:48

unset HASURA_GRAPHQL_AUTH_HOOK

9fb165f

add server option in scaling tests

9504395

fix test script

1632dd1

fix 'shouldReload' function

bf30119

arvi3411301 added 2 commits March 7, 2019 16:05

added relationship test

1d8c647

Merge branch 'issue-1182-cache-update' of github.com:rakeshkky/graphq…

054b26a

…l-engine into issue-1182-cache-update

arvi3411301 added 2 commits March 7, 2019 19:22

added pgbouncer to restart postgres connection

cf28c95

fix pgbouncer config path

4472a21

improve schema sync logic to support latest pg-client-hs lib

6338341

rakeshkky and others added 3 commits March 11, 2019 11:34

improve invoking schema sync threads & improve thread logging

7f98d5f

Merge branch 'master' into issue-1182-cache-update

cc45860

add pgbouncer user

4a74839

arvi3411301 added 7 commits March 11, 2019 15:15

add user flag to pgbouncer command

e5b9b85

fix tests

d0df8ac

add sleep for 30s

a5cf3a3

add 30s delay for test validation

db55aa7

fix tests

6a8c4a1

fix tests

81edef2

run pgbouncer as daemon

524bf91

shahidhk reviewed Mar 12, 2019

View reviewed changes

shahidhk approved these changes Mar 12, 2019

View reviewed changes

shahidhk merged commit e32f5a1 into hasura:master Mar 12, 2019

rakeshkky mentioned this pull request Jul 22, 2019

clear events related to schema cache updates (close #2542) #2585

Merged

jync mentioned this pull request Nov 12, 2020

docs: guide on horizontal scaling & auto-scaling #1183

Open

Allymahmoud mentioned this pull request Feb 17, 2023

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#175

Open

snyk-bot mentioned this pull request Feb 18, 2023

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#176

Open

This was referenced Apr 24, 2023

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#197

Open

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#200

Open

This was referenced Oct 13, 2023

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#329

Open

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#332

Open

This was referenced Feb 18, 2024

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#523

Open

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#524

Open

Allymahmoud mentioned this pull request Apr 5, 2024

[Snyk] Security upgrade graphiql from 1.0.0-alpha.0 to 1.0.0 Allymahmoud/graphql-engine#570

Open

manage schema cache when horizontally scaled (closes #1182) #1574

manage schema cache when horizontally scaled (closes #1182) #1574

Uh oh!

Conversation

rakeshkky commented Feb 8, 2019

Description

Related Issue

Solution and Design

Type

Checklist:

Uh oh!

hasura-bot commented Feb 8, 2019

Uh oh!

coco98 commented Feb 8, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0x777 Feb 11, 2019

Choose a reason for hiding this comment

Uh oh!

0x777 Feb 11, 2019

Choose a reason for hiding this comment

Uh oh!

0x777 Feb 11, 2019

Choose a reason for hiding this comment

Uh oh!

0x777 Feb 11, 2019

Choose a reason for hiding this comment

Uh oh!

0x777 Feb 11, 2019

Choose a reason for hiding this comment

Uh oh!

hasura-bot commented Mar 6, 2019

Uh oh!

hasura-bot commented Mar 7, 2019

Uh oh!

hasura-bot commented Mar 7, 2019

Uh oh!

hasura-bot commented Mar 7, 2019

Uh oh!

hasura-bot commented Mar 7, 2019

Uh oh!

hasura-bot commented Mar 7, 2019

Uh oh!

hasura-bot commented Mar 8, 2019

Uh oh!

hasura-bot commented Mar 11, 2019

Uh oh!

hasura-bot commented Mar 11, 2019

Uh oh!

shahidhk left a comment

Choose a reason for hiding this comment

Uh oh!

hasura-bot commented Mar 12, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants