recover from inconsistent database state (close #231) #1863

rakeshkky · 2019-03-25T16:35:55Z

Description

The server keeps track of all inconsistent metadata objects and reports them on /v1/query of type get_inconsistent_metadata.

API request:-

POST /v1/query
{
  "type": "get_inconsistent_metadata",
  "args": {}
}

Response:-

{ 
    "is_consistent": false,
    "inconsistent_objects": [
  {
    "definition": {
      "role": "user",
      "comment": null,
      "permission": {
        "allow_aggregations": false,
        "columns": [
          "author",
          "author_id",
          "id",
          "title"
        ],
        "filter": {
          "author": {
            "_eq": "X-Hasur\r\na-User-Id"
          }
        }
      },
      "table": "article"
    },
    "reason": "table \"article\" does not exist",
    "type": "select_permission"
  },
  {
    "definition": {
      "using": {
        "foreign_key_constrai\r\nnt_on": "author_id"
      },
      "name": "author_obj",
      "comment": null,
      "table": "article"
    },
    "reason": "table \"article\" does not exist",
    "type": "object_relation"
  },
  {
    "definition": {
      "using": {
        "foreign_key_constraint_on": {
          "column": "author_id",
          "table": "article"
        }
      },
      "name": "articles",
      "comment": null,
      "table": "author"
    },
    "reason": "t\r\nable \"article\" does not exist",
    "type": "array_relation"
  },
  {
    "definition": "article",
    "reason": "no such table/view exists in postgres : \"article\"",
    "type": "table"
  }
]
}

Use drop_inconsistent_metadata query type to purge all inconsistent objects.

API

POST /v1/query
{
  "type": "drop_inconsistent_metadata",
  "args": {}
}

Response:-

{
    "message": "success"
}

Affected components

Server
Console
Docs
Tests

Related Issues

close #231

Solution and Design

buildSchemaCache function from Hasura.RQL.DDL.Schema.Table module is updated to collect inconsistent objects and push to schema cache.
Add get_inconsistent_metadata and drop_inconsistent_metadata query types in /v1/query API
Remote schemas related metadata API code is refactored to support handling inconsistent ones.

Steps to test and verify

Start GraphQL Engine, add few tables and define permissions and relationships.
Through psql or adminer drop any table which is being tracked.
Now, restart GraphQL Engine or reload metadata in settings section of console.
Make get_inconsistent_metadata API query (spec given above), you'll get list of all inconsistent metadata objects.

Resolve Conflicts: server/src-lib/Hasura/GraphQL/Resolve.hs server/src-lib/Hasura/GraphQL/Resolve/Insert.hs server/src-lib/Hasura/GraphQL/Resolve/Mutation.hs server/src-lib/Hasura/RQL/DDL/Metadata.hs server/src-lib/Hasura/RQL/DDL/QueryTemplate.hs server/src-lib/Hasura/RQL/DDL/Schema/Table.hs server/src-lib/Hasura/RQL/DDL/Subscribe.hs server/src-lib/Hasura/RQL/DML/Count.hs server/src-lib/Hasura/RQL/DML/Delete.hs server/src-lib/Hasura/RQL/DML/QueryTemplate.hs server/src-lib/Hasura/RQL/DML/Select.hs server/src-lib/Hasura/RQL/DML/Update.hs server/src-lib/Hasura/Server/Query.hs

netlify · 2019-03-25T16:36:57Z

Deploy preview for hasura-docs ready!

Built with commit 174be30

https://deploy-preview-1863--hasura-docs.netlify.com

hasura-bot · 2019-03-25T17:12:24Z

Review app for commit 2d69ab2 deployed to Heroku: https://hge-ci-pull-1863.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1863-2d69ab2

Resolve Conflicts: server/src-lib/Hasura/GraphQL/Resolve.hs server/src-lib/Hasura/GraphQL/Resolve/Context.hs server/src-lib/Hasura/GraphQL/Resolve/Mutation.hs server/src-lib/Hasura/GraphQL/Resolve/Select.hs server/src-lib/Hasura/GraphQL/Transport/HTTP.hs server/src-lib/Hasura/RQL/DDL/EventTrigger.hs server/src-lib/Hasura/RQL/DDL/Schema/Table.hs

hasura-bot · 2019-03-26T12:52:22Z

Review app for commit 5b7678a deployed to Heroku: https://hge-ci-pull-1863.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1863-5b7678a

danielcompton · 2019-03-26T23:20:48Z

Thanks for this!

We were able to use the soft start feature to boot an app when our schema had changed externally. Previous Hasura wouldn't boot, so we couldn't reset or apply new metadata. This looks to be on the right track. I didn't try the API though, just using the CLI and the console.

hasura-bot · 2019-03-27T07:53:47Z

Review app for commit 5b7678a deployed to Heroku: https://hge-ci-pull-1863.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1863-5b7678a

hasura-bot · 2019-03-27T10:23:25Z

Review app for commit 5b8d129 deployed to Heroku: https://hge-ci-pull-1863.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1863-5b8d129

…ngine into issue-231-incon-meta

rikinsk-zz

docs + console approved

rikinsk-zz · 2019-04-16T11:56:13Z

@dsandip @wawhal This will make https://docs.hasura.io/1.0/graphql/manual/troubleshooting/index.html redundant right?

rikinsk-zz · 2019-04-16T11:59:28Z

related #415

wawhal · 2019-04-16T12:22:57Z

@rikinsk I think it should be there. Because finding those inconsistencies in the schema is not straight forward. We should just add a note at the top saying "If your version is above , you can look at all the inconsistencies at /metadata/status"

hasura-bot · 2019-04-17T08:50:24Z

Review app for commit b365e23 deployed to Heroku: https://hge-ci-pull-1863.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1863-b365e23

Resolve Conflicts: server/src-exec/Main.hs server/src-lib/Hasura/Server/App.hs server/src-lib/Hasura/Server/Init.hs server/src-lib/Hasura/Server/Query.hs server/src-lib/Hasura/Server/SchemaUpdate.hs

…ngine into issue-231-incon-meta

…nsistent objects log

hasura-bot · 2019-04-17T15:00:09Z

Review app for commit 174be30 deployed to Heroku: https://hge-ci-pull-1863.herokuapp.com
Docker image for server: hasura/graphql-engine:pull1863-174be303

hasura-bot · 2019-04-17T16:29:42Z

Review app https://hge-ci-pull-1863.herokuapp.com is deleted

…1863)

) Context: https://app.slack.com/client/T7GHF0SM9/C08PAUZG6F6 aside: this error message (or maybe it's `UnknownFieldInOutputPermissionsDefinition`) wasn't visible to the user. I couldn't immediately see how ErrorVisibility ultimately gets set for this type. Also looking at `set_span_attributes` it seems we don't have a mechanism for surfacing non-internal errors that end up in an internal span, is that what we want? If it is, should I make `build_metadata` user-visible? https://prod-gcp-grafana.hasura.io/d/a0aabf5f-9eb1-467b-8f5d-b7746a33c2a2/view-trace?orgId=1&var-traceId=1e444531ec996c040799e8ea9990de2b&from=now-6h&to=now&timezone=browser V3_GIT_ORIGIN_REV_ID: 28d76ac421a60994ef439f879794cc5e82af0e1f

rakeshkky added 2 commits March 25, 2019 18:38

add soft start and collect inconsistent objects

6fb1cd2

rakeshkky added s/do-not-merge Do not merge this pull request to master s/wip Status: This issue is a work in progress labels Mar 25, 2019

rakeshkky self-assigned this Mar 25, 2019

rakeshkky requested a review from 0x777 as a code owner March 25, 2019 16:35

rakeshkky added 2 commits March 26, 2019 12:06

remove '--enable-soft-start' flag & log inconsistent objects on startup

5b7678a

Merge branch 'master' into issue-231-incon-meta

00db94d

danielcompton mentioned this pull request Mar 26, 2019

Improve error messages for metadata apply errors #1877

Closed

add a test and update docs

7770752

rakeshkky requested a review from rikinsk-zz as a code owner March 27, 2019 09:52

rakeshkky added c/console Related to console c/docs Related to docs c/server Related to server s/do-not-merge Do not merge this pull request to master and removed s/do-not-merge Do not merge this pull request to master labels Mar 27, 2019

Merge branch 'master' into issue-231-incon-meta

5b8d129

wawhal added 6 commits March 27, 2019 16:15

handle redirects for inconsistent metadata

7e2f2d2

filtering schema based on inconsistent objects wip

67f7d06

get inconsitent objects on load or on catch

344b5ff

add filters for inconsistent triggers

bcd9ba8

add loading indicator in error boundary

29d6bce

add table to metadata page

1469c0a

wawhal added 2 commits April 16, 2019 16:01

handle backwards compatibility

b748364

Merge branch 'issue-231-incon-meta' of github.com:rakeshkky/graphql-e…

48abed5

…ngine into issue-231-incon-meta

wawhal dismissed rikinsk-zz’s stale review via 48abed5 April 16, 2019 10:32

rikinsk-zz previously approved these changes Apr 16, 2019

View reviewed changes

fix an issue with backwards compatibility

e19cdbf

wawhal dismissed rikinsk-zz’s stale review via e19cdbf April 16, 2019 11:24

add more wait time to modify table tests

377a47f

debug test failuer

b365e23

wawhal and others added 4 commits April 17, 2019 14:30

remove logs, add more wait time

4326bc1

increase wait time

c519a4c

Merge branch 'master' into issue-231-incon-meta

ce9c223

Resolve Conflicts: server/src-exec/Main.hs server/src-lib/Hasura/Server/App.hs server/src-lib/Hasura/Server/Init.hs server/src-lib/Hasura/Server/Query.hs server/src-lib/Hasura/Server/SchemaUpdate.hs

Merge branch 'issue-231-incon-meta' of github.com:rakeshkky/graphql-e…

1de849d

…ngine into issue-231-incon-meta

0x777 previously approved these changes Apr 17, 2019

View reviewed changes

load inconsistent objects when loading schema

6916ea0

wawhal dismissed 0x777’s stale review via 6916ea0 April 17, 2019 12:57

wawhal and others added 3 commits April 17, 2019 18:30

load inconsistent objects while loading triggers

c565ec4

dd /healthz GET endpoint to check server health & warn level for inco…

2fd9708

…nsistent objects log

Merge branch 'master' into issue-231-incon-meta

174be30

0x777 approved these changes Apr 17, 2019

View reviewed changes

0x777 merged commit 1d7cbc7 into hasura:master Apr 17, 2019

dsandip removed this from the release-candidates milestone Apr 18, 2019

rakeshkky mentioned this pull request Jun 11, 2019

remote schema enable/disable #1638

Open

polRk pushed a commit to polRk/graphql-engine that referenced this pull request Feb 12, 2020

recover from inconsistent database state (closes hasura#231) (hasura#…

243fab5

…1863)

recover from inconsistent database state (close #231) #1863

recover from inconsistent database state (close #231) #1863

Uh oh!

Conversation

rakeshkky commented Mar 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Affected components

Related Issues

Solution and Design

Steps to test and verify

Uh oh!

netlify bot commented Mar 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hasura-bot commented Mar 25, 2019

Uh oh!

hasura-bot commented Mar 26, 2019

Uh oh!

danielcompton commented Mar 26, 2019

Uh oh!

hasura-bot commented Mar 27, 2019

Uh oh!

hasura-bot commented Mar 27, 2019

Uh oh!

rikinsk-zz left a comment

Choose a reason for hiding this comment

Uh oh!

rikinsk-zz commented Apr 16, 2019

Uh oh!

rikinsk-zz commented Apr 16, 2019

Uh oh!

wawhal commented Apr 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hasura-bot commented Apr 17, 2019

Uh oh!

hasura-bot commented Apr 17, 2019

Uh oh!

hasura-bot commented Apr 17, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

rakeshkky commented Mar 25, 2019 •

edited

Loading

netlify bot commented Mar 25, 2019 •

edited

Loading

wawhal commented Apr 16, 2019 •

edited

Loading