这是indexloc提供的服务，不要输入任何密码
Skip to content

Make catalog metadata migrations work on all schema versions (fix #2826) #2379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Oct 11, 2019
Merged

Make catalog metadata migrations work on all schema versions (fix #2826) #2379

merged 12 commits into from
Oct 11, 2019

Conversation

@ajeetdsouza
Copy link
Contributor

@ajeetdsouza ajeetdsouza commented Jun 14, 2019 •
edited by lexi-lambda
Loading

Description

As reported in #2826, catalog metadata migrations can fail when migrating from older catalog versions. This happens because catalog metadata is updated by running ordinary RQL queries like add_existing_table_or_view and create_object_relationship, but the implementations of those APIs will always use the newest version of the code, which may depend on recent schema changes.

This change fixes that by running all database migrations before changing the metadata at all, then simply dropping all metadata for the entire Hasura catalog and recreating it from scratch.

Affected components

  • Server

Related Issues

#2826

Solution and Design

Changes to catalog metadata are no longer expressed as individual migrations. Instead, the src-rsr/hdb_metadata.yaml file is used directly to recreate all catalog metadata in one step. src-exec/Migrate.hs has been rewritten to both incorporate this behavioral change and to avoid so much duplication: all SQL migrations are now stored in separate files (now in the src-rsr/migrations/ directory), and Migrate.hs automatically loads the necessary migration scripts without needing to maintain an explicit list.

A few other changes have been made to support the main goal of this PR:

  1. The catalog version is now defined in a new Migrate.Version module instead of directly in Migrate. This is unfortunately necessary to workaround the GHC stage restriction, which forbids using a definition from the current module in a Template Haskell splice.

  2. A new HasSystemDefined typeclass has been introduced to parameterize metadata-altering queries over whether or not the current metadata objects being defined should be considered “system-defined” or not. This avoids needing to explicitly specify system_defined: true for every query in hdb_metadata.yaml.

  3. The use of the ginger library for templating has been removed and replaced with shakespeare. Profiling revealed that ginger is remarkably slow, which caused problems due to the higher frequency at which the new approach to migrations calls buildSchemaCache. Since the templates are all statically known at compile-time, shakespeare is dramatically more efficient: it uses Template Haskell to compile the templates to Haskell code that simply uses variables from the current lexical scope.

Steps to test and verify

Fire up an older version of graphql-engine using Docker, and test if migrations are working by switching to this version.

Limitations, known bugs & workarounds

None

@ajeetdsouza ajeetdsouza requested a review from 0x777 as a code owner June 14, 2019 11:44
@hasura-bot
Copy link
Contributor

hasura-bot commented Jun 14, 2019

Beep boop! 🤖

Hey @ajeetdsouza, thanks for your PR!

One of my human friends will review this PR and get back to you as soon as possible. 🕐

Stay awesome! 😎

@ajeetdsouza ajeetdsouza self-assigned this Jun 14, 2019
@ajeetdsouza ajeetdsouza requested a review from rakeshkky June 14, 2019 11:45
@netlify
Copy link

netlify bot commented Jun 14, 2019 •
edited
Loading

Deploy preview for hasura-docs ready!

Built with commit b9bfb06

https://deploy-preview-2379--hasura-docs.netlify.com

@strongSoda strongSoda added the c/server Related to server label Jun 14, 2019
@ajeetdsouza ajeetdsouza added the s/do-not-merge Do not merge this pull request to master label Jun 14, 2019
@hasura-bot
Copy link
Contributor

hasura-bot commented Jun 14, 2019

Review app for commit 368b476 deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-368b476a

@ajeetdsouza ajeetdsouza removed the s/do-not-merge Do not merge this pull request to master label Jun 17, 2019
@hasura-bot
Copy link
Contributor

hasura-bot commented Jun 17, 2019

Review app for commit 76bdfe6 deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-76bdfe65

@rakeshkky rakeshkky changed the title Improve migration on startup improve migration on startup Jun 20, 2019
rakeshkky
rakeshkky reviewed Jun 21, 2019
View reviewed changes
rakeshkky
rakeshkky reviewed Jun 21, 2019
View reviewed changes
rakeshkky
rakeshkky reviewed Jun 21, 2019
View reviewed changes
rakeshkky
rakeshkky reviewed Jun 21, 2019
View reviewed changes
rakeshkky
rakeshkky reviewed Jun 21, 2019
View reviewed changes
rakeshkky
rakeshkky reviewed Jun 21, 2019
View reviewed changes
rakeshkky
rakeshkky reviewed Jun 21, 2019
View reviewed changes
@hasura-bot
Copy link
Contributor

hasura-bot commented Jun 24, 2019

Review app for commit 7ba1b7c deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-7ba1b7cc

@hasura-bot
Copy link
Contributor

hasura-bot commented Jul 2, 2019

Review app for commit 2c60cc7 deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-2c60cc78

@hasura-bot
Copy link
Contributor

hasura-bot commented Jul 5, 2019

Review app for commit c10ace0 deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-c10ace06

@shahidhk
Copy link
Member

shahidhk commented Jul 5, 2019

@ajeetdsouza Can you update the description to indicate that this PR is talking about catalog migrations that happen when Hasura versions are upgraded and not the "CLI migrations" -- which the users are familiar with?

@marionschleifer marionschleifer requested review from lexi-lambda and removed request for 0x777 August 30, 2019 08:53
@jberryman
Copy link
Collaborator

jberryman commented Sep 12, 2019

#2826

@hasura-bot
Copy link
Contributor

hasura-bot commented Sep 17, 2019

Review app for commit fe7b6d2 deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-fe7b6d2c

@hasura-bot
Copy link
Contributor

hasura-bot commented Sep 17, 2019

Review app for commit 6cda36a deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-6cda36a4

@lexi-lambda lexi-lambda changed the title improve migration on startup Make catalog metadata migrations work on all schema versions (fix #2826) Oct 9, 2019
@lexi-lambda lexi-lambda requested review from rakeshkky and removed request for shahidhk October 9, 2019 10:26
@hasura-bot
Copy link
Contributor

hasura-bot commented Oct 9, 2019

Review app for commit 5650bcc deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-5650bcc1

0x777
0x777 reviewed Oct 10, 2019
View reviewed changes
@lexi-lambda lexi-lambda requested a review from 0x777 October 10, 2019 06:59
0x777
0x777 previously approved these changes Oct 10, 2019
View reviewed changes
@lexi-lambda lexi-lambda requested a review from 0x777 October 11, 2019 04:13
0x777
0x777 previously approved these changes Oct 11, 2019
View reviewed changes
lexi-lambda
lexi-lambda previously approved these changes Oct 11, 2019
View reviewed changes
@hasura-bot
Copy link
Contributor

hasura-bot commented Oct 11, 2019

Review app for commit fe8fce2 deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-fe8fce28

@lexi-lambda lexi-lambda dismissed stale reviews from 0x777 and themself via b9bfb06 October 11, 2019 04:42
@hasura-bot
Copy link
Contributor

hasura-bot commented Oct 11, 2019

Review app for commit b9bfb06 deployed to Heroku: https://hge-ci-pull-2379.herokuapp.com
Docker image for server: hasura/graphql-engine:pull2379-b9bfb069

@lexi-lambda lexi-lambda merged commit a66fb42 into hasura:master Oct 11, 2019
@hasura-bot
Copy link
Contributor

hasura-bot commented Oct 11, 2019

Review app https://hge-ci-pull-2379.herokuapp.com is deleted

@hasura-bot
Copy link
Contributor

hasura-bot commented Oct 11, 2019

Beep boop! 🤖

GIF

Awesome work @ajeetdsouza! All of us at Hasura ❤️ what you did.

Thanks again 🤗

@rakeshkky rakeshkky mentioned this pull request Oct 17, 2019
Merged
@lexi-lambda lexi-lambda mentioned this pull request Nov 21, 2019
Merged
33 tasks
@lexi-lambda lexi-lambda mentioned this pull request Jan 14, 2020
Merged
5 tasks
polRk pushed a commit to polRk/graphql-engine that referenced this pull request Feb 12, 2020
@ajeetdsouza @polRk
Make catalog metadata migrations work on all schema versions (fix has…
746ddf3 
…ura#2826) (hasura#2379)

* Separate DB and metadata migrations
* Refactor Migrate.hs to generate list of migrations at compile-time
* Replace ginger with shakespeare to improve performance
* Improve migration log messages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@0x777 0x777 0x777 left review comments

@rakeshkky rakeshkky Awaiting requested review from rakeshkky

+1 more reviewer

@lexi-lambda lexi-lambda lexi-lambda approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@lexi-lambda lexi-lambda

Labels

c/server Related to server

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@ajeetdsouza @hasura-bot @shahidhk @jberryman @lexi-lambda @0x777 @rakeshkky @strongSoda