Description

This PR fixes:-

• While defining new metadata on the console like adding tables/functions, relationships, permissions etc. server throws remote schema errors if the remote server is not available
• Sending raw response body (from a remote server on non 200s) as error message to the client

Details:-

• If any remote schema present in metadata, the server makes introspection query to the remote server to fetch graphql schema for each /v1/query which modifies the schema state, such as track_table, track_function etc.
• Adding a remote schema (add_remote_schema query type) makes 2 network requests. One to fetch and validate. The other is to fetch and stitch the remote schema at different levels.
• Making a bulk query takes significant time since each query make a network request to remote server and rebuilds the graphql schema

The above issues are solved by caching remote schema GraphQL types and avoid refreshing remote schema for each /v1/query API call.
To refresh remote schema, a new /v1/query query type reload_remote_schema is introduced and it is integrated with reload button under modify section of a remote schema in console.

Results:-

Tracking ~300 tables through console which actually makes a bulk query, in the local setup.

• Before: 26.308018138 sec
• After: 1.249884826 sec

Affected components

• Server
• Console
• Docs
• Tests

Design and Solution

Server:-

• Cache parsed GraphQL types from remote server introspection query response to avoid making network requests on every metadata update /v1/query.
• Move graphql schema generation to runQueryM function. This'll abort any changes made if graphql schema generation fails
• Introduce a new reload_remote_schema query type to re-fetch GraphQL types from a given remote server.
• A better error response when the remote schema server responded with non 200 status code
• Log the HTTP exception in JSON format under internal field of error if it occurs while contacting remote server to fetch schema

Console:-

• Use reload_remote_schema API instead of reload_metadata under reload section of a remote schema.

To Reviewer (server):-

Start with ../SchemaCache.hs file and look for RemoteSchemaCtx type

• No new modules are added
• Type RemoteGCtx moved to ..GraphQL/Context.hs since it represents graphQL context for remote schema
• Type RemoteSchemaCtx is context of remote schema which'll be stored in schema cache
• New functions buildGCtxMapPG and buildGCtxMap are defined to make GraphQL schema in state monad

critical modules:

• ../Server/Query.hs
• ../App.hs
• ../DDL/RemoteSchema.hs
• ../GraphQL/RemoteServer.hs

Related Issues

close #1679

Breaking Changes

1. Error response while adding a remote schema is improved to avoid proxying response body in error field on non 200 status code

Before

{
  "path": "$.args[0].args",
  "error": "<Large HTML Response Body from Remote Server>",
  "code": "remote-schema-error"
}

After:

{
  "internal": {
    "raw_body": "<Large HTML Response Body from Remote Server>"
  },
  "path": "$.args[0].args",
  "error": "introspection query to https://hasura.io/random has responded with 404 status code",
  "code": "remote-schema-error"
}

2. Now, http-log of the server will have a new internal property in error object when HTTP or API exception from remote server occurs while adding.

Before:

{
  "timestamp": "2019-06-19T18:59:18.060+0530",
  "level": "info",
  "type": "http-log",
  "detail": {
    "status": 400,
    "query_hash": "ddc917a7d389e0b2bf2ec4e566df8c5a1f3cc612",
    "http_version": "HTTP/1.1",
    "query_execution_time": 0.001634484,
    "request_id": null,
    "url": "/v1/query",
    "ip": "127.0.0.1",
    "response_size": 147,
    "user": {
      "x-hasura-role": "admin"
    },
    "method": "POST",
    "detail": {
      "error": {
        "path": "$.args[0].args",
        "error": "HTTP exception occurred while sending the request to http://localhost:8181/graphql",
        "code": "remote-schema-error"
      },
      "request": "{\"type\":\"bulk\",\"args\":[{\"type\":\"add_remote_schema\",\"args\":{\"name\":\"random\",\"definition\":{\"url\":\"http://localhost:8181/graphql\",\"headers\":[],\"forward_client_headers\":false}}}]}"
    }
  }
}

After:

{
  "timestamp": "2019-06-19T18:53:40.965+0530",
  "level": "info",
  "type": "http-log",
  "detail": {
    "status": 400,
    "query_hash": "a0818e9c99f276c36346affe4d4122a16c864e2e",
    "http_version": "HTTP/1.1",
    "query_execution_time": 0.001481765,
    "request_id": null,
    "url": "/v1/query",
    "ip": "127.0.0.1",
    "response_size": 482,
    "user": {
      "x-hasura-role": "admin"
    },
    "method": "POST",
    "detail": {
      "error": {
        "internal": {
          "message": "ConnectionFailure Network.Socket.connect: <socket: 18>: does not exist (Connection refused)",
          "request": {
            "proxy": null,
            "secure": "False",
            "path": "/graphql",
            "responseTimeout": "ResponseTimeoutDefault",
            "method": "POST",
            "host": "localhost",
            "requestVersion": "HTTP/1.1",
            "redirectCount": "10",
            "port": "8181"
          }
        },
        "path": "$.args[0].args",
        "error": "HTTP exception occurred while sending the request to http://localhost:8181/graphql",
        "code": "remote-schema-error"
      },
      "request": "{\"type\":\"bulk\",\"args\":[{\"type\":\"add_remote_schema\",\"args\":{\"name\":\"random\",\"definition\":{\"url\":\"http://localhost:8181/graphql\",\"headers\":[],\"forward_client_headers\":false}}}]}"
    }
  }
}

Limitation

While adding multiple remote schemas in a bulk query, the error lacks the context of the remote schema which caused merge conflicts, if any, since stitching of all remote schemas happens at the end of the query.

Updates after discussion with @0x777 (3rd July 2019):

Do not build GraphQL context at the end of the bulk query. Fallback to the current method. However, for each query in bulk query, the GraphQL context is built but without fetching schemas from remote servers as we're caching them. We should preserve the semantics of bulk query ie. treat each query as isolated. Hence, GraphQL schema needs to be built for each bulk query.