这是indexloc提供的服务,不要输入任何密码
Skip to content

server/bigquery: computed fields ain't working with limit #8562

@rakeshkky

Description

@rakeshkky

TL;DR – The limit argument for computed fields is not included in final SQL via LIMIT clause. BigQuery throws an exception when LIMIT is used after FROM <table-value function> expression in a computed field SELECT sub-query. Navigate to the end of this description to see probable solutions.

I'll try to elaborate the issue with an example.

Consider the following table-valued function, fetch_articles.

CREATE TABLE FUNCTION `<redacted_project_name>.hasura_test.fetch_articles`(a_id INT64, search STRING) AS (
(
(SELECT t.* FROM hasura_test.article t where t.author_id = a_id AND (t.title like `search` or t.content like `search`))
)
);

The above function returns rows of article table given author id a_id and search keyword. A computed field is defined to author table with name search_articles using the following definition.

    definition:
      function:
        dataset: hasura_test
        name: fetch_articles
      argument_mapping:
        a_id: id
      return_table:
        name: article
        dataset: hasura_test

Let's consider the following GraphQL query that includes the computed field, search_articles.

query {
  hasura_test_author{
    search_articles(args: {search: "%1%"}){
      id
      title
      content
    }
  }
}

Generates following SQL query (replaced parameters)

SELECT
  ARRAY(
    (
      SELECT
        AS STRUCT `fetch_articles1`.`id` AS `id`,
        `fetch_articles1`.`title` AS `title`,
        `fetch_articles1`.`content` AS `content`
      FROM
        `hasura_test`.`fetch_articles`(`a_id` => `t_author1`.`id`, `search` => '%1%') AS `fetch_articles1`
    )
  ) AS `search_articles_2`
FROM
  `hasura_test`.`author` AS `t_author1`
LIMIT
  1000

Let's apply LIMIT to the computed field selection,

SELECT
  ARRAY(
    (
      SELECT
        AS STRUCT `fetch_articles1`.`id` AS `id`,
        `fetch_articles1`.`title` AS `title`,
        `fetch_articles1`.`content` AS `content`
      FROM
        `hasura_test`.`fetch_articles`(`a_id` => `t_author1`.`id`, `search` => '%1%') AS `fetch_articles1`
      LIMIT 1
    )
  ) AS `search_articles_2`
FROM
  `hasura_test`.`author` AS `t_author1`
LIMIT
  1000

Running the above query in BigQuery cloud console gives the following error.

Correlated subqueries that reference other tables are not supported unless they can be de-correlated, such as by transforming them into an efficient JOIN. 

We apply the global limit to all SELECT queries generated. This results in a query exception for computed field queries. To mitigate this problem, we chose not to include LIMIT clause for computed field sub-queries. It is not an ideal solution (kind of hack), and limit argument for computed field queries became obsolete.

I (@rakeshkky) think, we can solve this by the following ways;

  1. Investigate further about the error and figure out an alternate SQL that includes LIMIT without any exceptions.
  2. Don't generate GraphQL schema for limit argument for computed fields, and document this limitation at https://hasura.io/docs/latest/graphql/core/index/ in Databases > BigQuery > Schema > Computed fields by updating this file.

Try 1 if it's not possible and go with 2.

Update:
Thoughts on this by @soupi

say a computed fields returns 1000000 rows, would that use a lot of memory on the data loader side (and the graphql-engine)?

Technically, yes. Probably we should only look for the solution 1.

Metadata

Metadata

Assignees

Labels

k/bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions