-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Description
TL;DR – The limit argument for computed fields is not included in final SQL via LIMIT clause. BigQuery throws an exception when LIMIT is used after FROM <table-value function> expression in a computed field SELECT sub-query. Navigate to the end of this description to see probable solutions.
I'll try to elaborate the issue with an example.
Consider the following table-valued function, fetch_articles.
CREATE TABLE FUNCTION `<redacted_project_name>.hasura_test.fetch_articles`(a_id INT64, search STRING) AS (
(
(SELECT t.* FROM hasura_test.article t where t.author_id = a_id AND (t.title like `search` or t.content like `search`))
)
);The above function returns rows of article table given author id a_id and search keyword. A computed field is defined to author table with name search_articles using the following definition.
definition:
function:
dataset: hasura_test
name: fetch_articles
argument_mapping:
a_id: id
return_table:
name: article
dataset: hasura_testLet's consider the following GraphQL query that includes the computed field, search_articles.
query {
hasura_test_author{
search_articles(args: {search: "%1%"}){
id
title
content
}
}
}Generates following SQL query (replaced parameters)
SELECT
ARRAY(
(
SELECT
AS STRUCT `fetch_articles1`.`id` AS `id`,
`fetch_articles1`.`title` AS `title`,
`fetch_articles1`.`content` AS `content`
FROM
`hasura_test`.`fetch_articles`(`a_id` => `t_author1`.`id`, `search` => '%1%') AS `fetch_articles1`
)
) AS `search_articles_2`
FROM
`hasura_test`.`author` AS `t_author1`
LIMIT
1000Let's apply LIMIT to the computed field selection,
SELECT
ARRAY(
(
SELECT
AS STRUCT `fetch_articles1`.`id` AS `id`,
`fetch_articles1`.`title` AS `title`,
`fetch_articles1`.`content` AS `content`
FROM
`hasura_test`.`fetch_articles`(`a_id` => `t_author1`.`id`, `search` => '%1%') AS `fetch_articles1`
LIMIT 1
)
) AS `search_articles_2`
FROM
`hasura_test`.`author` AS `t_author1`
LIMIT
1000Running the above query in BigQuery cloud console gives the following error.
Correlated subqueries that reference other tables are not supported unless they can be de-correlated, such as by transforming them into an efficient JOIN.
We apply the global limit to all SELECT queries generated. This results in a query exception for computed field queries. To mitigate this problem, we chose not to include LIMIT clause for computed field sub-queries. It is not an ideal solution (kind of hack), and limit argument for computed field queries became obsolete.
I (@rakeshkky) think, we can solve this by the following ways;
- Investigate further about the error and figure out an alternate SQL that includes
LIMITwithout any exceptions. - Don't generate GraphQL schema for
limitargument for computed fields, and document this limitation at https://hasura.io/docs/latest/graphql/core/index/ inDatabases > BigQuery > Schema > Computed fieldsby updating this file.
Try 1 if it's not possible and go with 2.
Update:
Thoughts on this by @soupi
say a computed fields returns 1000000 rows, would that use a lot of memory on the data loader side (and the graphql-engine)?
Technically, yes. Probably we should only look for the solution 1.