这是indexloc提供的服务，不要输入任何密码
Skip to content

feat: job creation mode GA #1480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 28, 2025
+58 −48
Merged

feat: job creation mode GA #1480

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
158df16
feat: job creation mode GA
alvarowolfx May 27, 2025
e5cebcc
Merge branch 'main' into feat-short-mode-ga
alvarowolfx May 28, 2025
f283e63
chore: rename short mode to job optional
alvarowolfx May 28, 2025
f3f32df
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] May 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ Samples are in the [`samples/`](https://github.com/googleapis/nodejs-bigquery/tr
| Query Dry Run | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryDryRun.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryDryRun.js,samples/README.md) |
| Query External GCS Perm | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryExternalGCSPerm.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryExternalGCSPerm.js,samples/README.md) |
| Query External GCS Temp | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryExternalGCSTemp.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryExternalGCSTemp.js,samples/README.md) |
| Query Job Optional | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryJobOptional.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryJobOptional.js,samples/README.md) |
| Query Legacy | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryLegacy.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryLegacy.js,samples/README.md) |
| Query Legacy Large Results | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryLegacyLargeResults.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryLegacyLargeResults.js,samples/README.md) |
| Query Pagination | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryPagination.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryPagination.js,samples/README.md) |
Expand All @@ -163,7 +164,6 @@ Samples are in the [`samples/`](https://github.com/googleapis/nodejs-bigquery/tr
| Query Params Positional Types | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryParamsPositionalTypes.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryParamsPositionalTypes.js,samples/README.md) |
| Query Params Structs | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryParamsStructs.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryParamsStructs.js,samples/README.md) |
| Query Params Timestamps | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryParamsTimestamps.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryParamsTimestamps.js,samples/README.md) |
| Query Short Mode | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryShortMode.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryShortMode.js,samples/README.md) |
| Query Stack Overflow | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryStackOverflow.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryStackOverflow.js,samples/README.md) |
| Quickstart | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/quickstart.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/quickstart.js,samples/README.md) |
| Relax Column | [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/relaxColumn.js) | [![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/relaxColumn.js,samples/README.md) |
Expand Down
36 changes: 18 additions & 18 deletions samples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@
* [Query Dry Run](#query-dry-run)
* [Query External GCS Perm](#query-external-gcs-perm)
* [Query External GCS Temp](#query-external-gcs-temp)
* [Query Job Optional](#query-job-optional)
* [Query Legacy](#query-legacy)
* [Query Legacy Large Results](#query-legacy-large-results)
* [Query Pagination](#query-pagination)
Expand All @@ -96,7 +97,6 @@
* [Query Params Positional Types](#query-params-positional-types)
* [Query Params Structs](#query-params-structs)
* [Query Params Timestamps](#query-params-timestamps)
* [Query Short Mode](#query-short-mode)
* [Query Stack Overflow](#query-stack-overflow)
* [Quickstart](#quickstart)
* [Relax Column](#relax-column)
Expand Down Expand Up @@ -1408,6 +1408,23 @@ __Usage:__



### Query Job Optional

View the [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryJobOptional.js).

[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryJobOptional.js,samples/README.md)

__Usage:__


`node samples/queryJobOptional.js`


-----




### Query Legacy

View the [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryLegacy.js).
Expand Down Expand Up @@ -1578,23 +1595,6 @@ __Usage:__



### Query Short Mode

View the [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryShortMode.js).

[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/nodejs-bigquery&page=editor&open_in_editor=samples/queryShortMode.js,samples/README.md)

__Usage:__


`node samples/queryShortMode.js`


-----




### Query Stack Overflow

View the [source code](https://github.com/googleapis/nodejs-bigquery/blob/main/samples/queryStackOverflow.js).
Expand Down
21 changes: 13 additions & 8 deletions samples/queryShortMode.js → samples/queryJobOptional.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,17 @@
'use strict';

function main() {
// [START bigquery_query_shortmode]
// [START bigquery_query_job_optional]
// Demonstrates issuing a query that may be run in short query mode.
// To enable the short query mode preview feature, the QUERY_PREVIEW_ENABLED
// environmental variable should be set to `TRUE`.

// Import the Google Cloud client library
const {BigQuery} = require('@google-cloud/bigquery');
const bigquery = new BigQuery();
const bigquery = new BigQuery({
// default behavior is to create jobs when using the jobs.query API
defaultJobCreationMode: 'JOB_CREATION_REQUIRED',
});

async function queryShortMode() {
async function queryJobOptional() {
// SQL query to run.

const sqlQuery = `
Expand All @@ -35,7 +36,11 @@ function main() {
LIMIT 10`;

// Run the query
const [rows, , res] = await bigquery.query(sqlQuery);
const [rows, , res] = await bigquery.query({
query: sqlQuery,
// Skip job creation to enable short mode.
jobCreationMode: 'JOB_CREATION_OPTIONAL',
});

if (!res.jobReference) {
console.log(`Query was run in short mode. Query ID: ${res.queryId}`);
Expand All @@ -50,7 +55,7 @@ function main() {
console.log('Rows:');
rows.forEach(row => console.log(row));
}
// [END bigquery_query_shortmode]
queryShortMode();
// [END bigquery_query_job_optional]
queryJobOptional();
}
main(...process.argv.slice(2));
4 changes: 2 additions & 2 deletions samples/test/queries.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ describe('Queries', () => {
assert.match(output, /name/);
});

it('should run a query in short mode', async () => {
const output = execSync('node queryShortMode.js');
it('should run a query stateless mode', async () => {
const output = execSync('node queryJobOptional.js');
assert.match(output, /Rows:/);
assert.match(output, /name/);
});
Expand Down
34 changes: 18 additions & 16 deletions src/bigquery.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@
bigquery.IJob
>;

type JobCreationMode = bigquery.IQueryRequest['jobCreationMode'];

export type Query = JobRequest<bigquery.IJobConfigurationQuery> & {
destination?: Table;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
Expand All @@ -119,6 +121,8 @@
pageToken?: string;
wrapIntegers?: boolean | IntegerTypeCastOptions;
parseJSON?: boolean;
// Overrides default job creation mode set on the client.
jobCreationMode?: JobCreationMode;
};

export type QueryParamTypeStruct = {
Expand Down Expand Up @@ -271,6 +275,13 @@
* Defaults to `googleapis.com`.
*/
universeDomain?: string;

/**
* Controls the job creation mode used when executing queries that can be
* accelerated via the jobs.Query API. Users may experience performance
* improvements by leveraging the JOB_CREATION_OPTIONAL mode.
*/
defaultJobCreationMode?: JobCreationMode;
}

export interface IntegerTypeCastOptions {
Expand Down Expand Up @@ -320,12 +331,6 @@
* We will create a table with the correct schema, import the public CSV file
* into that table, and query it for data.
*
* This client supports enabling query-related preview features via environmental
* variables. By setting the environment variable QUERY_PREVIEW_ENABLED to the string
* "TRUE", the client will enable preview features, though behavior may still be
* controlled via the bigquery service as well. Currently, the feature(s) in scope
* include: stateless queries (query execution without corresponding job metadata).
*
* @class
*
* See {@link https://cloud.google.com/bigquery/what-is-bigquery| What is BigQuery?}
Expand Down Expand Up @@ -362,19 +367,19 @@
export class BigQuery extends Service {
location?: string;
private _universeDomain: string;
private _enableQueryPreview: boolean;
private _defaultJobCreationMode: JobCreationMode;

createQueryStream(options?: Query | string): ResourceStream<RowMetadata> {

Check warning on line 372 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

'options' is defined but never used
// placeholder body, overwritten in constructor
return new ResourceStream<RowMetadata>({}, () => {});
}

getDatasetsStream(options?: GetDatasetsOptions): ResourceStream<Dataset> {

Check warning on line 377 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

'options' is defined but never used
// placeholder body, overwritten in constructor
return new ResourceStream<Dataset>({}, () => {});
}

getJobsStream(options?: GetJobsOptions): ResourceStream<Job> {

Check warning on line 382 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

'options' is defined but never used
// placeholder body, overwritten in constructor
return new ResourceStream<Job>({}, () => {});
}
Expand Down Expand Up @@ -421,12 +426,8 @@

super(config, options);

const QUERY_PREVIEW_ENABLED = process.env.QUERY_PREVIEW_ENABLED;
this._enableQueryPreview = false;
if (typeof QUERY_PREVIEW_ENABLED === 'string') {
if (QUERY_PREVIEW_ENABLED.toUpperCase() === 'TRUE') {
this._enableQueryPreview = true;
}
if (options.defaultJobCreationMode) {
this._defaultJobCreationMode = options.defaultJobCreationMode;
}

this._universeDomain = universeDomain;
Expand Down Expand Up @@ -1588,7 +1589,7 @@
const parameterMode = is.array(params) ? 'positional' : 'named';
const queryParameters: bigquery.IQueryParameter[] = [];
if (parameterMode === 'named') {
const namedParams = params as {[param: string]: any};

Check warning on line 1592 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

Unexpected any. Specify a different type
for (const namedParameter of Object.getOwnPropertyNames(namedParams)) {
const value = namedParams[namedParameter];
let queryParameter;
Expand Down Expand Up @@ -2236,7 +2237,7 @@

options = extend({job}, queryOpts, options);
if (res && res.jobComplete) {
let rows: any = [];

Check warning on line 2240 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

Unexpected any. Specify a different type
if (res.schema && res.rows) {
rows = BigQuery.mergeSchemaWithRows_(res.schema, res.rows, {
wrapIntegers: options.wrapIntegers || false,
Expand Down Expand Up @@ -2338,7 +2339,7 @@
query: queryObj.query,
useLegacySql: false,
requestId: randomUUID(),
jobCreationMode: 'JOB_CREATION_OPTIONAL',
jobCreationMode: this._defaultJobCreationMode,
reservation: queryObj.reservation,
continuous: queryObj.continuous,
destinationEncryptionConfiguration:
Expand All @@ -2347,8 +2348,9 @@
connectionProperties: queryObj.connectionProperties,
preserveNulls: queryObj.preserveNulls,
};
if (!this._enableQueryPreview) {
delete req.jobCreationMode;
if (queryObj.jobCreationMode) {
// override default job creation mode
req.jobCreationMode = queryObj.jobCreationMode;
}
const {parameterMode, params} = this.buildQueryParams_(
queryObj.params,
Expand Down
9 changes: 6 additions & 3 deletions test/bigquery.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,8 +189,10 @@ describe('BigQuery', () => {
beforeEach(() => {
Object.assign(fakeUtil, originalFakeUtil);
BigQuery = Object.assign(BigQuery, BigQueryCached);
bq = new BigQuery({projectId: PROJECT_ID});
bq._enableQueryPreview = true;
bq = new BigQuery({
projectId: PROJECT_ID,
defaultJobCreationMode: 'JOB_CREATION_OPTIONAL',
});
});

after(() => {
Expand Down Expand Up @@ -3360,6 +3362,7 @@ describe('BigQuery', () => {
labels: {
key: 'value',
},
jobCreationMode: 'JOB_CREATION_REQUIRED',
};
const req = bq.buildQueryRequest_(q, {});
for (const key in req) {
Expand Down Expand Up @@ -3392,7 +3395,7 @@ describe('BigQuery', () => {
labels: {
key: 'value',
},
jobCreationMode: 'JOB_CREATION_OPTIONAL',
jobCreationMode: 'JOB_CREATION_REQUIRED',
formatOptions: {
useInt64Timestamp: true,
},
Expand Down
Loading