feat(firestore-bigquery-exports) add an option to skip BigQuery init steps #2469
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Pull Request Overview
This PR adds a new --skip-init command-line option to the firestore-bigquery-export import script that allows users to bypass BigQuery initialization steps during backfill operations. This addresses cases where the BigQuery table and view are already properly configured and don't need to be altered.
- Adds a new boolean
skipInitoption to the CLI configuration interface - Implements conditional logic to skip
initializeDataSinkwhen the option is enabled - Updates both command-line argument parsing and interactive prompt handling
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| types.ts | Adds skipInit boolean property to the CliConfig interface |
| program.ts | Adds --skip-init command-line option definition |
| index.ts | Implements conditional logic to skip initialization when option is enabled |
| config.ts | Adds interactive prompt for skip initialization and integrates the option into config parsing |
| "--skip-init [true|false]", | ||
| "Whether to skip BigQuery init steps." |
There was a problem hiding this comment.
The option syntax [true|false] suggests optional values, but boolean flags typically work better as simple flags without values. Consider using "--skip-init" without the value specification, or use "--skip-init <boolean>" if you want to require a value.
| "--skip-init [true|false]", | |
| "Whether to skip BigQuery init steps." | |
| "--skip-init", | |
| "Skip BigQuery init steps.", | |
| false |
Copilot uses AI. Check for mistakes.
| @@ -270,6 +276,7 @@ export async function parseConfig(): Promise<CliConfig | CliConfigError> { | |||
| rawChangeLogName, | |||
| cursorPositionFile, | |||
| failedBatchOutput: program.failedBatchOutput, | |||
| skipInit: program.skipInit, | |||
There was a problem hiding this comment.
The program.skipInit value needs to be properly converted to a boolean. Commander.js may return a string or undefined for optional boolean arguments, which could cause issues. Consider using program.skipInit === true || program.skipInit === 'true' or similar boolean conversion.
Copilot uses AI. Check for mistakes.
This feature would solve this issue: #2468
When running backfill command with
@firebaseextensions/fs-bq-import-collection, there is no reason to alter existing BigQuery table or view, when everything is working fine.This MR adds an option
--skip-initto skip those steps (alterning clustering / partitionning of the raw_changelog, creating the latest view).