Tags: databricks/cli
Tags
direct: Use read/write locking inside DAG (#3745) ## Changes Fix locking inside DAG to use proper read/write lock. ## Why Handle multiple resources referencing one resource correctly. ## Tests New test to trigger the bug: `go test ../../.. -run ^TestAccept$/^bundle$/^resource_deps$/^id_star$/direct -count=50 -failfast` passes on this branch; on main it crashes with: ``` acceptance_test.go:803: Diff: --- bundle/resource_deps/id_star/output.txt +++ /var/folders/5y/9kkdnjw91p11vsqwk0cvmk200000gp/T/TestAcceptbundleresource_depsid_starDATABRICKS_BUNDLE_ENGINE331072887/001/output.txt @@ -1 +1,17 @@ +panic: internal DAG error, concurrent access to "resources.jobs.a" + +goroutine 49 [running]: +github.com/databricks/cli/bundle/deployplan.(*Plan).LockEntry(0x140004d01e0, {0x1400000e6a0, 0x10}) + /Users/denis.bilenko/work/cli-side/bundle/deployplan/plan.go:86 +0x1bc +github.com/databricks/cli/bundle/direct.(*DeploymentBundle).LookupReferenceLocal(0x1400023c828, {0x13?, 0x3?}, 0x1400061a2a0) + /Users/denis.bilenko/work/cli-side/bundle/direct/bundle_plan.go:284 +0x80 +github.com/databricks/cli/bundle/direct.(*DeploymentBundle).resolveReferences(0x1400023c828, {0x1038fec58, 0x140005c12c0}, 0x1400049e840, {0x1400004c1e0, 0x1c}, 0x1) + /Users/denis.bilenko/work/cli-side/bundle/direct/bundle_plan.go:387 +0x36c +github.com/databricks/cli/bundle/direct.(*DeploymentBundle).CalculatePlan.func1({0x140003bdff0, 0x10}, 0x0) + /Users/denis.bilenko/work/cli-side/bundle/direct/bundle_plan.go:119 +0x1b8 +github.com/databricks/cli/libs/dagrun.runWorkerLoop(0x0?, 0x140006ae310, 0x140006ae380, 0x140004ac330) + /Users/denis.bilenko/work/cli-side/libs/dagrun/dagrun.go:235 +0xb8 +created by github.com/databricks/cli/libs/dagrun.(*Graph).Run in goroutine 1 + /Users/denis.bilenko/work/cli-side/libs/dagrun/dagrun.go:184 +0x3d8 + +Exit code: 2 ```
## Release v0.272.0 ### Bundles * Fix processing short pip flags in environment dependencies ([#3708](#3708)) * Add support for referencing local files in -e pip flag for environment dependencies ([#3708](#3708)) * Add error for when an etag is specified in dashboard configuration. Setting etags was never supported / valid in bundles but now users will see this error during validation rather than deployment. ([#3723](#3723)) * Fix PIP flag processing in pipeline environment dependencies ([#3734](#3734))
## Release v0.271.0 ### Bundles * Added support for a "template_dir" option in the databricks_template_schema.json format. ([#3671](#3671)). * Remove resources.apps.config section ([#3680](#3680)) * Prompt for serverless compute in `dbt-sql` template (defaults to `yes`) ([#3668](#3668)) ### API Changes * Added `databricks account account-groups-v2` command group. * Added `databricks account account-iam-v2` command group. * Added `databricks feature-engineering` command group. * Added `databricks shares list-shares` command.
## Release v0.270.0 ### Notable Changes * Add 'databricks bundle plan' command. This command shows the deployment plan for the current bundle configuration without making any changes. ([#3530](#3530)) ### Bundles * Add 'databricks bundle plan' command ([#3530](#3530)) * Add new Lakeflow Pipelines support for bundle generate ([#3568](#3568)) * Fix bundle deploy to not update permissions or grants for unbound resources ([#3642](#3642)) * Introduce new bundle variable: `${workspace.current_user.domain_friendly_name}` ([#3623](#3623)) * Improve the output of bundle run when bundle is not deployed ([#3652](#3652))
## Release v0.267.0 ### CLI * Introduce retries to `databricks psql` command ([#3492](#3492)) * Add rule files for coding agents working on the CLI code base ([#3245](#3245)) ### Dependency updates * Upgrade TF provider to 1.88.0 ([#3529](#3529)) * Upgrade Go SDK to 0.82.0 ### Bundles * Update default-python template to make DB Connect work out of the box for unit tests, using uv to install dependencies ([#3254](#3254)) * Add support for `TaskRetryMode` for continuous jobs ([#3529](#3529)) * Add support for specifying database instance as an application resource ([#3529](#3529)) * Allow referencing job libraries outside bundle root without the need to specify sync root ([#2842](#2842)) * Add top level `run_as` support for Lakeflow Declarative Pipelines ([#3307](#3307))
## Release v0.266.0 ### Notable Changes * Breaking change: DABs now return an error when paths are incorrectly defined relative to the job or pipeline definition location instead of the configuration file location. Previously, the CLI would show a warning and fallback to resolving the path relative to the resource location. Users must update their bundle configurations to define all relative paths relative to the configuration file where the path is specified. See more details here: ([#3225](#3225)) * Add support volumes in Python support ([#3383])(#3383)) ### Bundles * [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](#3225)) * Add support for Lakebase synced database tables in DABs ([#3467](#3467)) * Rename Delta Live Tables to Lakeflow Declarative Pipelines in the default-python template ([#3476](#3476)). * Fixed bundle init not working on Standard tier ([#3496](#3496))
## Release v0.265.0 ### CLI * Fix "cache: token not found" for auth token command ([#3447](#3447)) ### Dependency updates * Upgrade TF provider to 1.87.0 ([#3430](#3430)) * Upgrade Go SDK to 0.81.0 ([#3449](#3449)) ### Bundles * Add support for Lakebase database instances in DABs ([#3283](#3283)) * Add support for Lakebase database catalogs in DABs ([#3436](#3436)) * Improve error message for SDK/API errors ([#3379](#3379)) * Separate generated classes between jobs and pipelines in Python support ([#3428](#3428))
PreviousNext