Support overriding defaults.yml based on the directory hierarchy
#500
+276
−32
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
087669a to
27a2bca
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #500 +/- ##
==========================================
+ Coverage 92.35% 92.54% +0.18%
==========================================
Files 11 11
Lines 1099 1140 +41
==========================================
+ Hits 1015 1055 +40
- Misses 84 85 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
tatiana
added a commit
that referenced
this pull request
Jul 25, 2025
defaults.yml arguments using directory hierarchydefaults.yml based on the directory hierarchy
This was referenced Jul 25, 2025
2 tasks
pankajkoti
approved these changes
Jul 28, 2025
pankajastro
approved these changes
Jul 28, 2025
tatiana
added a commit
that referenced
this pull request
Jul 28, 2025
Draft
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduce support to multiple
defaults.yamlfiles, allowing properties to be overridden following the proximity to the YAML DAG definition.Use predictable merge behaviour where the level closest to the DAG YAML files wins. Keep the default section in-DAG for fine-grained overrides and as a template pattern for scoped DAGs.
defaults.ymlnot only acceptsdefault_args, but also DAG-level arguments;defaults.ymlper folder, and merging it with the nearest one will get more priority;defaults.ymlconfigurations merged.Closes: #469
The documentation was added in #504.
There are some open questions:
default_args_config_pathanddefault_args_config_dictsupport overriding not onlydefault_args, but also any DAG-level args, we should rename them. I've logged a follow-up ticket: Renamedefault_args_config_pathanddefault_args_config_dict#505default_args_config_pathanddefault_args_config_dictare exposed to users that leverageDagFactory(...).generate_dags(). If we intend to deprecate this method, should we expose these fields viaload_yaml_dags?