这是indexloc提供的服务,不要输入任何密码
Skip to content

Support tasks and task_groups as lists #487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 24, 2025
Merged

Conversation

pankajkoti
Copy link
Contributor

@pankajkoti pankajkoti commented Jul 22, 2025

What?

This change adds first-class support for list-based syntax in DAG-Factory YAML:

  1. Tasks as a list
  2. Task Groups as a list

Until now, both sections had to be dictionaries keyed by task_id / group_id. Authors can now opt for a cleaner, order-preserving list format while everything still works in dictionary form.

Why?

The list style is more natural, readable, and aligns with common YAML patterns.

Example before ⬇️

tasks:
  task_1:
    operator: airflow.operators.bash.BashOperator
    bash_command: "echo 1"

Example after ⬇️

tasks:
  - task_id: task_1
    operator: airflow.operators.bash.BashOperator
    bash_command: "echo 1"

Same goes for task_groups:

task_groups:
  - group_name: extract
    tooltip: "extraction steps"

Implementation details

DagBuilder.build() now:
Detects if tasks or task_groups is a list.
Converts each list to the existing dict structure (task_id / group_name key becomes dict key).
Performs validation:

  1. Each entry must be a mapping with the required identifier field.
  2. Duplicate identifiers raise DagFactoryConfigException.
    No downstream code had to change – all later logic still sees a dictionary.

Backwards compatibility

Existing dictionary syntax continues to work unchanged.
Tests covering legacy pattern remain green.

closes: #468

@codecov-commenter
Copy link

codecov-commenter commented Jul 22, 2025

Codecov Report

Attention: Patch coverage is 92.10526% with 3 lines in your changes missing coverage. Please review.

Project coverage is 93.70%. Comparing base (ea0ab42) to head (bdbf8df).

Files with missing lines Patch % Lines
dagfactory/dagbuilder.py 92.10% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #487      +/-   ##
==========================================
- Coverage   93.76%   93.70%   -0.06%     
==========================================
  Files          11       11              
  Lines        1042     1080      +38     
==========================================
+ Hits          977     1012      +35     
- Misses         65       68       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pankajkoti pankajkoti marked this pull request as ready for review July 22, 2025 10:04
@pankajkoti pankajkoti requested a review from a team as a code owner July 22, 2025 10:04
@pankajkoti
Copy link
Contributor Author

I have logged a follow-up ticket: #491 to update example DAGs and docs to reflect authoring tasks as list. Will work on a quick follow-up PR to do that once this change is avaialble and merged. Keeping the scope of this PR limited to ease reviews.

Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a very exciting feature, and it's great it is backwards compatible.
I left an implementation feedback inline, but happy with the feature!

@pankajkoti pankajkoti force-pushed the task-taskgroups-as-list branch from e0457d2 to bdbf8df Compare July 24, 2025 11:26
@pankajkoti pankajkoti merged commit f35955f into main Jul 24, 2025
63 checks passed
@pankajkoti pankajkoti deleted the task-taskgroups-as-list branch July 24, 2025 11:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make tasks a list
3 participants