这是indexloc提供的服务,不要输入任何密码
Skip to content

Airflow DAG version keeps increasing #58310

@awakener1986

Description

@awakener1986

Apache Airflow version

Other Airflow 2/3 version (please specify below)

If "Other Airflow 2/3 version" selected, which one?

3.1.0

What happened?

We have a DAG which each time it is triggered would create 3 DAG versions. Question is why is this happening , as the impact is that the serialized_dag table is growing and would require clean up

Image

What you think should happen instead?

would like to understand how does DAG version being triggered , we would not like to have multiple versions

How to reproduce

in this example i am not sure what triggered the DAG versions being increased , but basically we are using some custom operators and looping through a list to generate python operators

Operating System

RHEL

Versions of Apache Airflow Providers

apache-airflow==3.1.0rc2
apache-airflow-core==3.1.0rc2
apache-airflow-providers-amazon==9.14.0
apache-airflow-providers-celery==3.12.3
apache-airflow-providers-cncf-kubernetes==10.5.0
apache-airflow-providers-common-compat==1.7.4
apache-airflow-providers-common-io==1.6.3
apache-airflow-providers-common-messaging==2.0.0
apache-airflow-providers-common-sql==1.28.1
apache-airflow-providers-docker==4.4.3
apache-airflow-providers-elasticsearch==6.3.3
apache-airflow-providers-fab==2.4.3
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.8
apache-airflow-providers-google==18.0.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-http==5.3.4
apache-airflow-providers-microsoft-azure==12.7.1
apache-airflow-providers-microsoft-mssql==4.3.2
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openlineage==2.7.1
apache-airflow-providers-postgres==6.3.0
apache-airflow-providers-redis==4.3.1
apache-airflow-providers-sendgrid==4.1.3
apache-airflow-providers-sftp==5.4.0
apache-airflow-providers-slack==9.3.0
apache-airflow-providers-smtp==2.2.1
apache-airflow-providers-snowflake==6.5.4
apache-airflow-providers-ssh==4.1.4
apache-airflow-providers-standard==1.8.0
apache-airflow-task-sdk==1.1.0rc2

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

this issue occurs everytime DAG is triggered ( scheduled daily at midnight ) and will generate 3 versions

Image Image

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions