这是indexloc提供的服务,不要输入任何密码
Skip to content

Dynamic DAGs that disappear end up stuck in queued state. #15488

@ashb

Description

@ashb

I can observe the same problem with version 2.0.2:

  • Tasks fail, because a DAG/task has gone missing (we are using dynamically created DAGs, and they can go missing)
  • The scheduler keeps those queued
  • The pool gradually fills up with these queued tasks
  • The whole operation stops, because of this behaviour

My current remedy:

  • Manually remove those queued tasks

My desired solution:

When a DAG/task goes missing while it is queued, it should end up in a failed state.

Originally posted by @lukas-at-harren in #13542 (comment)

Metadata

Metadata

Assignees

Labels

affected_version:2.0Issues Reported for 2.0area:Schedulerincluding HA (high availability) schedulerkind:bugThis is a clearly a bugpriority:mediumBug that should be fixed before next release but would not block a release

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions