When start a dag in airflow 3.0.0, scheduler could be crashed but works well in airflow 2.10.4 #50382

fengchutadie · 2025-05-09T07:22:59Z

fengchutadie
May 9, 2025

ENV

Linux: Ubuntu
Python:3.12.10
Apache-airflow: 3.0.0

Issue

I use conda to create the basic python3.12.10 env and then install airflow as

pip install "apache-airflow==3.0.0" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0/constraints-3.12.txt"

use "airflow standalone" to launch it , it create related airflow processes and all works well. but when I start a example dag (tutorial_dag.py), then "airflow scheduler" process would be "defunct" and in webUI, first task in this dag would always be queued.
in console log, it prints:

scheduler  | [2025-05-09T05:45:45.641+0000] {_client.py:1026} INFO - HTTP Request: PATCH http://localhost:8080/execution/task-instances/0196ae44-2cb1-7b4c-893d-23ae27697461/run "HTTP/1.1 504 Gateway Timeout"
scheduler  | 2025-05-09 05:45:45 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=2414628 signal_sent=SIGKILL
scheduler  | [2025-05-09T05:45:45.652+0000] {local_executor.py:96} ERROR - uhoh
scheduler  | Traceback (most recent call last):
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 92, in _run_worker
scheduler  | _execute_work(log, workload)
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 113, in _execute_work
scheduler  | supervise(
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1313, in supervise
scheduler  | process = ActivitySubprocess.start(
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 785, in start
scheduler  | proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 796, in _on_child_started
scheduler  | ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 146, in start
scheduler  | resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/httpx/_client.py", line 1219, in patch
scheduler  | return self.request(
scheduler  | ^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
scheduler  | return copy(f, *args, **kw)
scheduler  | ^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
scheduler  | do = self.iter(retry_state=retry_state)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
scheduler  | result = action(retry_state)
scheduler  | ^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/tenacity/__init__.py", line 420, in exc_check
scheduler  | raise retry_exc.reraise()
scheduler  | ^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/tenacity/__init__.py", line 187, in reraise
scheduler  | raise self.last_attempt.result()
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/concurrent/futures/_base.py", line 449, in result
scheduler  | return self.__get_result()
scheduler  | ^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
scheduler  | raise self._exception
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
scheduler  | result = fn(*args, **kwargs)
scheduler  | ^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 629, in request
scheduler  | return super().request(*args, **kwargs)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/httpx/_client.py", line 827, in request
scheduler  | return self.send(request, auth=auth, follow_redirects=follow_redirects)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
scheduler  | response = self._send_handling_auth(
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
scheduler  | response = self._send_handling_redirects(
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
scheduler  | raise exc
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
scheduler  | hook(response)
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 117, in raise_on_4xx_5xx_with_note
scheduler  | return get_json_error(response) or response.raise_for_status()
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/httpx/_models.py", line 761, in raise_for_status
scheduler  | raise HTTPStatusError(message, request=request, response=self)
scheduler  | httpx.HTTPStatusError: Server error '504 Gateway Timeout' for url 'http://localhost:8080/execution/task-instances/0196ae44-2cb1-7b4c-893d-23ae27697461/run'
scheduler  | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504
scheduler  | Correlation-id=0196b394-096e-7018-9574-5db4fa6afa22
scheduler  | [2025-05-09T05:45:45.833+0000] {scheduler_job_runner.py:1019} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
scheduler  | Traceback (most recent call last):
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/jobs/scheduler_job_runner.py", line 1015, in _execute
scheduler  | self._run_scheduler_loop()
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/jobs/scheduler_job_runner.py", line 1304, in _run_scheduler_loop
scheduler  | executor.heartbeat()
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/traces/tracer.py", line 57, in wrapper
scheduler  | return func(*args, **kwargs)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/executors/base_executor.py", line 286, in heartbeat
scheduler  | self.sync()
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 205, in sync
scheduler  | self._read_results()
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 210, in _read_results
scheduler  | key, state, exc = self.result_queue.get()
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/congma/miniconda3/envs/airflow/lib/python3.12/multiprocessing/queues.py", line 389, in get
scheduler  | return _ForkingPickler.loads(res)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | TypeError: HTTPStatusError.__init__() missing 2 required keyword-only arguments: 'request' and 'response'
scheduler  | [2025-05-09T05:45:45.836+0000] {local_executor.py:216} INFO - Shutting down LocalExecutor; waiting for running tasks to finish.  Signal again if you don't want to wait.

it always happen when I delete ~/airflow several times.

but if I install Airflow2.10.4，this issue would not happen and everything works well, it seems not my gateway issue.

pip install "apache-airflow==2.10.4" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.10.4/constraints-3.12.txt"

Does anyone encouter this issue?

potiuk · 2025-05-09T08:50:50Z

potiuk
May 9, 2025
Collaborator

Please try 3.0.1rc1 -> it's currently being tested and I think there was a case fixed where API calls could go into infinite loop that would resilut in similar issue.

3 replies

fengchutadie May 12, 2025
Author

I just prepare the whole env for version 3.0.1rc1. it seems a little different.

3.0.0, when I trigger the example dag (tutorial_dag.py), the "airflow scheduler" would be defunct in several seconds.
3.0.1rc1, trigger the same dag, it would be in queued status about several minutes and then "airflow scheduler" would be defunct.

but they encounter the same issue, I still can't schedule any dag.

potiuk May 12, 2025
Collaborator

Maybe you simply run out of resources- memory for example. It's impossible to tell what problem you have without you digging deeper and providin more information.

fengchutadie May 13, 2025
Author

it seems it's not about out of resources.
in console log, part of the scheduler log as below

scheduler  | [2025-05-13T01:58:37.100+0000] {_client.py:1026} INFO - HTTP Request: PATCH http://localhost:8080/execution/task-instances/0196c1ed-1bfe-7616-961a-af093a67ca90/run "HTTP/1.1 504 Gateway Timeout"
scheduler  | 2025-05-13 01:58:37 [warning  ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client]
scheduler  | [2025-05-13T01:58:38.897+0000] {_client.py:1026} INFO - HTTP Request: PATCH http://localhost:8080/execution/task-instances/0196c1ed-1bfe-7616-961a-af093a67ca90/run "HTTP/1.1 504 Gateway Timeout"
scheduler  | 2025-05-13 01:58:38 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=3129186 signal_sent=SIGKILL
scheduler  | [2025-05-13T01:58:38.907+0000] {local_executor.py:96} ERROR - uhoh
scheduler  | httpx.HTTPStatusError: Server error '504 Gateway Timeout' for url 'http://localhost:8080/execution/task-instances/0196c1ed-1bfe-7616-961a-af093a67ca90/run'
scheduler  | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504
scheduler  | Correlation-id=0196c75d-8bf5-75ca-bd0b-365e12de83ce
scheduler  | [2025-05-13T01:58:39.917+0000] {scheduler_job_runner.py:1019} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
scheduler  | TypeError: HTTPStatusError.__init__() missing 2 required keyword-only arguments: 'request' and 'response'
scheduler  | [2025-05-13T01:58:39.921+0000] {local_executor.py:223} INFO - Shutting down LocalExecutor; waiting for running tasks to finish.  Signal again if you don't want to wait.
scheduler  | [2025-05-13T01:58:39.935+0000] {scheduler_job_runner.py:1031} INFO - Exited execute loop
scheduler  | TypeError: HTTPStatusError.__init__() missing 2 required keyword-only arguments: 'request' and 'response'
scheduler  | [2025-05-13 01:58:40 +0000] [3129173] [INFO] Shutting down: Master

after serveral times of http request:
HTTP Request: PATCH http://localhost:8080/execution/task-instances/0196c1ed-1bfe-7616-961a-af093a67ca90/run "HTTP/1.1 504 Gateway Timeout", then it would kill scheduler proactively as log

scheduler  | 2025-05-13 01:58:38 [info] Process exited  [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=3129186 signal_sent=SIGKILL

in the host Linux, when I execute
wget http://localhost:8080/execution/task-instances/0196c1ed-1bfe-7616-961a-af093a67ca90/run
it prints

--2025-05-13 02:02:35--  http://localhost:8080/execution/task-instances/0196c1ed-1bfe-7616-961a-af093a67ca90/run
Connecting to 10.158.100.9:8080... connected.
Proxy request sent, awaiting response... 504 Gateway Timeout
Retrying.

and when I get it in another host with browser http://10.57.200.121:8080/execution/task-instances/0196c1ed-1bfe-7616-961a-af093a67ca90/run, it shows
{ "detail": "Method Not Allowed" }
it's not gateway timeout if I replace localhost by 10.57.200.121.

in airflow 2.10.4, it works well and processes in backend are different with processes in airflow 3.0.1rc1, I'm not sure if the difference about process communication mechanism causes this issue.

sehoffmann · 2025-05-13T09:59:08Z

sehoffmann
May 13, 2025

I have the same issue and error. I can not get a simple dag to run without the scheduler crashing. I also get the same HTTP response when manually going to API url in my browser.

1 reply

jtantry Jul 1, 2025

Is there a solution for this? I am facing the same issue for Airflow 3.0.2 and Python 3.12 version.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

When start a dag in airflow 3.0.0, scheduler could be crashed but works well in airflow 2.10.4 #50382

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

When start a dag in airflow 3.0.0, scheduler could be crashed but works well in airflow 2.10.4 #50382

Uh oh!

fengchutadie May 9, 2025

ENV

Issue

Replies: 2 comments · 4 replies

Uh oh!

potiuk May 9, 2025 Collaborator

Uh oh!

fengchutadie May 12, 2025 Author

Uh oh!

potiuk May 12, 2025 Collaborator

Uh oh!

fengchutadie May 13, 2025 Author

Uh oh!

Uh oh!

sehoffmann May 13, 2025

Uh oh!

jtantry Jul 1, 2025

fengchutadie
May 9, 2025

Replies: 2 comments 4 replies

potiuk
May 9, 2025
Collaborator

fengchutadie May 12, 2025
Author

potiuk May 12, 2025
Collaborator

fengchutadie May 13, 2025
Author

sehoffmann
May 13, 2025