state-changed-externally error. reproducable #52813

MagicFrogSJTU · 2025-07-03T02:52:59Z

MagicFrogSJTU
Jul 3, 2025

Apache Airflow version

3.0.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

when deploying a standalone airflow app, running dags will get random failure due to state-changed-externally.

On UI:

<html>
<body>
<!--StartFragment-->
state mismatch |   | Executor LocalExecutor(parallelism=32) reported that the task instance <TaskInstance: my_test_dag.test1 manual__2025-07-03T02:46:50.167047+00:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally
-- | -- | --


<!--EndFragment-->
</body>
</html>

On backend log

api-server | 2025-07-03 02:47:02 [warning  ] Cannot start Task Instance in invalid state previous_state=scheduled ti_id=0197ce2e-19c7-7387-9504-4f9af0dce503
api-server | INFO:     127.0.0.1:45456 - "PATCH /execution/task-instances/0197ce2e-19c7-7387-9504-4f9af0dce503/run HTTP/1.1" 409 Conflict
scheduler  | [2025-07-03T02:47:02.470+0000] {_client.py:1026} INFO - HTTP Request: PATCH http://localhost:8080/execution/task-instances/0197ce2e-19c7-7387-9504-4f9af0dce503/run "HTTP/1.1 409 Conflict"
scheduler  | 2025-07-03 02:47:02 [warning  ] Server error                   [airflow.sdk.api.client] detail={'detail': {'reason': 'invalid_state', 'message': 'TI was not in a state where it could be marked as running', 'previous_state': 'scheduled'}}
scheduler  | 2025-07-03 02:47:02 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGKILL: -9> pid=181 signal_sent=SIGKILL
scheduler  | [2025-07-03T02:47:02.478+0000] {local_executor.py:96} ERROR - uhoh
scheduler  | Traceback (most recent call last):
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 92, in _run_worker
scheduler  | _execute_work(log, workload)
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py", line 120, in _execute_work
scheduler  | supervise(
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1540, in supervise
scheduler  | process = ActivitySubprocess.start(
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 800, in start
scheduler  | proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info)
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 811, in _on_child_started
scheduler  | ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 149, in start
scheduler  | resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json())
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1219, in patch
scheduler  | return self.request(
scheduler  | ^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
scheduler  | return copy(f, *args, **kw)
scheduler  | ^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
scheduler  | do = self.iter(retry_state=retry_state)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
scheduler  | result = action(retry_state)
scheduler  | ^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in <lambda>
scheduler  | self._add_action_func(lambda rs: rs.outcome.result())
scheduler  | ^^^^^^^^^^^^^^^^^^^
scheduler  | File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
scheduler  | return self.__get_result()
scheduler  | ^^^^^^^^^^^^^^^^^^^
scheduler  | File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
scheduler  | raise self._exception
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
scheduler  | result = fn(*args, **kwargs)
scheduler  | ^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 707, in request
scheduler  | return super().request(*args, **kwargs)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 827, in request
scheduler  | return self.send(request, auth=auth, follow_redirects=follow_redirects)
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
scheduler  | response = self._send_handling_auth(
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
scheduler  | response = self._send_handling_redirects(
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
scheduler  | raise exc
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
scheduler  | hook(response)
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 120, in raise_on_4xx_5xx_with_note
scheduler  | return get_json_error(response) or response.raise_for_status()
scheduler  | ^^^^^^^^^^^^^^^^^^^^^^^^
scheduler  | File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 110, in get_json_error
scheduler  | raise err
scheduler  | airflow.sdk.api.client.ServerResponseError: Server returned error
scheduler  | Correlation-id=0197ce2e-2e3c-7586-bb6a-fbb887a2ce8d
api-server | INFO:     10.94.131.213:9912 - "GET /api/v2/eventLogs?limit=50&offset=0&order_by=-when&dag_id=my_test_dag&task_id=test1&run_id=manual__2025-07-03T02%3A46%3A50.167047%2B00%3A00 HTTP/1.1" 200 OK
scheduler  | [2025-07-03T02:47:02.595+0000] {scheduler_job_runner.py:810} INFO - Received executor event with state failed for task instance TaskInstanceKey(dag_id='my_test_dag', task_id='test1', run_id='manual__2025-07-03T02:46:57.193275+00:00', try_number=1, map_index=-1)
scheduler  | [2025-07-03T02:47:02.601+0000] {scheduler_job_runner.py:852} INFO - TaskInstance Finished: dag_id=my_test_dag, task_id=test1, run_id=manual__2025-07-03T02:46:57.193275+00:00, map_index=-1, run_start_date=None, run_end_date=None, run_duration=None, state=queued, executor=LocalExecut
or(parallelism=32), executor_state=failed, try_number=1, max_tries=0, pool=default_pool, queue=default, priority_weight=1, operator=_PythonDecoratedOperator, queued_dttm=2025-07-03 02:47:02.421256+00:00, scheduled_dttm=2025-07-03 02:47:02.345147+00:00,queued_by_job_id=2, pid=None
scheduler  | [2025-07-03T02:47:02.602+0000] {scheduler_job_runner.py:925} ERROR - DAG 'my_test_dag' for task instance <TaskInstance: my_test_dag.test1 manual__2025-07-03T02:46:57.193275+00:00 [queued]> not found in serialized_dag table
scheduler  | [2025-07-03T02:47:02.602+0000] {taskinstance.py:1882} ERROR - Executor LocalExecutor(parallelism=32) reported that the task instance <TaskInstance: my_test_dag.test1 manual__2025-07-03T02:46:57.193275+00:00 [queued]> finished with state failed, but the task instance's state attribu
te is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally
scheduler  | [2025-07-03T02:47:02.605+0000] {taskinstance.py:2005} INFO - Marking task as FAILED. dag_id=my_test_dag, task_id=test1, run_id=manual__2025-07-03T02:46:57.193275+00:00, logical_date=20250703T024656, start_date=, end_date=20250703T024702
scheduler  | [2025-07-03T02:47:02.794+0000] {dagrun.py:1147} INFO - Marking run <DagRun my_test_dag @ 2025-07-03 02:46:56.418000+00:00: manual__2025-07-03T02:46:57.193275+00:00, state:running, queued_at: 2025-07-03 02:46:57.201924+00:00. run_type: manual> failed
scheduler  | [2025-07-03T02:47:02.794+0000] {dagrun.py:1226} INFO - DagRun Finished: dag_id=my_test_dag, logical_date=2025-07-03 02:46:56.418000+00:00, run_id=manual__2025-07-03T02:46:57.193275+00:00, run_start_date=2025-07-03 02:47:02.234724+00:00, run_end_date=2025-07-03 02:47:02.794287+00:00
, run_duration=0.559563, state=failed, run_type=manual, data_interval_start=2025-07-03 02:46:56.418000+00:00, data_interval_end=2025-07-03 02:46:56.418000+00:00,

What you think should happen instead?

No response

How to reproduce

#!/bin/bash
set -xeuo pipefail

rm -r test_app/ || true
mkdir -p test_app/dags

cat <<EOF > test_app/dags/test_app.py
import pendulum
from airflow import DAG
from airflow.sdk import Param
from airflow.decorators import task

@task.python
def test1():
    print(1111)

with DAG(
    dag_id="my_test_dag",
    schedule=None,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
    tags=["cd", "deployment"],
    max_active_runs=1,
    max_active_tasks=1,
):
    t = test1()
EOF

cat <<EOF > test_app/simple_auth_manager_passwords.json.generated
{"admin": "admin"}
EOF

docker run \
    -it --rm \
    -p 8080:8080 \
    -u $(id -u):0 \
    -v `realpath test_app`:/home/test_app/ \
    -e AIRFLOW_HOME=/home/test_app/ \
    -e AIRFLOW__CORE__LOAD_EXAMPLES=false \
    --name test-airflow-app \
    apache/airflow:3.0.2 \
    standalone

then,

open http://127.0.0.1:8080
enable the my_test_dag
manually trigger the dag multiple times and quickly.

Operating System

PRETTY_NAME="Ubuntu 24.04.2 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.2 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble LOGO=ubuntu-logo

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

potiuk · 2025-07-03T17:20:33Z

potiuk
Jul 3, 2025
Collaborator

Tnis means that SOMETHING killed your task forcefully - the erorrs you show are not the "cause" of the problem but "effect" of the task being killed.

Now - why it is killed is something that only you can find out by inspecting your deployment logs (whatever deployment you use). It might be lack of memory, hardware failure, or a library that uses illegal instructions in processor causing SIGKILL (though that more likely would generate SIGSEGV). Generally somewhere, something is sending -9 kill signal to your process.... And without you digging in the logs of your deployment and finding any log indicating that something killed the task and why - there might be only guesses.

I converted it into a discussion and likely it should be continued after you gather a bit more evidences from your deployment.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

state-changed-externally error. reproducable #52813

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

state-changed-externally error. reproducable #52813

Uh oh!

MagicFrogSJTU Jul 3, 2025

Apache Airflow version

If "Other Airflow 2 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Replies: 1 comment

Uh oh!

potiuk Jul 3, 2025 Collaborator

MagicFrogSJTU
Jul 3, 2025

potiuk
Jul 3, 2025
Collaborator