Intermittent "JWT token is not valid: Signature has expired" and PendingRollbackError after upgrade to FAB 5 #57065
Description
Apache Airflow version
3.1.0
If "Other Airflow 2/3 version" selected, which one?
No response
What happened?
After upgrading to apache-airflow-providers-fab==3.0.0, which uses Flask AppBuilder 5, we started observing intermittent authentication-related crashes that make the Airflow Api Server unavailable.
The issue appears to be triggered by an expired or invalid JWT token, after which the Flask AppBuilder authentication manager fails to recover gracefully and raises a PendingRollbackError in SQLAlchemy.
2025-10-22T07:25:37.653971Z [error ] JWT token is not valid: Signature has expired [airflow.api_fastapi.auth.managers.base_auth_manager] loc=base_auth_manager.py:107
INFO: 10.67.70.212:57914 - "GET /ui/config HTTP/1.1" 401 Unauthorized
2025-10-22T07:25:37.824145Z [error ] Exception on /login/ [GET] [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1821, in full_dispatch_request
rv = self.preprocess_request()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2313, in preprocess_request
rv = self.ensure_sync(before_func)()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/security_manager.py", line 57, in before_request
g.user = get_auth_manager().get_user()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 278, in get_user
if current_user.is_anonymous and getattr(g, "user", None) is not None and not g.user.is_anonymous:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line 316, in __get__
obj = instance._get_current_object()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line 520, in _get_current_object
return get_name(local()) # type: ignore
^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 25, in <lambda>
current_user = LocalProxy(lambda: _get_user())
^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 370, in _get_user
current_app.login_manager._load_user()
File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py", line 364, in _load_user
user = self._user_callback(user_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
user = self.get_user_by_id(int(pk))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
return self.session.get(self.user_model, pk)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2853, in get
return self._get_impl(
^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2975, in _get_impl
return db_load_fn(
^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity
session.execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
result = conn._execute_20(statement, params or {}, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
conn = self._revalidate_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 650, in _revalidate_connection
self._invalid_transaction()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 622, in _invalid_transaction
raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)
After this occurs, the Api Server becomes unresponsive until it’s restarted.
What you think should happen instead?
The system should gracefully handle expired JWT tokens - e.g., invalidate the session, redirect the user to /login, and not leave SQLAlchemy sessions in a broken state.
How to reproduce
Unfortunately, this issue is intermittent - it’s not clear how to consistently reproduce it.
It tends to occur when:
- The user session has expired and the UI tries to refresh(/ui/config or /login/),
- Or when multiple requests hit the webserver simultaneously after an idle period.
Operating System
Debian
Versions of Apache Airflow Providers
apache-airflow==3.1.0
apache-airflow-providers-google==18.0.0
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-slack==9.3.0
apache-airflow-providers-vertica==4.1.2
apache-airflow-providers-apache-livy==4.4.3
apache-airflow-providers-ssh==4.1.4
apache-airflow-providers-common-sql==1.28.1
apache-airflow-providers-cncf-kubernetes==10.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-standard==1.9.0
apache-airflow-providers-postgres==6.3.0
apache-airflow-providers-apache-cassandra==3.8.2
apache-airflow-providers-fab==3.0.0
apache-airflow-providers-amazon==9.15.0
Deployment
Official Apache Airflow Helm Chart
Deployment details
airflow 3.1.0 in k8s deployed with official chart. We're also use pgbouncer for connection pooling
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct