FORBIDDEN access to worker's logs #17251
-
|
Apache Airflow version: Airflow Web UI is not able to fetch logs (actually stopped working) from the worker's container. It fails with the following error:
When I checked the logs at the worker container, I can see all logs at the worker's container. What could be wrong here? |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 10 replies
-
|
You need to have Webserver secret key the same in webserver and workers: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, secret key helped to resolve my problem |
Beta Was this translation helpful? Give feedback.
-
|
@denisvolokh Can you tell me how you fixed it? |
Beta Was this translation helpful? Give feedback.
-
|
Ref: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#webserver-secret-key |
Beta Was this translation helpful? Give feedback.
-
|
Good morning, Same error as in the first message from @denisvolokh. *** Log file does not exist: <path_to_logs>/logs/dag_id=example_branch_labels/run_id=manual__2022-05-09T09:05:11.152836+00:00/task_id=task1/attempt=1.log
*** Fetching from: http://<server_name>:8793/log/dag_id=example_branch_labels/run_id=manual__2022-05-09T09:05:11.152836+00:00/task_id=task1/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://<server_name>:8793/log/dag_id=example_branch_labels/run_id=manual__2022-05-09T09:05:11.152836+00:00/task_id=task1/attempt=1.log'
For more information check: https://httpstatuses.com/403What configuration could I be missing? thank you very much, |
Beta Was this translation helpful? Give feedback.
-
|
Seting TIME not timezone on all machines is important. Make sure time is synchronized on all your machines. Timezone does not matter, but syncing time (with ntp for example) is crucial. |
Beta Was this translation helpful? Give feedback.
-
|
Also -you might have some firewalls in between your webserver and workers - so you need to track down any networking you have between webserver and worker. |
Beta Was this translation helpful? Give feedback.
-
|
Another thing is - maybe you somewhat override the secret. There are many ways airflow configuration variables are set - https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html. - variables, secrets, _CMD. Maybe you have different settings outside of teh .cfg - or maybe you are using different .cfg file on one of your webserver/worker altogether. I saw that happening in the past. |
Beta Was this translation helpful? Give feedback.
-
|
Good morning, Thank you very much for your help. |
Beta Was this translation helpful? Give feedback.
-
|
@mnaranjorion : Could you please share details on how you fixed configuration problem with NTP? |
Beta Was this translation helpful? Give feedback.
-
|
firstly login to all airflow containers like scheduler,webserver,worker and see if you see same time . if yes it means time is already synchronised. build: .environment: |
Beta Was this translation helpful? Give feedback.
-
|
Ref: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#webserver-secret-key |
Beta Was this translation helpful? Give feedback.
You need to have Webserver secret key the same in webserver and workers: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key