Deploy - MQTT gateway disconnection cause hummingbot degraded performance #87
Description
Describe the bug
When testing deploy, observed that when deploy needs to be down/up or restarted the mqtt broker gateway connection is unresponsive causing degraded performance which can be easily reproduce when its trying to establish connection or status command
After reconnection, check backend-api logs:
ConnectionError
INFO: Started server process [17]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2025-04-18 13:31:11,929 - root - ERROR - Error writing configs: [Errno 2] No such file or directory: '/opt/conda/envs/backend-api/lib/python3.12/site-packages/conf/conf_client.yml'
Traceback (most recent call last):
File "/opt/conda/envs/backend-api/lib/python3.12/site-packages/hummingbot/client/config/config_helpers.py", line 837, in save_to_yml
with open(yml_path, "w", encoding="utf-8") as outfile:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/envs/backend-api/lib/python3.12/site-packages/conf/conf_client.yml'
hummingbot logs:
2025-04-18 13:09:59,001 - 17 - hummingbot.strategy.script_strategy_base - INFO - Restarting controller pmm-okx-xrpusdt_0.1.
2025-04-18 13:10:18,791 - 17 - hummingbot.client.hummingbot_application - INFO - Creating the clock with tick size: 1.0
2025-04-18 13:10:18,792 - 17 - hummingbot.client.hummingbot_application - INFO - start command initiated.
2025-04-18 13:10:18,926 - 17 - hummingbot.connector.exchange.okx.okx_exchange.OkxExchange - INFO - Network status has changed to NetworkStatus.CONNECTED. Starting networking...
2025-04-18 13:10:19,000 - 17 - hummingbot.strategy.script_strategy_base - WARNING - okx is not ready. Please wait...
2025-04-18 13:10:19,048 - 17 - hummingbot.core.data_type.order_book_tracker - INFO - Initialized order book for XRP-USDT. 1/1 completed.
2025-04-18 13:10:19,407 - 17 - hummingbot.connector.exchange.okx.okx_api_order_book_data_source.OkxAPIOrderBookDataSource - INFO - Subscribed to public order book and trade channels...
2025-04-18 13:10:19,508 - 17 - hummingbot.connector.exchange.okx.okx_api_user_stream_data_source.OkxAPIUserStreamDataSource - INFO - Subscribed to private account and orders channels...
2025-04-18 13:10:20,001 - 17 - hummingbot.strategy.script_strategy_base - WARNING - okx is not ready. Please wait...
2025-04-18 13:10:20,932 - 17 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT BUY order 93027a12dac34fBCBXPUT66d23db8f11 for 7.291919 XRP-USDT at 2.0570.
2025-04-18 13:10:20,941 - 17 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1744981820.0, "type": "OrderType.LIMIT", "trading_pair": "XRP-USDT", "amount": "7.291919", "price": "2.0570", "order_id": "93027a12dac34fBCBXPUT66d23db8f11", "creation_timestamp": 1744981820.0, "exchange_order_id": "2432005801360433154", "leverage": 1, "position": "NIL", "event_name": "BuyOrderCreatedEvent", "event_source": "okx"}
2025-04-18 13:10:20,941 - 17 - hummingbot.connector.client_order_tracker - INFO - Created LIMIT SELL order 93027a12dac34fBCSXPUTdb7fc95e2df for 7.147525 XRP-USDT at 2.0986.
2025-04-18 13:10:20,949 - 17 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1744981820.0, "type": "OrderType.LIMIT", "trading_pair": "XRP-USDT", "amount": "7.147525", "price": "2.0986", "order_id": "93027a12dac34fBCSXPUTdb7fc95e2df", "creation_timestamp": 1744981820.0, "exchange_order_id": "2432005801393987584", "leverage": 1, "position": "NIL", "event_name": "SellOrderCreatedEvent", "event_source": "okx"}
2025-04-18 13:10:21,520 - 17 - hummingbot.core.rate_oracle.rate_oracle - INFO - Network status has changed to NetworkStatus.CONNECTED. Starting networking...
2025-04-18 13:10:55,002 - 17 - hummingbot.strategy.script_strategy_base - INFO - Manual cash out for controller pmm-okx-xrpusdt_0.1.
2025-04-18 13:10:55,028 - 17 - hummingbot.strategy.script_strategy_base - INFO - (XRP-USDT) Canceling the limit order 93027a12dac34fBCBXPUT66d23db8f11. [clock=2025-04-18 13:10:55+00:00]
2025-04-18 13:10:55,028 - 17 - hummingbot.strategy.script_strategy_base - INFO - (XRP-USDT) Canceling the limit order 93027a12dac34fBCSXPUTdb7fc95e2df. [clock=2025-04-18 13:10:55+00:00]
2025-04-18 13:10:55,087 - 17 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1744981855.0, "order_id": "93027a12dac34fBCBXPUT66d23db8f11", "exchange_order_id": "2432005801360433154", "event_name": "OrderCancelledEvent", "event_source": "okx"}
2025-04-18 13:10:55,087 - 17 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 93027a12dac34fBCBXPUT66d23db8f11.
2025-04-18 13:10:55,183 - 17 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1744981855.0, "order_id": "93027a12dac34fBCSXPUTdb7fc95e2df", "exchange_order_id": "2432005801393987584", "event_name": "OrderCancelledEvent", "event_source": "okx"}
2025-04-18 13:10:55,184 - 17 - hummingbot.connector.client_order_tracker - INFO - Successfully canceled order 93027a12dac34fBCSXPUTdb7fc95e2df.
2025-04-18 13:19:58,625 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:20:03,667 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: cannot schedule new futures after shutdown. Sleeping 10 seconds before retry.
2025-04-18 13:20:13,672 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:20:18,673 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: . Sleeping 10 seconds before retry.
2025-04-18 13:20:28,675 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:21:38,745 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: cannot schedule new futures after shutdown. Sleeping 10 seconds before retry.
2025-04-18 13:21:48,746 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:21:53,748 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: . Sleeping 10 seconds before retry.
2025-04-18 13:22:03,749 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:23:13,821 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: cannot schedule new futures after shutdown. Sleeping 10 seconds before retry.
2025-04-18 13:23:23,822 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:23:28,823 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: . Sleeping 10 seconds before retry.
2025-04-18 13:23:38,825 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:24:48,893 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: cannot schedule new futures after shutdown. Sleeping 10 seconds before retry.
2025-04-18 13:24:58,895 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:25:03,896 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: . Sleeping 10 seconds before retry.
2025-04-18 13:25:13,897 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:26:23,978 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: cannot schedule new futures after shutdown. Sleeping 10 seconds before retry.
2025-04-18 13:26:33,980 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:26:38,982 - 17 - hummingbot.client.hummingbot_application - ERROR - MQTT Gateway failed to reconnect: . Sleeping 10 seconds before retry.
2025-04-18 13:26:48,983 - 17 - hummingbot.client.hummingbot_application - WARNING - MQTT Gateway is disconnected, attempting to reconnect.
2025-04-18 13:27:54,092 - 17 - hummingbot.connector.exchange.okx.okx_api_user_stream_data_source.OkxAPIUserStreamDataSource - WARNING - The websocket connection was closed (Cannot write to closing transport)
2025-04-18 13:27:54,623 - 17 - hummingbot.connector.exchange.okx.okx_api_user_stream_data_source.OkxAPIUserStreamDataSource - INFO - Subscribed to private account and orders channels...
Steps to reproduce bug
- Setup deploy on docker
- Deploy any instance using staging image 2.5.0
- Run it for couple of minutes
- On terminal you can docker down/up or
docker compose restart - Observe behavior of hummingbot instance when attached again