[Question]: Increase completion timeout to prevent crash? #1417
Description
How are you running AnythingLLM?
Docker (local)
What happened?
Using LocalAI for backend -- loading the llama3 70b model, Anything LLM container crashed with a socket timeout.
2024-05-15 22:24:20 [TELEMETRY SENT] {
2024-05-15 22:24:20 event: 'sent_chat',
2024-05-15 22:24:20 distinctId: 'be3ac3d9-aa83-4458-ae1a-583a3fcc909b',
2024-05-15 22:24:20 properties: {
2024-05-15 22:24:20 multiUserMode: false,
2024-05-15 22:24:20 LLMSelection: 'localai',
2024-05-15 22:24:20 Embedder: 'openai',
2024-05-15 22:24:20 VectorDbSelection: 'lancedb',
2024-05-15 22:24:20 runtime: 'docker'
2024-05-15 22:24:20 }
2024-05-15 22:24:20 }
2024-05-15 22:24:20 [Event Logged] - sent_chat
2024-05-15 23:04:01 Cannonball results 3511 -> 470 tokens.
2024-05-15 23:04:01 Cannonball results 356 -> 286 tokens.
2024-05-15 23:04:53 [TELEMETRY SENT] {
2024-05-15 23:04:53 event: 'sent_chat',
2024-05-15 23:04:53 distinctId: 'be3ac3d9-aa83-4458-ae1a-583a3fcc909b',
2024-05-15 23:04:53 properties: {
2024-05-15 23:04:53 multiUserMode: false,
2024-05-15 23:04:53 LLMSelection: 'localai',
2024-05-15 23:04:53 Embedder: 'openai',
2024-05-15 23:04:53 VectorDbSelection: 'lancedb',
2024-05-15 23:04:53 runtime: 'docker'
2024-05-15 23:04:53 }
2024-05-15 23:04:53 }
2024-05-15 23:04:53 [Event Logged] - sent_chat
2024-05-15 23:25:00 node:internal/process/promises:288
2024-05-15 23:25:00 triggerUncaughtException(err, true /* fromPromise */);
2024-05-15 23:25:00 ^
2024-05-15 23:25:00
2024-05-15 23:25:00 Error: Socket timeout
2024-05-15 23:25:00 at Socket.onTimeout (/app/server/node_modules/agentkeepalive/lib/agent.js:350:23)
2024-05-15 23:25:00 at Socket.emit (node:events:529:35)
2024-05-15 23:25:00 at Socket._onTimeout (node:net:598:8)
2024-05-15 23:25:00 at listOnTimeout (node:internal/timers:569:17)
2024-05-15 23:25:00 at process.processTimers (node:internal/timers:512:7) {
2024-05-15 23:25:00 code: 'ERR_SOCKET_TIMEOUT',
2024-05-15 23:25:00 timeout: 601000
2024-05-15 23:25:00 }
2024-05-15 23:25:00
2024-05-15 23:25:00 Node.js v18.19.1
A timeout I think is fine , it takes a while to load. However I didn't expect the container to crash, I kind of expected just to re-initiate the thread.
Are there known steps to reproduce?
I think this should be reproducable with any load time of greater than 10 minutes.