[diagramsnet container]: Prevent Chromium zombie processes from stacking #1826
Description
Issue:
When specifying low memory limits for a diagramsnet container in Kubernetes, chromium processes could remain in a zombie state if we send a lot of parallel requests.
The diagramsnet service just stopped working with following error, but the container is still alive.
Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:40619
E.g:
$ seq 10 | xargs -P 10 -I -- curl -k https://kroki.local/diagramsnet/svg/eNpLyUwvSizIUHBXqPZIzcnJ17ULzy_KSanlAgB1EAjQ
Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
at Array.map (<anonymous>)
at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
at Array.map (<anonymous>)
at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)
at runNextTicks (node:internal/process/task_queues:60:5)
at listOnTimeout (node:internal/timers:540:9)
at process.processTimers (node:internal/timers:514:7)Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
at Array.map (<anonymous>)
at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)
at runNextTicks (node:internal/process/task_queues:60:5)
at listOnTimeout (node:internal/timers:540:9)
at process.processTimers (node:internal/timers:514:7)Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
at Array.map (<anonymous>)
at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
at Array.map (<anonymous>)
at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)
Normal state:
/usr/local/kroki $ ps aux
PID USER TIME COMMAND
1 kroki 0:00 node src/index.js
17 kroki 0:00 {chromium} /usr/lib/chromium/chrome --allow-pre-commit-input --disable-background-networking --disable-background-timer-throttl
19 kroki 0:00 /usr/lib/chromium/chromium --type=zygote --no-zygote-sandbox --no-sandbox --headless --headless
20 kroki 0:00 /usr/lib/chromium/chromium --type=zygote --no-sandbox --headless --headless
36 kroki 0:00 /usr/lib/chromium/chromium --type=gpu-process --no-sandbox --disable-dev-shm-usage --disable-breakpad --headless --ozone-platfo
42 kroki 0:00 /usr/lib/chromium/chromium --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=n
46 kroki 0:00 /usr/lib/chromium/chromium --type=renderer --headless --first-renderer-process --no-sandbox --disable-dev-shm-usage --disable-b
Hanging Chromium processes in the zombie state:
/usr/local/kroki $ ps
PID USER TIME COMMAND
1 kroki 0:00 node src/index.js
19 kroki 0:00 [chromium]
20 kroki 0:00 [chromium]
36 kroki 0:00 [chromium]
42 kroki 0:00 [chromium]
46 kroki 0:00 [chromium]
76 kroki 0:00 sh -c clear; (bash || ash || sh)
83 kroki 0:00 ash
113 kroki 0:00 sh
157 kroki 0:00 [chromium]
162 kroki 0:00 [chromium]
168 kroki 0:00 [chromium]
174 kroki 0:00 [chromium]
181 kroki 0:00 [chromium]
243 kroki 0:00 ps
What did you expect to see?
The Chromium processes should be properly handled.