Can't run more than 4 SSH sessions to a workspace - connection refused on port 12049 #1929
Description
What happened?
When opening multiple parallel SSH sessions to an already existing workspace, only 4 can be established. The 5th and subsequent connections hang and ultimately fail.
What did you expect to happen instead?
Connect to 5+ ssh session to a same devpod workspace without issue.
How can we reproduce the bug?
- Set up a workspace using the local Docker provider (with gpg-forwarding, ssh-forwarding, using gpg-agent on the host. See ~/.ssh/config excerpt provided)
- Open 5 separate terminal windows.
- In each terminal, run
ssh <workspace-host-name>. - Observe that 4 connect, but the fifth hangs.
~/.ssh/config excerpt
# DevPod Start homelab.devpod
Host homelab.devpod
ForwardAgent yes
LogLevel error
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
HostKeyAlgorithms rsa-sha2-256,rsa-sha2-512,ssh-rsa
ProxyCommand "/usr/local/bin/devpod" ssh --stdio --context default --user vscode homelab --gpg-agent-forwarding
User vscode
# DevPod End homelab.devpod
Full debug trace
This is the trace when trying to connect a fifth session, letting it hang, then terminating from another terminal a previously open session:
devpod ssh homelab --debug
20:46:38 debug Acquire workspace lock...
20:46:38 debug Acquired workspace lock...
20:46:38 debug Execute command locally
20:46:38 debug Successfully connected to host
20:46:38 debug Run container tunnel
20:46:38 info Execute SSH server command: bash -c '/usr/local/bin/devpod' agent container-tunnel --workspace-info 'H4sIAAAAAA.....' --debug
20:46:38 info Use /home/owner/.devpod/agent/contexts/default/workspaces/homelab as workspace dir
20:46:38 info Workspace Folder already exists /home/owner/code/homelab
20:46:38 info Using docker command 'docker'
20:46:38 info execute inject script
20:46:41 info Received line after pong: done
20:46:41 info done inject
20:46:41 info done injecting
20:46:41 info Done InjectAgentAndExecute
20:46:41 info done exec
20:46:42 debug Successfully connected to container
20:46:42 info Execute SSH server command: bash -c su -c "gpg -K" 'vscode'
20:46:42 info Execute SSH server command: bash -c cat /var/run/devpod/result.json
20:46:42 debug Successfully parsed result at /var/run/devpod/result.json
20:46:42 info Execute SSH server command: bash -c '/usr/local/bin/devpod' agent container credentials-server --user 'vscode' --configure-git-helper --configure-docker-helper --debug
20:46:42 debug Received ping from agent
20:46:42 debug Port 12049 not available, exiting
------> HANGS HERE UNTIL I TERMINATE ANOTHER SSH SESSION, then this appears: <--------
20:47:00 info gpg: can't connect to the gpg-agent: End of file
20:47:00 debug [GPG] exporting gpg owner trust from host
20:47:00 debug [GPG] detecting gpg-agent socket path on host
20:47:00 debug [GPG] detected gpg-agent socket path /run/user/1000/gnupg/S.gpg-agent.extra
20:47:00 debug [GPG] no git signkey detected, skipping
20:47:00 debug [GPG] start reverse forward of gpg-agent socket /run/user/1000/gnupg/S.gpg-agent.extra, keeping connection open
20:47:00 info Reverse forwarding local unix//run/user/1000/gnupg/S.gpg-agent.extra to remote unix//run/user/1000/gnupg/S.gpg-agent.extra
20:47:00 info attempt to bind socket /run/user/1000/gnupg/S.gpg-agent.extra
20:47:00 info /run/user/1000/gnupg/S.gpg-agent.extra already exists, removing
20:47:00 info Execute SSH server command: bash -c su -c "/usr/local/bin/devpod agent workspace setup-gpg --ownertrust IyBMaXN0IG...... --socketpath /run/user/1000/gnupg/S.gpg-agent.extra --debug" 'vscode'
20:47:00 info Initializing gpg-agent forwarding
20:47:00 info Fetching public key
20:47:00 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
20:47:01 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
20:47:01 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
20:47:01 info Error calling gpg-public-keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
20:47:01 info Fetch public key: get public gpg keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
20:47:01 info get public gpg keys: Post "http://localhost:12049/gpg-public-keys": dial tcp [::1]:12049: connect: connection refused
20:47:01 debug Connection to container closed
20:47:01 debug Tunnel to host closed
20:47:01 error Error tunneling to container: wait: remote command exited without exit status or exit signal
20:47:01 debug Container tunnel exited
20:47:02 fatal run in container: run gpg agent setup command: Process exited with status 1
tunnel to container
github.com/loft-sh/devpod/pkg/tunnel.(*ContainerTunnel).Run
/home/runner/work/devpod/devpod/pkg/tunnel/container.go:139
github.com/loft-sh/devpod/cmd.(*SSHCmd).jumpContainer
/home/runner/work/devpod/devpod/cmd/ssh.go:308
github.com/loft-sh/devpod/cmd.(*SSHCmd).Run
/home/runner/work/devpod/devpod/cmd/ssh.go:159
github.com/loft-sh/devpod/cmd.NewSSHCmd.func1
/home/runner/work/devpod/devpod/cmd/ssh.go:102
github.com/spf13/cobra.(*Command).execute
/home/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
/home/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1117
github.com/spf13/cobra.(*Command).Execute
/home/runner/work/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1041
github.com/loft-sh/devpod/cmd.Execute
/home/runner/work/devpod/devpod/cmd/root.go:81
main.main
/home/runner/work/devpod/devpod/main.go:6
runtime.main
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.linux-amd64/src/runtime/proc.go:271
runtime.goexit
/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.6.linux-amd64/src/runtime/asm_amd64.s:1695