Problem: Laggy SSH. Noted particulary over local LAN WiFi connections

This is a problem that affects both Termux and Connectbot, so it's probably not strictly a Termux issue.

Symptoms

Very slow / laggy response to SSH input. Display fails to respond for several seconds, up to 24 measured (see below), and possibly longer experienced. It's frequently faster to disconnect and reconnect (under screen or termux) than wait for the remote system to respond.

I've conducted a number of experiments to try to get some scope of the problem.

Repeated SSH connections, comparing local and remote timestamps

The test: Check the local time, connect to the remote system, check its time, and compare the two (subtracting remote from local, in seconds from epoch). Lack of clock coordination (each should be on a timeserver ... sigh) means that remote is 3 seconds ahead of local. Any value > -3 indicates a lag.

Note the value of 21 (which is to say, a lag of 24 seconds), on the 11th row of output.

$ for i in {0..16}; do check-ssh-lag ; done
 -2  -3  -2  -3  -2  -2  -2  -3  -3  -2  -2  -2   5  -2  -2  -2  -2  -2   2  -2
  1   0  -2   0  -2  -3  -2  -3  -2  -3  -3  -3  -3  -3  -3  -3  -3  -2  -3  -3
 -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -3  -3
 -2  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -2  -3  -2  -3  -3  -3  -3
 -2  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3
 -2  -3  -2  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -2  -3  -2  -3
 -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -2
 -3  -3  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -2  -3  -2  -3  -3  -3  -3
 -3  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -3  -3  -2  -3
 -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -2  -3  -2  -3  -3  -3
 -3  -2  -2  -3  -3  -3  -2  -2  -2   3  -1  21  -2  -2  -3  -2  -3  -3  -3  -3
 -3  -3  -2  -3  -3  -3  -3  -2  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3
 -3  -2  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3
 -3  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3
 -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -2  -3  -2  -3  -3  -3  -3
 -3  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -2  -3  -2  -3  -3
 -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -2  -3  -3  -3  -3  -3  -3  -3  -3

The bash function:

check-ssh-lag () 
{ 
    for ((i=0; i<20; i++ ))
    do
        printf "%3i " "$( echo "-1 * $(date +%s) + $(ssh remote date +%s)" | bc)";
    done;
    echo
}

(ssh-agent is running to provide key-based auth.)

Simultaneously running an interactive shell in a 'screen' split terminal, with a loop printing the date every second, there was no observable lag.

This suggests that it's data being sent from, rather than received by, the android device which is a problem. And not networking.

tcpdump

I've installed tcpdump on the remote host and am watching what it's doing.

I've already noted that whilst locally generated keystrokes aren't being registered remotely, if I'm running a split-screen session with some process generating constant output (e.g., top, or tailing some logfile), I will see the generated output, but cannot send data.

I've been logging tcpdump to a file as below:

tcpdump -c 1000000 -U -C  1 -w /tmp/tcpdump-android-ssh \
    '(src host 192.168.xx.xx) and (port ssh)'

And (in Yet Another Screen Window), am tailing that under watch:

watch 'sudo tcpdump -r /tmp/tcpdump-android-ssh | tail -8 '

What I'll see are apparently a bunch of null-length packets being received, then, after some seconds of delay, the data start to flow again.

The empty packets seem to have length 0, though I want to confirm.

This strongly suggests, though, that communications isn't the problem but ssh on the local (Android) host not sending data is.

Terminal behaviour

One thing I've noticed is that straight text entry is rarely problematic. E.g., open a vim session on the remote host and type:

Now is the time for all good men to come to the aid of their party.

... repeatedly. I don't observe lag.

But ... if I am engaged in a complex editing task (moving around the file repeatedly), or commandline interactions (I do a log of on-shell editing via Readline, etc.), I observe the problem far more frequently.

This suggests that terminal interactions, possibly on Termux itself, are a hangup.

Given that I've also identified Termux lagg in scrollback as an identified issue ... I've got some reason to suspect this here as well.

termux/termux-app#368

Further steps

Identifying whether / how the local ssh client is sending or receiving data would be useful. I suspect that's going to involve debuggers.