SSH - the Secure SHell

How can an SSH connection succeed / fail intermittently with no error message despite verbose mode ?

Situation :

Details :

Looks like you've done nothing wrong. The problem must be :

Solution :

There are chances you actually shot your own foot with hardened security rules like this .

How to mount / umount a SFTP directory ?

  • SFTP != FTPS
  • SFTP is what you do while using scp (and also WinSCP or FileZilla — in their respective SFTP modes (i.e. SSH-based file transfer), since they support other protocols as well).
For a quick / one-time use, consider using the SFTP client built in many file managers (Caja has one) by opening :

mount, umount and CLI access :

apt install sshfs
As a non-root user :
interactively or with SSH keys
sshfs stuart@stfpServer:/path/to/remote/directory /mount/point
password-based authentication (this solution is a little dirty)
echo password | sshfs -o password_stdin stuart@stfpServer:/path/to/remote/directory /mount/point
As a non-root user :
fusermount -u /mount/point

How to automate password-based SSH logins ?

Situation :

Details :

For reasons unrelated to the current article, I also have inter-operability constraints between GNU/Linux and Windows, and when saying KeePass, I mean : which both can handle .kbdx files.

Solution :

The idea is to :

  1. chain SSH logins from a single command-line (source) :
    ssh -At hopLogin@hop ssh -A targetLogin@target
  2. store hosts + credentials in KeePass, avoiding repeating stuff thanks to field references
  3. let KeePass build and auto-type the full SSH connection command, based on a template like :

    ssh -At {REF:U@I:C03D4B434F400B4C843F682A994B0A02}@{REF:A@I:C03D4B434F400B4C843F682A994B0A02}{ENTER}{DELAY 100}{REF:P@I:C03D4B434F400B4C843F682A994B0A02}{ENTER}ssh -At {USERNAME}@{URL}{ENTER}{DELAY 100}{PASSWORD}

    {whatever} placeholders will be replaced by the value of the named variable or key or action :
    • USERNAME : username of the current entry
    • REF:U@I:C03D4B434F400B4C843F682A994B0A02 : reference to the value of another entry (details)
    • ENTER :
    • ...

Auto-type the SSH connection command :

  1. open the KeePass vault and select the entry you'd like to auto-type
  2. give the focus to the window that will receive the simulated keystrokes
  3. give the focus back to KeePass and hit Ctrl-v to fire Perform Auto-Type
  4. if this fails complaining The 'xdotool' utility/package is required for auto-type.
  5. see the magic happen
  • During setup / tests, you may send simulated keystrokes to a text editor to see the generated SSH connection command and debug the auto-type template. Keep in mind your passwords will be echo'ed to screen.
  • Don't forget to connect manually to each SSH server (gateways + targets), since you'll be prompted to accept the SSH host keys.
don't mix sshpass and "auto-type" !
You may have landed here after having automated (or tried to) password-based SSH logins with sshpass. Your process may have lines such as :
sshpass -p 'password' ssh kevin@sshServer
If you plan to use any kind of "auto-type" feature, do NOT use it to generate SSH connection commands including sshpass, otherwise you will : Actually, you don't even need sshpass anymore since the "auto-type" feature will "type" the password for you (for interactive logins only, I'm afraid ). As for KeePass, you'll end with auto-type sequences like :

The 'xdotool' utility/package is required for auto-type.

apt install xdotool


Starting ssh-agent :

  • ssh-agent is installed with openssh-client :
    dpkg -S $(which ssh-agent)
    openssh-client: /usr/bin/ssh-agent
  • It is automatically started at login :
    ps aux | grep [s]sh-agent
    bob      982  0.0  0.0  11088   328 ?        Ss   Oct08   0:00 /usr/bin/ssh-agent startxfce4
run the agent manually (should not be necessary) :
	eval `ssh-agent`

kill the agent :
	ssh-agent -k

env | grep SSH

check the agent is running :
	/!\ not '$SSH_AGENT_SOCK' as said in the doc

list private keys currently accessible to the agent:
	ssh-add -l



Not possible to store passwords in .ssh/config

Workaround :

install sshpass
Host Home
   User netmoon
   Port 22
   ProxyCommand sshpass -pmypass ssh netmoon@%h-raw nc localhost %p


Usage :

Generate / manage / convert / ... / SSH keys

Flags :

Flag Usage
-b numberBitsKey Specifies the number of bits in the key to create.
  • minimum : 1024 bits
  • default : 2048 (considered sufficient)
keys must be exactly 1024 bits as specified by FIPS 186-2
the -b flag determines the key length by selecting from one of three elliptic curve sizes: 256, 384 or 521 bits. All other values will fail.
Ed25519 :
keys have a fixed length and the -b flag will be ignored
-C comment Will append comment at the end of the public key ( :
ssh-rsa AAAAB3NzaCDvrOAYDtfSbKsIzDt4fOKQ1yc2EA...UadaPsKA2ofpo95n/Altv9t4mu1Ob/YHaZcdX0aCzS7WVQ== comment
ssh-ed25519 AAAACxOtLlwWTzud97ZBIpcQ3NzaC1lZDI1NTE5...IHyXmUPVMe+qyxB5II3CeXx9k comment
comment defaults to username@hostname
-f keyFile Specifies the filename of the key file.
-N newPassphrase Specifies the new passphrase
-o save private keys using the new OpenSSH format rather than the more compatible PEM format. The new format has increased resistance to brute-force password cracking but is not supported by versions of OpenSSH prior to 6.5 (check it : dpkg -l | grep openssh).
Ed25519 keys always use the new private key format.
-p Requests changing the passphrase of a private key file instead of creating a new private key. The program will prompt for the file containing the private key, for the old passphrase, and twice for the new passphrase.
-R host Removes all keys belonging to host from a known_hosts file (example)
-t type Specifies the type of key to create :
  • dsa
  • ecdsa
  • ed25519 : probably the strongest mathematically (and also the fastest), but not yet widely supported. As a bonus, it has stronger encryption (password-protection) of the private key by default than other key types.
  • rsa : the best bet if you can't use ed25519.

Example :

The default OpenSSH key encryption is worse than plaintext :

Not sure how it could actually be "worse" than plaintext, but it sounds rather poor anyway (see gory technical details below). To workaround this :
  • replace old keys with keys using the new private key format :
    ssh-keygen -t ed25519
  • update private keys to the new storage format :
    1. privateKey=~/.ssh/id_rsa; oldPrivateKey="$privateKey.old"; cp "$privateKey" "$oldPrivateKey"; ssh-keygen -p -o -f "$privateKey"
    2. make sure the key pair still works (just in case...)
    3. rm -i "$oldPrivateKey"
  • This applies only to private keys having a passphrase. Commands will work on passwordless keys but are merely useless.
  • Compatibility with servers is not a concern, because the private key never leaves your machine.
Technical details :
Make a private key without passphrase (i.e. unencrypted private key, source) :
myKey='/run/shm/myKey'; ssh-keygen -N '' -f "$myKey"; head -5 "$myKey"; rm "$myKey"*
MIIEoAIBAAKCAQEA0Fcqiy4b2hoYLdy2jxgJ4n4Ek+S5rKCwfkbMTarq5WsSJ4Rf	MII... : the base64 DER clue that an RSA key follows

The private key is an ASN.1 data structure, serialized to a byte string using DER, and then Base64-encoded.

View the ASN.1 structure (source):
myKey='/run/shm/myKey'; ssh-keygen -N '' -f "$myKey"; openssl asn1parse -in "$myKey"; rm "$myKey"*
Outputs the ASN.1 structure, made of 9 integers.
Make a private key with a passphrase (source) :
myKey='/run/shm/myKey'; ssh-keygen -N 'secret' -f "$myKey"; head -5 "$myKey"; rm "$myKey"*
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,D634E4857AF0E3B1EF0CA8518425C70E	"D634E4857AF0E3B1EF0CA8518425C70E" is the Initialization Vector (IV)

Try to view its ASN.1 structure :
myKey='/run/shm/myKey'; ssh-keygen -N 'secret' -f "$myKey"; openssl asn1parse -in "$myKey"; rm "$myKey"*
fails because not in ASN.1 format anymore : it's encrypted
Generate an encrypted key, then decode it (source):
password='123456'; myKey='/run/shm/myKey'; ssh-keygen -N "$password" -f "$myKey"; openssl rsa -text -in "$myKey" -passin "pass:$password"; head -5 "$myKey"; rm "$myKey"*
==> displays the decoded key.
Trying to do it manually :

password='123456'; myKey='/run/shm/myKey'; ssh-keygen -N "$password" -f "$myKey"; head -5 "$myKey"
	Proc-Type: 4,ENCRYPTED
	DEK-Info: AES-128-CBC,DD75D306F2CEADFCD9F5B0AA88E72368


echo -n 123456DD75D306 | md5sum

tail -n +5 "$myKey" | grep -v 'END ' | base64 -d | openssl aes-128-cbc -d -iv DD75D306F2CEADFCD9F5B0AA88E72368 -K 500d7daf9dac3883117c63b073996144 | openssl asn1parse -inform DER
	==> fails :-(((

openssl rsa -text -in "$myKey" -passin "pass:$password";
rm "$myKey"*

...doesn't work 

Long story short, the AES encryption key (used to encrypt the private key) is built with :

md5(passphrase + 8 first bytes of the IV)
  • MD5 is used in the process of encrypting the private key with the passphrase.
  • It is used only once
  • both AES and MD5 functions are cheap in terms of CPU and memory
this key is prone to brute-forcing, especially if the passphrase is short/poor quality

How to revoke a host key from SSH's ~/.ssh/known_hosts file ?

ssh-keygen -R
# Host found: line 124
/home/kevin/.ssh/known_hosts updated.
Original contents retained as /home/kevin/.ssh/known_hosts.old


ssh-copy-id is a script used to copy your SSH public key to the specified host. If necessary, it can also create the remote :

How to hide the connection banner ?

Situation :

Displaying a connection banner is fine when manually opening a connection to a server, but it's getting ugly / awkward when :

Solution :

From light to heavy weaponry :
Use SSH client's quiet mode :
ssh -q host command
Decrease verbosity :
ssh -o LogLevel=QUIET host command
Explicitly filter the unwanted banner :
ssh -q host command 2>&1 | sed -r '/banner/d'
This banner shouldn't be there, actually :
  • When you run ssh user@computer command, command is run non-interactively (source)
  • and there's no point in printing a connection banner to non-interactive shells (source)
  • so this should NOT receive a banner ()

How to hop through a SSH host ?

Situation :

Considering : it is possible to open an SSH session from workstation directly to sshTarget. To do so :

Solution :

  1. On workstation, add to ~/.ssh/config :
    Host sshTarget
    	User stuart
    	ProxyCommand ssh %r@sshHop -W %h:%p
  2. then, from workstation : ssh sshTarget
The Host directive accepts wildcards to fit various scenarios :
Host *

Host *

Advanced SSH client configuration with ~/.ssh/config

The SSH client configuration can be found in 2 files :

Save keystrokes by associating host + key + key file :

Host serverName
	User DrEinsteinVonBrainstorm
	IdentityFile ~/.ssh/myPrivateKey

Mixing things up :

Host HAL	this can be an alias
	# Real hostname (if alias given above)
	HostName realHostname
	# Port to connect to on the remote host. Defaults to 22
	Port 443
	User DrEinsteinVonBrainstorm
	IdentityFile ~/.ssh/myPrivateKey
	# Go through hop host
	ProxyCommand ssh -e none hop exec nc -w 5 %h %p

Keep SSH sessions alive

Host * is a valid statement. (Read more : 1, 2)

If SSH ignores your ~/.ssh/config file, be more explicit with -F.


Usage :

Configuration files :
File Purpose
client configuration file
~/.ssh/known_hosts local cache of the server keys
~/.ssh/id_rsa default private RSA key

The use of authorized_keys2 for protocol 2 has been deprecated since 2001 (sources : 1, 2).

Flags :

Flag Usage
-A Enable forwarding of the authentication agent connection (example)
-b bindAddress Use bindAddress on the local machine as the source address of the connection
-f Requests ssh to go to background just before command execution. Implies -n.
-F configFile Specifies an alternative per-user configuration file.
-i privateKey identity file
-J jumpHost connect to targetHost via jumpHost :
ssh -J jumpHost targetHost
This is equivalent to using the ProxyJump client configuration directive.
-M Enable ssh Master mode allowing connection sharing (for details, see ControlMaster) (See -S)
-n Redirects stdin from /dev/null (actually, prevents reading from stdin).
-N Do Not execute a remote command, just forward ports (protocol v2 only).
-p portNumber port number on which sshd will listen (default is 22).
-q quiet mode. Causes most warning and diagnostic messages to be suppressed (may be overzealous)
-S /path/to/controlSocket Specifies the path to the control socket for connection sharing (to be used with -M).
-t Force pseudo-tty allocation. This can be used to :
  • transmit signals between the ssh process and the remote one
  • allow sudo'ed commands to prompt for a password (example)
-T Disable pseudo-tty allocation. This proves useful only in scripts.
-v verbose mode. Repeat to increase verbosity : -vvv
-X enable X11 forwarding
-Y enable trusted X11 forwarding. These are not subjected to the X11 SECURITY extension controls.

Client configuration directives (for ~/.ssh/config) :

Flag Usage
BindInterface Use the address of the specified interface on the local machine as the source address of the connection
This feature was introduced by OpenSSH 7.7. As of June 2019, it must be installed —on Debian Stretch— via the testing repository (details, sources : 1, 2).

Example :

Send command to SSH host :

  • ssh kevin@myServer 'command'
  • ssh -t haproxyServer 'sudo /usr/sbin/haproxy -c -V -f /etc/haproxy/conf.d/myConfig.cfg'
  • for server in server1 server2; do ssh "$server" 'df -h /home'; done

If the SSH session is interrupted (with CTRL-c or anything else) the remote process receives no signal and continues executing. To prevent this, you may allocate a pseudo-tty to ssh with -t. (source)

SSH : how to authentify using a public/private key pair ?

Generate keys :

On Linux :
generates a private (keyName) + public ( key pair at the OpenSSH format. If not specified, keyName defaults to ~/.ssh/id_rsa
  • the type of key to generate (DSA, ECDSA, ed25519, RSA, ...) can be specified with the -t type flag, and defaults to rsa for SSH protocol version 2. Choosing the type of encryption
  • RSA key length defaults to 2048 bits, which is now a bare minimum.
  • the minimum length of an ECDSA key is 256 bits.
  • In OpenSSH, DSA keys are limited to 1024 bits, which is far from enough today. Don't use DSA keys.
  • If you need more security than RSA-2048 offers, the way to go would be to switch to elliptic curve cryptography — not to continue using RSA. (source)
  • If supported, ECDSA should be used instead of RSA.
  • Virtual machines and machines having a short uptime have not enough entropy to benefit from cryptographically-acceptable randomness and generate strong keys.

generate a 2048-bit RSA key pair (~/.ssh/id_rsa + ~/.ssh/
ssh-keygen -t rsa
generate a 256-bit ECDSA key pair (~/.ssh/id_ecdsa + ~/.ssh/
ssh-keygen -t ecdsa -b 256
On windows, for Putty :

You'll need PuTTYgen and this procedure to generate a private (keyName.ppk) + public ( key pair at the Putty format.

Copy the public key to a SSH server (source)

If the key already has the right format, this is as fast as :

ssh-copy-id -i ~/.ssh/ stuart@sshServer

  1. Your public key is a text file such as :
    ---- BEGIN SSH2 PUBLIC KEY ----
    Comment: "this-is-my-key"
    ---- END SSH2 PUBLIC KEY ----
    It contains a header + a comment + the public key itself + a footer
  2. On the SSH server, create the file ~/.ssh/authorized_keys (read about some required chmod's)
  3. Edit it respecting the format :
    ssh-rsa <public key> <comment such as key owner id>
    This must be on a single line
  4. Configure Putty accordingly : procedure (Search "Configure PuTTY").
Installing the key on an SSH server for the impatients :

As these commands use the ~ wildcard, run them as the user owning the ~ directory.

  1. On the SSH server side : mkdir ~/.ssh; touch ~/.ssh/authorized_keys; chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys
  2. On the SSH client side : cat ~/.ssh/ | ssh kevin@myServer "cat - >> ~/.ssh/authorized_keys"
Log in using the key :
  1. ssh kevin@myServer -i ~/.ssh/id_rsa
  2. which can be aliased :
    echo "alias ssh='ssh -i ~/.ssh/id_rsa'" >> ~/.bashrc && source ~/.bashrc
  3. then :
    ssh kevin@myServer


sshd supports SSH1 and SSH2 protocols, but SSH1 is not secure, so it mustn't be used !

Existing connections can survive sshd restart (source) :

Configuration files :

File Purpose
/etc/ssh/sshd_config This is the main configuration file (Configuration directives). sshd can't start if this file is unavailable.
/etc/motd welcome message to be displayed after a successful login, either "normal" login or via ssh (or grep Banner /etc/ssh/sshd_config )
~/.ssh/authorized_keys list of public keys. This can also hold some configuration directives.
/etc/nologin If this file exists, sshd refuses to let anyone except root log in.
ACL that should be enforced by tcp-wrappers are defined here.

Server configuration directives (source) :

Directive Usage
AcceptEnv myEnvironmentVariable Declare environment variables that can be set by the SSH client (aka whitelist . A single AcceptEnv directive can declare several variables, separated by a space character.
Banner points to a banner file that can be found anywhere. The content of this banner is displayed during an ssh login, between the login name and password input.
ClientAliveCountMax n Allow at most n unanswered requests sent by ClientAliveInterval before disconnecting the client.
The ClientAliveInterval timer is reset after sending every request.
ClientAliveInterval n Start a timer for n seconds before requesting client response over the encrypted channel (this has nothing to do with KeepAlive, which is a TCP setting).
0 disables this function.
GatewayPorts Specifies whether remote hosts are allowed to connect to ports forwarded for the client (details)
PermitRootLogin value
  • yes : root can log in using ssh
  • prohibit-password or without-password : password and keyboard-interactive authentication are disabled for root
  • forced-commands-only : root login with public key authentication will be allowed, but only if the command option has been specified (which may be useful for taking remote backups even if root login is normally not allowed). All other authentication methods are disabled for root.
  • no : root is not allowed to log in
UseDNS Specifies whether sshd should look up the remote host name and check that the resolved host name for the remote IP address maps back to the very same IP address. Set this to no to disable server lookups.
X11DisplayOffset n Specifies the first display number available for sshd's X11 forwarding. This prevents sshd from interfering with real X11 servers. Defaults to 10.

Server configuration and security settings (sources : ANSSI, 1, 2)

In the configuration file :
Use SSH protocol version 2 only (version 1 has flaws allowing MitM attacks) :
Protocol 2
Deny root login :
PermitRootLogin no
Disable password authentication :
Disable port forwarding (unless this is explicitly required, by X2Go, for instance ) :
AllowTcpForwarding no
Disable X11 forwarding (which could allow : key logging, sending signals to applications, redirecting display) :
X11Forwarding no
ForwardX11Trusted no
Define ACL (or consider PAM) :
  • whitelist users : AllowUsers kevin stuart bob
  • whitelist users and the client IP address : AllowUsers admin@192.168.17/24
  • whitelist groups / IP addresses : AllowGroups developers wheel@192.168.17/24
  • blacklist users / groups / IPs : DenyUsers root alice foo
  • To explicitly blacklist all the existing local users (except kevin, stuart and bob), you can quickly generate the list with :
    cut -d: -f1 /etc/passwd | grep -Ev 'kevin|stuart|bob' | tr '\n' ' '
Check directories / key files permissions of the user side before opening a session :
StrictModes yes
Automatically logout idle users after n seconds of inactivity :
ClientAliveInterval n
ClientAliveCountMax 0
Ignore ~/.rhosts and ~/.shosts files (if they exist) :
IgnoreRhosts yes
Disable host-based authentication :
HostbasedAuthentication no
Disable empty passwords :
PermitEmptyPasswords no
Login restrictions :
MaxAuthTries 2
LoginGraceTime 30
Some extra directives for ssh_config and sshd_config :
Ciphers aes256-ctr, aes192-ctr, aes128-ctr
# Pour les versions 6.3+, OpenSSH supporte le ETM (encrypt-then-mac),
# plus sûr que les anciennes implémentations (mac-then-encrypt)

# S'il s'agit d'une ancienne version, utilisez uniquement "hmac-sha1"
MACs hmac-sha2-512, hmac-sha2-256, hmac-sha1
By default SSH listens to all available interfaces and IP address on the system. You can limit this with :
Port 443
Or :
Filter sshd port (defaults to TCP 22) (See NetFilter) :
iptables -I INPUT -s -m state --state NEW -p tcp --dport 22 -j ACCEPT
Rate-limit incoming connections :
iptables -I INPUT -p tcp --dport $sshPort -i $interface -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport $sshPort -i $interface -m state --state NEW -m recent --update --seconds 60 --hitcount 5 -j DROP
Or consider fail2ban to block brute force attacks (if you don't mind it eating your CPU...)
Use TCP Wrappers : it's a host-based networking ACL system, used to filter network access to Internet.
Edit /etc/hosts.allow to allow SSH only from and :
sshd :
Last but not least :
  • analyze logs (logwatch, logcheck)
  • keep software up-to-date (apt upgrade, ...)

About the daemon sshd itself :

Start / Stop / ...
sshd is started at boot time. You can drive it with :
systemctl start|stop|restart|status|... sshd
It forks 1 new daemon for each new connection.

Advanced Features

  • log on a remote host as a specified user : ssh bob@sshServer
  • receive display of remote application :
    1. log in remote host : ssh -X stuart@sshServer
    2. start remote application
  • execute a remote application on its own (remote) display :
    1. log in remote host : ssh kevin@sshServer
    2. mangle its DISPLAY environment variable to use the local display : export DISPLAY=:0.0
    3. then start remote application
  • copy a file ("push" mode) : scp path/to/file sshServer:/remote/destination/directory
  • copy a file ("pull" mode) : scp sshServer:/path/to/file local/destination/directory
  • copy files with scp as a specified user on the remote system : scp /path/to/file steve@sshServer:/remote/destination/directory
  • Preserve file rights and timestamp with scp : use the -p flag : scp -p ...

Extra configuration directives in ~/.ssh/authorized_keys :

accept the key only for clients coming from the 192.168.15/24 network :
from="192.168.15.*" ssh-ecdsa AAAAE2Vj...
accept the key only for clients coming from but not
from="!*, *" ssh-ecdsa AAAAE2Vj...
disable the agent forwarding for the specified key
no-agent-forwarding ssh-ecdsa AAAAE2Vj...

SSH tunneling

Tunnel types (source) :

  • local tunnels : forward a local port to a remote host
  • remote tunnels : forward a remote port to a local host. This is similar to a local tunnel, with reversed directions.

Local tunnels

SSH tunnel - local

ssh kevin@SSH -L

  • the local IP is implicit, there's not need to specify it.
  • the remote IP is relative to the SSH host (=as seen by the SSH host)

Remote tunnels

SSH tunnel - remote

ssh kevin@SSH -R SSH:s_port:REMOTE:r_port


  • Don't forget that the tunnel entry is
  • When setting tunnels, you are prompted for the password of the user stuart on the host SSH. If successful, a shell opens on SSH with stuart's credentials. As long as this shell is active, the tunnel is up. So, to close the tunnel, just "exit".
  • To make sure the tunnel is up : netstat -a | grep ssh. In netstat output, connections listed as TIME_WAIT are waiting for the protocol session end (=timeout).
  • (???) To find the tunnel PID : ps aux | grep ssh | grep (part of the REMOTE IP) (????)
  • With both tunnel types, sshd only accepts connections from the loopback address, so that only local programs can use the tunnel (better security). To workaround this, edit /etc/ssh/sshd_config and set GatewayPorts to yes. (details)