Skip to content

ssh.md

Latest commit

 

History

History
545 lines (415 loc) · 35.9 KB

ssh.md

File metadata and controls

545 lines (415 loc) · 35.9 KB
Raw

Secure Shell (SSH)

Published on: 16th October 2022

Last updated: 5th November 2025

(Back to Home)

Table of Contents

Introduction

  • Secure SHell (SSH) is a protocol used to securely connect with remote machines and establish secure tunnels to communicate with them.
  • SSH is a Layer 7 (Application Layer) protocol that operates on port no. 22 by default.
  • ssh (using applications like OpenSSH or PuTTY) is the client and sshd (OpenSSH Daemon) is the server.
  • Basic syntax: ssh <username>@<hostname_or_IP>
    • If the SSH port is not the default (22), then the -p flag can be used to specify the port number. Eg: ssh username@0.0.0.0 -p 8000
  • SSH is the secure connection and tunnel for protocols such as SFTP (Secure/SSH File Transfer Protocol) and SCP (Secure Copy).

Authentication Methods

  • Password

    SSH Client Password Connection Example

  • Public-private key pair

    SSH Client Key Connection Example

    • The server usually sends the client a challenge ciphertext encrypted under the client's public key that the client has to decrypt using their secret key, do some processing and send back to the server to authenticate itself.
    • Refer to the Sharing Keys with the Server section to figure out how the server knows the client's public key.

  • There are many more authentication methods, but the above are the most common ones.

The Need for SSH

  • Telnet, which was an older way to connect to remote machines, exchanges data in plain text, which is easier for an attacker to snoop on a compromised connection and is vulnerable to MITM (Monkey-in-the-Middle) attacks. SSH is a secure replacement for Telnet, as it encrypts the connection.
  • As mentioned before SSH has the option to use public and private key pairs to communicate. These Cryptographically-generated keys are much more secure than passwords, which can turn out to be relatively simple. (Rainbow Tables)
    • Also, once connections have been made using the public and private keys, the client automatically chooses the appropriate key for a particular connection and the user might not have to enter their password as well, which makes it convenient to connect to the machine, as they don't have to remember their password.
  • Over time SSH has been used for tunnelling, forwarding TCP ports, creating X11 connections, being used as the underlying security protocol for SFTP and SCP, etc.
  • OpenSSH features
  • A brief history of SSH and remote access (Telnet, rlogin, rsh, rcp, etc.)

SSH Commands

ssh

Connect to a remote machine and forward ports. For more on port forwarding and its types, look at the 'Tunneling and port forwarding' bullet point in the Resources section below.

Usually provided by the openssh-client package.

ssh man(ual) page

$ ssh username@hostname
$ ssh username@hostname -p 22
$ ssh username@hostname -i ~/.ssh/private_key_file
$ ssh username@hostname -o PasswordAuthentication=yes -o PubkeyAuthentication=no -o PreferredAuthentications=password

# Local port forwarding
$ ssh -L <local_ip_optional>:<local_port>:<remote_ip>:<remote_port> username@hostname

# Remote port forwarding
$ ssh -R <remote_ip_optional>:<remote_port>:<local_or_destination_ip>:<local_or_destination_port>

NOTE: When it comes to automatically trying keys on a user's system while connecting to a server, SSH only auto-tries keys of the format id_<algorithm_name> (E.g.: id_rsa, id_ed25519, etc.). It will not auto-try keys with names not in the format mentioned before. Such keys have to be manually specified to SSH using the -i flag as shown in the codeblock above.

ssh-keygen

Generate keypairs (public and private keys) and get the fingerprint of keys.

ssh-keygen man(ual) page

$ ssh-keygen

# `-b` is the bit-length of the key (This is NOT the length of the generated text output of the key.)
$ ssh-keygen -t rsa -b 4096 -C "comment/e-mail"

# `-a` is the number of rounds
$ ssh-keygen -t ed25519 -a 32

# Get public key from private key
$ ssh-keygen -y -f /path/to/private/key

# Remove a key from the known_hosts file (which stores public keys of machines previously connected to)
$ ssh-keygen -R "<hostname_or_ip>:<optional_port>" -f /path/to/known_hosts/file

# Get the fingerprint of a key
# NOTE: In a keypair, both keys (the public and private keys) will produce the same fingerprint
$ ssh-keygen -l -E sha256 -f /path/to/public/key
$ ssh-keygen -l -E sha256 -f /path/to/private/key

ssh-copy-id

Appends public keys to the ~/.ssh/authorized_keys file on the server. The client can then log in without a password.

NOTE: Unless a public key was installed on the server during OS installation and that key pair is being used to add more public keys to the server using this command, a password (or some other form of authentication) will be required.

ssh-copy-id man(ual) page

# Copying all local public keys to the server
$ ssh-copy-id username@hostname

# Copying only a specific local public key to the server
$ ssh-copy-id -i /path/to/public/key.pub username@hostname

ssh-import-id

Import public keys from GitHub (gh) or Launchpad (lp) and append them to the ~/.ssh/authorized_keys file on the server. The client can then log in without a password.

Adding a new SSH key to your GitHub account

ssh-import-id man(ual) page

$ ssh-import-id gh:<github_username>
$ ssh-import-id lp:<launchpad_username>

ssh-agent

Stores SSH private keys and if keys have a passphrase, i.e., are password-protected, it remembers that and doesn't prompt the user for it after the first time.

ssh-agent man(ual) page

# To populate the SSH_AUTH_SOCK env var, so that ssh-add can communicate with ssh-agent
$ eval "$(ssh-agent)"

ssh-add

Adds, lists and removes private keys from the ssh-agent utility. Removing/deleting a key does not delete the actual key file from the system.

ssh-add man(ual) page

# To populate the SSH_AUTH_SOCK env var, so that ssh-add can communicate with ssh-agent
$ eval "$(ssh-agent)"

# List private keys managed by ssh-agent
$ ssh-add -l

# Add a private key to ssh-agent
$ ssh-add /path/to/private/key

# Delete a private key from ssh-agent
# Does not delete private key file from `~/.ssh`
$ ssh-add -d /path/to/private/key

Generating Keys

  • ssh-keygen is the utility that generates a public-private key pair.
  • Location of the generated key pair:
    • Windows: C:/Users/<username>/.ssh
    • Linux: /users/<username>/.ssh
  • key_name is the Private Key file and the key_name.pub is the Public Key file.
  • The Public Key should be put in the server in the authorized_keys file in the .ssh directory on the server.
  • On connection, if there is a publickey error, then use ssh-add <path_to_private_key> to add the keys to ssh-agent and try connecting again. (More details)
    • The path to the appropriate private key file can be linked to during connection as well, using the -i flag. Eg: ssh username@hostname.com -p 22 -i ./.ssh/private_key_file
  • Further SSH config, like the sshd_config file can be found in /etc/ssh.
  • Keys can themselves be password protected as well.

Sharing Keys with the Server

The authorized_keys file in the .ssh directory on the server usually holds all the public keys from clients that are allowed to connect with the server. How do those public keys get there, though?

There are usually two ways to do this:

  • Log in with a pre-configured username and password and manually add the public key (or copy it over directly using something like SCP).
    • After the initial use of the password to transfer the public key to the remote server/instance, the password authentication method can be turned off in the remote server/instance settings to improve security and prevent brute force attacks, but if the user loses their private key, they permanently lose access to the remote server/instance.
  • A service pre-configures it for the user and just hands them the private key to directly connect with their remote instance.
    • Eg: AWS, GENI, etc.
    • The service obviously uses the first method to configure it for the user to reduce the hassle for them and to improve the service's security as well, so that their infrastructure is not vulnerable due to the user's miscalculations (if any).
    • If these pre-configured keys are lost though, then it usually results in a permanent loss.

File and Directory Permissions

SSH requires specific file and directory owner and access permissions to be met for security purposes.

Ensure the following on the server being SSHed into:

  • Make sure PubkeyAuthentication yes is uncommented in /etc/ssh/sshd_config (not /etc/ssh/ssh_config).
    • Reload the SSH daemon post any config change
  • File/directory ownership permissions
    • Make sure the /path/to/user_name/home and the /path/to/user_name/home/.ssh directory (and all its contents) of the user being logged in as, are owned by that same user and their default group.
      • sudo chown -R user_name:group_name /path/to/user_name/home
  • File/directory access permissions
  • Make sure a firewall (like UFW on Ubuntu) is not blocking SSH

Building Blocks of SSH

SSH uses an underlying reliable connection protocol or service over which it enables secure communication and other services. The underlying connection protocol is almost always TCP, but other protocols like WebSocket can theoretically be used as well.

On top of TCP, SSH has three parts, namely the SSH Transport Layer Protocol, the SSH User Authentication Protocol and the SSH Connection Protocol

SSH Config Files

  • SSH client
    • In order of overriding behaviour from top to bottom
      • Command-line options
        • Eg: ssh -o PasswordAuthentication=yes -o PubkeyAuthentication=no -o PreferredAuthentications=password username@hostname
      • User-specific config: ~/.ssh/config
      • If present (and included above all config lines in the global config mentioned in the next point): /etc/ssh/ssh_config.d/*.conf
      • Global config: /etc/ssh/ssh_config
  • SSH server
    • The SSH server runs a daemon called sshd.
    • In order of overriding behaviour from top to bottom
      • If present (and included above all config lines in the global config mentioned in the next point): /etc/ssh/sshd_config.d/*.conf
      • Global config: /etc/ssh/sshd_config
        • Note the difference in the SSH client and server config files: ssh_config vs sshd_config (d = daemon)

NOTE: If any changes are made to any configuration files, please restart the SSH and/or SSHD service or reboot the machine.

A SSH Connection

Legend:

  • C = Client
  • S = Server
  • -> = Arrow indicating the direction of communication
  • ACK = TCP Acknowledgement flag
  • PSH = TCP Push flag

  • A TCP handshake takes place between the client and the server to establish an underlying connection.

    Wireshark: TCP Handshake

Initialization

  • C -> S: Identification String Exchange

    • Eg: SSH-2.0-OpenSSH_for_Windows_8.6 (Packet no. 24 in the image below.), SSH-2.0-PuTTY_Release_0.73, etc.
    • Contents of packet
      • The SSH Protocol version used by the client.
      • The SSH initiating software name and version used by the client.
    • The server sends a TCP ACK packet to acknowledge that. (Packet no. 25 in the image below.)


    Wireshark: SSH Client to Server Identification

  • S -> C: Identification String Message

    • Eg: SSH-2.0-OpenSSH_8.0 (Packet no. 26 in the image below.), SSH-2.0-OpenSSH_7.1p2 Debian-1, etc.
    • Contents of packet
      • The SSH Protocol version used by the server.
      • The SSH initiating software name and version used by the server.
    • The client sends a TCP ACK packet to acknowledge that. (Packet no. 27 in the image below.)


    Wireshark: SSH Server to Client Identification

Algorithm Negotiation

  • Decisions for the following types of algorithms are made in this stage:

    • Key Exchange (KEX) Algorithms
    • Server Host Key Algorithms
    • Encryption Algorithms
      • Symmetric/Secret/Private Key Cryptography
      • Eg: ChaCha20-Poly1305, AES 256 GCM, AES 128 CTR, etc.
      • Fastest for encryption, so arriving to this key through Key exchange and other processes is carried out.
    • Message Authentication Code (MAC) Algorithms
      • Proves integrity (and implicit authentication) of message.
      • Also provides protection against Replay Attacks, as the MAC will not be the same for repeated requests. (Every request will have a different sequence number.)
      • Eg: HMAC SHA2 256, UMAC 64 ETM, etc.
    • Compression Algorithms
      • Eg: Zlib, etc.

  • C -> S: Key Exchange Initialization (SSH_MSG_KEXINIT)

    • The client sends the server all the algorithms it supports. (Packet nos. 27 and 28 in the image below.)
    • The list is in order of preference. The ones at the start/top of the list are more preferred by the client than the ones below it.


    Wireshark: SSH Client to Server Key Exchange Initialization

  • S -> C: Key Exchange Initialization (SSH_MSG_KEXINIT)

    • The server sends the client all the algorithms it supports. (Packet no. 29 in the image below.)
    • The list is in order of preference. The ones at the start/top of the list are more preferred by the client than the ones below it.
    • The server acknowledges the previous C -> S SSH_MSG_KEXINIT message (ACK flag set) along with sending its own SSH_MSG_KEXINIT message in the same TCP segment (PSH push flag set). (This can be verified using the Wireshark Trace files given in the 'Resources' section below.)


    Wireshark: SSH Server to Client Key Exchange Initialization

  • If both, the client and the server, are able to find common grounds for each type of algorithm, the connection can move to the next phase or else it will fail.

Key Exchange Phase

Learn about Elliptic Curve Diffie-Hellman (ECDH).

  • C -> S: Elliptic Curve Diffie-Hellman Key Exchange Initialization (SSH_MSG_KEX_ECDH_INIT)

    • The client sends its ECDH ephemeral public key to the server. (Packet no. 30 in the image below.)
      • The first half of the Key Exchange process.
    • NOTE: This ECDH key exchange is carried out so that keys of an encryption algorithm can be generated/exchanged securely.
    • Again, this same TCP segment acknowledges the S -> C SSH_MSG_KEXINIT message and sends the SSH_MSG_KEX_ECDH_INIT data of this message along with it. (ACK flag is set along with the PSH flag that indicates data being sent.)


    Wireshark: SSH Client to Server ECDH Key Exchange Initialization

  • S -> C: Elliptic Curve Diffie-Hellman Key Exchange Reply (SSH_MSG_KEX_ECDH_REPLY)

    • The server sends multiple things to the client. (Packet no. 31 in the image below.)

      • The server's Host (Digital Signature/Certificate) public key

        • The authenticity of this public key has to be verified either manually (the fingerprint that SSH asks the user to verify when connecting to a host for the first time) or through some other certificates.
          • This fingerprinting process to authenticate a server is a big security problem, as most users don't actually check the fingerprints that the SSH client prompts the user with, which opens up the MITM (Monkey-in-the-Middle) Attack vector. Having automated checks like maybe DNSSEC DNS queries to verify the public key belonging to the server or Digital Certificates to verify the public key belonging to the server would be nice to have.

      • The server's ECDH ephemeral public key

        • The second half of the Key Exchange process.

      • A signature of a key exchange hash (The server signs it with its private host key)

        • The public key that the server gave, if successfully verified, is used to verify this signature with the hash that the client also generates.

        Explanation:

        On server:
    -   Compute `server_hash`
        -   Computation includes the final computed ECDH shared secret.
    -   `sign(server_hash, server_private_key)` = `signature`
    -   Send (signature, server_public_key, ecdh_ephemeral_key) to client

On client:
    -   `verify(server_public_key)` = `valid` or `invalid`
        -   If successfully verified, then the `server_public_key` actually belongs to the server that it initially claimed to be.
    -   Compute `client_hash`
        -   Computation includes the final computed ECDH shared secret.
    -   `verify(client_hash, signature, server_public_key)` = `valid` or `invalid`
        -   If successfully verified, then the `ecdh_ephemeral_key` can be trusted to be coming from the server and it can also be confirmed that both, the client and the server, have the same ECDH shared secret.

    • This step also proves to the client that the server is actually who it claims to be and there isn't a MITM (Monkey-in-the-Middle) Attack going on. Thus, the server authenticates itself to the client.

    • It is very interesting to explore the generation of the hash and the consequence of the Diffie Hellman shared key.

    • As before, this TCP segment has both ACK and PSH flags set. This is a common pattern across the entire communication, where there is an ACK for every packet from C -> S and S -> C, and instead of sending it separately, it is often combined with another data containing packet (which will have the PSH push flag set).


    Wireshark: SSH Client to Server ECDH Key Exchange Initialization

End of Key Exchange

  • The end of the key exchange phase is signalled by SSH_MSG_NEWKEYS from S -> C and C -> S.

  • It also indicates that all communication after this will be encrypted.

  • S -> C (Packet no. 31 in the image below.)

    Wireshark: SSH Client to Server ECDH Key Exchange Initialization

  • C -> S (Packet no. 32 in the image below.)

    Wireshark: SSH Client to Server ECDH Key Exchange Initialization

Subsequent Encrypted Communication

  • After this, all the communication between the client and server are encrypted. (Thus data payload is not visible in the encrypted packets that Wireshark captures.)

  • The server sends its fingerprint for the user to verify its identity.

    • The SSH client will ask the user to manually verify the identity of the server by presenting the fingerprint of one of the server's public key located at /etc/ssh/*.pub.

      • These are generally not manually-generated, but are initially set up by SSH itself.

    • If the user approves of the fingerprint, then SSH will store the public key(s) of the server in the ~/.ssh/known_hosts file.

    • The advantage of storing the server's public key in the file is that the next time the user connects to the server, the SSH client will not ask the user to approve the server's fingerprint, but will automatically verify it with the server's public key stored in the known_hosts file.

    • Fingerprinting demo: The server presents its fingerprint to the client.

      • On the server

        • The server's public key(s) for identifying itself can be found at /etc/ssh

          $ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:EPvxjioRUt/Non2BGcrwIgN5i19uvpuOQ7WbpU8HzPU root@Vostro-3525 (ED25519)

        • The output of the above command is what is presented to the client. This is the fingerprint of the public key.

      • On the client

        • When connecting to the server for the first time, the above fingerprint is what is presented to the client.
        • If the client wants to check the fingerprint, they can do so in a few ways (non-exhaustive list):
          • Have physical access to the server they want to connect to via SSH and run the above fingerprinting command and manually compare the fingerprints.
          • If the client does not have physical access to the server, some out-of-band system that is trusted by the user (e.g. the automation software that installed the OS on the serber) should provide the fingerprint to the client after recording it by running the above ssh-keygen command.

  • Now that the user knows that they are talking to the correct server, the SSH client authenticates itself to the server here. (SSH_MSG_USERAUTH_REQUEST)

    • The client authenticates itself during this encrypted communication phase rather than before so that it is able to securely transmit passwords and other sensitive data to the server securely.
    • The client has different authentication methods that it can choose from.

  • What can be interesting about packets of information that are encrypted? Well, SSH does something clever here and a similar behaviour is observed in Telnet as well.

  • After connection with the remote instance, one's intuition expects that SSH sends the user-typed command to the remote instance after the Enter key is hit, but SSH actually sends a request to the remote instance every time the user hits a key on their keyboard. The remote instance acknowledges the receipt of the keystroke, checks what happens when that keystroke is executed on the terminal and sends back the action to the client. The client then displays whatever the remote instance responds with, on the user-facing terminal.

    • Why send every keystroke to the remote instance? This might not make sense for commands such as ls (because they can be communicated to the remote instance after the Enter key is hit), but imagine using something like the Vim editor which requires the use of the Esc key to do certain actions. One does not hit the Enter key after the Esc key as that might mean something else to the program, so communicating every keystroke and getting back the reaction of that keystroke is necessary.
    • Article: SSH uses four TCP segments for each character you type

  • In the image below, a connection with csa3.bu.edu is established, the ls command is run on the remote instance and then the exit command is run on the remote instance to gracefully end the connection.

    SSH Client Password Connection Example

  • The above exchange was analysed using Wireshark.

    • The 2nd Wireshark Trace file as linked in the 'Resources' section below.
    • Connection to the remote instance in csa3.bu.edu has already been established.


    Wireshark: SSH Client-Server 'ls' command request-response

  • Breaking down the exchange for the ls command. (As shown in the two images above.)

    • On hitting the l key on the client, the client sends a request to the server. (Packet no. 60 in the image above.)
    • The server responds with a combined TCP segment containing the ACK flag for the receipt of the keystroke and the data with the reaction of the keystroke (PSH flag set) so that the client can draw it on the user-facing terminal. (Packet no. 61 in the image above.)
    • The client acknowledges that with a TCP ACK segment. (Packet no. 62 in the image above.)
    • The same goes for the s key. (Packet nos. 63 to 65 in the image above.)
    • The user then hits the Enter key after typing the ls command (Packet no. 66 in the image above.) and the server acknowledges this key press and replies with the reaction of pressing the Enter key in subsequent TCP segments (Packet nos. 67 to 70 in the image above.).
    • The client then acknowledges the receipt of the server's replies with a TCP ACK segment. (Packet no. 71 in the image above.)

Connection Termination

  • Connection either times out if left idle for too long or can be gracefully terminated using the exit command.

  • A TCP FIN segment is exchanged to terminate the connection to the remote instance. (Packet no. 253 in the image below.)

    Wireshark: Client to Server TCP Connection Termination

Resources