Managing multiple SSH environments

Daily, I end up having to SSH between VMs in the cloud, my desk workstation in the office, my home server, and things I cannot remember right now. Most of the time just doing a ssh user@<desired host>doesn’t cut it. If it did, this blog would end right here.

To give some context, I work on a few different projects. A project may have some presence in the cloud and some presence in the data centers that the company operates. Most of these servers are running in-progress development environments with new features being tested out. And in the cloud, these environments might be spread across cloud-regions for various reasons.

Apart from my desk workstation and my home server, for security reasons, the hosts I need to connect to are walled off behind a bastion host(s), with generally a different set of bastions per environment and or project. Hence to actually connect to the hosts of interest through the bastion, I need to use a feature of SSH called ProxyCommandor more recently ProxyForwardwhich allows SSH to use the bastion as a proxy host and connect to the server of interest, while looking like a regular SSH connection to the user.

Lets start with an example with the following setting:

• All the servers have FQDNs of the form <server>.<environment>.example.com
• For the dev environment in us-west-2region (an AWS region for example), we have
• bastion.us-west-2.example.com
• And a server of interest say nginx.us-west-2.example.com
• All the servers in this environment use the rsa_us-west-2private key

All this on the command line would look like

ssh -i ~/.ssh/rsa_us-west-2 \
-o ProxyCommand='ssh user@bastion.us-west-2.example.com nc %h %p'\
user@nginx.us-west-2.example.com

Now imagine you have to type that command in multiple times everyday….

Enter SSH config file

SSH has a config file to deal with this situation. You can define all of the above in the config file and just use a short hand name and let SSH apply the configs from the file and get the same result.

For those who have never seen or dealt with an SSH config file, here is a quick introduction with the basic concepts and all you need to understand this blog. This is in no way complete and I would direct you to elsewhere in the internet if you are interested in other capabilities and options available. A good place to start is man ssh_config.

The default location of your SSH config file is ~/.ssh/configand it probably won’t exist if you have never used it. You can create one by running

touch ~/.ssh/config

And open it in your favorite editor.

The config file consists of a bunch of what is called a Hostsections which apply a bunch of options to a given set of hosts.
The format is roughly

Host <host-pattern>
    Option1 value1
    Option2 Value2
    ...
    OptionN ValueN

While ~/.ssh/configis the default config file that ssh will load, you can specify an arbitrary config file using the -Foption to ssh:

ssh -F <path-to-config-file>

With the default being equivalent to

ssh -F ~/.ssh/config

And hence you can keep your config files anywhere you would like and use them using the above option and save you a bunch of time.

Here is an example of what can be in ~/.ssh/config:

Host bastion.us-west-2.example.com
    IdentityFile ~/.ssh/rsa_us-west-2

Now whenever you run ssh bastion.us-west-2.example.com, SSH will use the rsa_us-west-2key.

Within a give ssh config file, multiple host patterns might match a given host, in such a case, ssh will make a union of all options with the firstof overlapping options taking affect. For example you could have the following in your config file:

Host *
    Option1 value1
    
Host bastion.us-west-2.example.com
     Option2 value2

When you now run ssh bastion.us-west-2.example.comboth Option1and Option2will be used. We will see how to use this feature to our advantage later in this blog.

Putting it all together for my nginx example above, we can have the following in ~/.ssh/config

Host bastion.us-west-2.example.com
    User user
    IdentityFile ~/.ssh/rsa_us-west-2Host nginx.us-west-2.example.com
    User user
    IdentityFile ~/.ssh/rsa_us-west-2
    ProxyCommand ssh user@bastion.us-west-2.example.com nc %h %p

And now all I need to type on my terminal is ssh nginx.us-west-2.example.comand I will be going through the bastion host.

Multiple Environments

As I alluded to earlier, the main point of this blog is easily managing different SSH configurations per environment and/or project you are working on. The config file I described above becomes an important piece of the solution, which might seem very obvious by now: a config file per environment.

Now you could very well put all your config files named by environment and/or project in a single directory and get away with it. However this can easily be hard to manage, a problem I personally encountered.

Here’s the structure I came up with. Let’s start by looking at the ~/.sshdirectory:

~/.ssh
    └ project1
         └ us-east-1
            └ config
            └ private_key
            └ public_key
         └ us-west-2
            └ config
            └ private_key
            └ public_key
    └ project2
         └ us-west-1
            └ config
            └ private_key
            └ public_key
     ....

I just used ~/.sshas the directory of choice to use for managing the configurations since everything in there is related to SSH, however you could very well choose any other directory for this purpose.

Next we have one sub-folder per-project and a nested sub-folder per environment of that project. I have used the AWS regions us-east-1, us-west-1, and us-west-2as example environments above. Each of these folders contain a file called configthat contains all the SSH configuration needed for it, a private and public key file (this can be omitted if you have one public-private key across everywhere you need to access). The configwill be configured to use the correct private key file and will need an absolute path (you can still use ~) to it due to lack of features in the ssh config file.

All of this would work pretty nicely for anyone who don’t need any fancy stuff like ProxyCommand.

But I do, and here’s why it doesn’t just work as expected.

Let’s say the config file I had at the end of the last section is the one in ~/.ssh/project1/us-west-2/config. Here’s what it looks like:

Host bastion.us-west-2.example.com
    User user
    IdentityFile ~/.ssh/project1/us-west-2/private_keyHost nginx.us-west-2.example.com
    User user
    IdentityFile ~/.ssh/project1/us-west-2/private_key
    ProxyCommand ssh user@bastion.us-west-2.example.com nc %h %p

Now I should in theory be able to do the following:

ssh -F ~/.ssh/project1/us-west-2/config nginx.us-west-2.example.com

And since I claimed it worked when the config file was in ~/.ssh/config, this should in the most logical case work.

But it doesn’t, and here’s the reason why. If you look at the ProxyCommandline, there is another ssh command being executed. So what is actually is happening is when I run ssh -F ~/.ssh/project1/us-west-2/config nginx.us-west-2.example.comis the following:

1. SSH first opens up the config file
2. Looks for Host entries that match nginx.us-west-2.example.comand it finds one
3. It loads all the options and notices the ProxyCommandoption.
4. Due to this, it will create a child process with whatever the ProxyCommand‘s value portion is and try to relay the ssh protocol over the stdin/stdout of the child process instead of a TCP connection directly to the host in question.

All of this looks good so far, but the devil is in the details. The child process created above executes blindly the command provided, which in this case is ssh user@bastion.us-west-2.example.com nc %h %p. This looks like a regular SSH command and does not contain the -F ~/.ssh/project1/us-west-2/configoption and hence will resort to using ~/.ssh/config,if one exists, to look for options to be applied for bastion.us-west-2.example.com. If you are lucky and something does match, the connection might go through, but I found out the hard way. It turns out that the -Foption is not passed in to any sub process that are spawned off as a result of the ProxyCommand, which from the SSH developers perspective is a reasonable assumption, but doesn’t seem to be well documented, or I was just expecting too much!

So the easy fix is to update the config file to look as follows:

Host bastion.us-west-2.example.com
    User user
    IdentityFile ~/.ssh/project1/us-west-2/private_keyHost nginx.us-west-2.example.com
    User user
    IdentityFile ~/.ssh/project1/us-west-2/private_key
    ProxyCommand ssh -F ~/.ssh/project1/us-west-2/config user@bastion.us-west-2.example.com nc %h %p

Streamlining this approach

I personally feel typing ssh -F ~/.ssh/project1/us-west-2/config nginx.us-west-2.example.comis too much to type on the shell when I want to quickly get into the host, especially if there is an ongoing issue with it.

First, let’s shorten the host part. Since SSH is doing a regex match on the Hostentries with what is provided on the command line, we can do the following:

Host bastion.us-west-2.example.com
    User user
    IdentityFile ~/.ssh/project1/us-west-2/private_keyHost nginx
    HostName nginx.us-west-2.example.com
    User user
    IdentityFile ~/.ssh/project1/us-west-2/private_key
    ProxyCommand ssh -F ~/.ssh/project1/us-west-2/config user@bastion.us-west-2.example.com nc %h %p

Now the following would work just as before ssh -F ~/.ssh/project1/us-west-2/config nginx.

Now to get rid off the path to config file, I just used BASH aliases to fix it. For example, for the above you could have the following in your bash configuration.

alias sshp1uw2="ssh -F ~/.ssh/project1/us-west-2/config" # ssh (p)roject(1) (u)s-(w)est-(2)

And now you can sshp1uw2 nginxand get the same effect.

The alias idea can be applied to other SSH based commands such as scp.

Other tips and tricks

I mentioned before that SSH will match all matching Host patterns for a given host in the config file. This allows us to use wildcards and move common configurations to one section.

Host *
    IdentitiesOnly yes
    IdentityFile ~/.ssh/project1/us-west-2/private_key
    UserKnownHostsFile ~/.ssh/project1/us-west-2/known_hosts
    User user# ProxyCommand only for non-bastion hosts        
Match !host bastion
    ProxyCommand ssh -F ~/.ssh/project1/us-west-2/config bastion nc %h %p
       
Host bastion
     HostName bastion.us-west-2.example.com
   
Host nginx
     HostName nginx.us-west-2.example.comHost db
     HostName db.us-west-2.example.com

Above you can see that some of the common configuration such as IdentityFileare defined in the single section Host *which matches all hosts. We then have a Matchsection which adds options when specific conditions are met. In my example I have added a condition to add the ProxyCommandconfiguration for all non-bastion hosts. Without the Matchsection, we would encounter an infinite loop trying to ssh into the bastion because you would spawn off a new ssh process as a result of the ProxyCommandthat loads the same file and applies the ProxyCommandagain and repeats infinitely.

So now, if you have your Bash alias and the above config, the following would work

sshp1uw2 bastion
sshp1uw2 nginx
sshp1uw2 db

Notice the following two options in the catch-all host section:

• IdentitiesOnly: This tells SSH to only use the identity configured in the config file. For key-based authentication, only the private key file configured using the IdentityFilewill be used. The main advantage is that SSH will not try any other private key files that may be configured on your SSH Agent. It comes handy when your hosts have a maximum authentication tries limit set and SSH tries every other key before the correct one.
• UserKnownHostsFile: If your environments have overlapping IPs, using the single default ~/.ssh/known_hostsfile wouldn’t work, since SSH will complain about the host signatures not matching. By setting this, you ask SSH to read a different known hosts file, which in this example I have set it to be within the config folder for that project and environment.

Conclusion

Everything you just read sounds, at a high level, very similar to the virtualenvin the Python world for example. Currently, I created all the directory hierarchies and the config files within them manually. But at some point this will definitely get out of hand and having something similar to virtualenvwhere I can activate and deactivate SSH environments would be really great.

I hope the blog helps you streamline your SSH configuration files and prevents a lot of typing on the shell when you are trying to SSH into different machines.