by Dennis G. Allard<allard@oceanpark.com>,
- revised -

This page collects into one place the essential steps needed to
generate a private/public key pair and use ssh to connect to remote
hosts without having to enter your password or passphrase more than
once per boot of your trusted workstation.

The idea is that you can engage in a "multi-machine ssh session"
in which all authentication is tunneled back to the first machine
from which you started your session.

This means you can securely ssh around the net to hosts on which you
have accounts without entering your passphrase when you connect,
yet all connections and authentication occurs securely. The article
by Daniel Robbins on Agent forwarding and keychain provides
excellent overall background material.  This page merely serves
to summarize configuration steps concisely.

The last section of this Web page includes several usage examples
and a technique for running remote scripts from a local crontab
script.

Very brief synopsis
More detailed synopsis
Example usages
Example of local crontab controlling remote scripts

WARNING: OpenSSH specifications and configuration file format and
defaults are a moving target.  I had all this working for OpenSSH since
Red Hat 7.3 using ssh and sshd version OpenSSH_3.1p1.  Currently I am
using everything as documented here on Red Hat Enterprise Linux ES
and AS.  If the instructions here do not work, you may have to
do some man page reading and guessing.  For example, authorized_keys
needs to be named authorized_keys2 for older versions of sshd.  Some
items, such as StictModes are simply not well defined by the sshd man
page, since the page states that file permissions are checked but does
not say what they are checked for!  So if you are having difficulties,
carefully read the OpenSSH manual for both ssh and sshd and be
prepared to do some staring at fine print and guessing.

I will keep this page up to date for the version of Red Hat and
OpenSSH that I currently run.



Very brief synopsis

1. Generate private/public key pair (id_rsa and id_rsa.pub).
Save private key in ~/.ssh/id_rsa on trusted hosts you will ssh from.
Append public key to ~/.ssh/authorized_keys on hosts you will ssh to.
Do not use ~/.ssh/authorized_keys2.  If you see that file, remove it
and upgrade to a recent version of SSH.  I recently (2021-02-19) saw
a case where I also had to save id_rsa.pub in ~/.ssh/id_rsa.pub.

2. Set ForwardAgent yes in ssh_config on all hosts you ssh
from.  Set PubkeyAuthentication yes in sshd_config on all
hosts you ssh to.  Maybe set StrictModes no in sshd_config.
Set AllowAgentForwarding yes in sshd_config on all hosts you ssh to.

3. Install keychain then configure by adding the following lines
to your $HOME/.bash_profile:

    /usr/local/bin/keychain $HOME/.ssh/id_rsa
    source $HOME/.keychain/${HOSTNAME}-sh

4. For crontab scripts that use ssh, add the following line in
order to avoid having to manually enter a passphrase:

    source $HOME/.keychain/${HOSTNAME}-sh

5. ssh from host to host without needing to enter passwords.  Where
you can ssh, you can also scp, sftp, etc., all securely but without
having to enter your passwords on the various hosts you use.



More detailed synopsis and example usage

1. Generating and storing private and public keys.

Generate a 1024 bit RSA private/public key pair by doing:

    ssh-keygen -t rsa

This will create a private and public key file in the current
directory.  The -f option to ssh-keygen permits you to name your keys
but if you omit the -f option, the default names are id_rsa for the
private key (aka identity file) and id_rsa.pub for the public key.

On the trusted host you will connect from via ssh, store your
private key (identity file).  Normally you will connect to <remote host>
via ssh <remote host>, using local identity file, $HOME/.ssh/id_rsa.
That means that you should save your identity file as $HOME/.ssh/id_rsa
on your trusted host, where $HOME is your home directory.  However,
you can force ssh to use a specific identity file by connecting via
ssh -i <full path to local identity file> <remote host>.

On every host you will connect to via ssh, append the public
key (id_rsa.pub) to the file ~<user>/.ssh/authorized_keys
for each user you will connect as (e.g. via ssh -l <user>).
I recently (2021-02-19) saw a case where I also had to save
id_rsa.pub to ~<user>/.ssh/id_rsa.pub.  This solved a problem
where sshing to the host generated this output:

* Warning: Cannot find separate public key for /home/allard/.ssh/id_rsa.

and I was prompted for my passphrase.  Saving ~<user>/.ssh/id_rsa.pub
solved that and I was able to passwordlessly ssh.

For some older versions of OpenSSH use authorized_keys2 instead of
authorized_keys.

I believe the commercial version of SSH provides for a client-side
file named .ssh/identification that can contain a list of identities.
That makes it unnecessary to use the -i option to ssh.  I am not sure
if any such facility exists for OpenSSH.  The man page for ssh does
not mention it.  If you know about this, please contact me so I can
update this web page.

Note, you only need to and only should install your private key on
your trusted host from which you start out on your multi-machine ssh
sessions.  Your public key is what you install on all remote hosts you
wish to connect to.  You do not need to and should not install your
private key on those remote hosts unless they are trusted and you
begin multi-machine ssh sessions from them.



2. ssh_config and sshd_config files

Arrange that the ssh config file (often /etc/ssh/ssh_config) on
every host you will connect from (both trusted and untrusted)
contains the directive ForwardAgent yes.

Make sure that the default setting PubkeyAuthentication yes is in
sshd_config (often /etc/ssh/sshd_config) on hosts you will connect to.

Make sure that the default setting AllowAgentForwarding yes is in
sshd_config (often /etc/ssh/sshd_config) on hosts you will connect to.

Possibly set StrictModes no in sshd_config as well.
I am not sure that the StictModes no is needed, but it seemed to be
needed on at least one host I was connecting to when I first got all
this to work for me.

If you do not have root privileges on the host, you will have to
convince a root user of that host to do the above settings for you. (-;


3. Install and configure keychain

To avoid having to enter your passphrase more than once in a great
while, use keychain.  Once installed, you will only have to enter your
passphrase one time per boot of your trusted host(s).

After downloading and installing keychain, all you do to configure it
is add the following lines to your $HOME/.bash_profile (I assume you
installed the keychain script into /usr/local/bin, naturally):

    # Start keychain, passing as args all private keys to be cached
    /usr/local/bin/keychain $HOME/.ssh/id_rsa #, other_key1, other_key2, ...
    # That creates a file which must be sourced to set env vars
    source $HOME/.keychain/${HOSTNAME}-sh

Notice that you can install more than one private key.  That
permits you to passwordlessly ssh -l <user> to any host that has a
matching public key installed in /home/<user>/.ssh that
matches any one of your installed private keys.


4. Running crontab scripts that require passphrases

On  a trusted host on which you installed keychain and logged
in since the host was last booted, your crontab scripts that use
ssh and would normally prompt for a passphrase will run without
prompting for a passphrase as long as you start them off with
the line:

    source $HOME/.keychain/${HOSTNAME}-sh




5. Example multi-machine ssh session

Here is the transcript of an actual ssh seesion I conducted after
configuring ssh and keychain as described above.  The example involves
three hosts:

oceanpark.com     trusted workstation where I begin my ssh session
isis.cs3-inc.com  untrusted server to which authentication is forwarded
router2           trusted server on the cs3-inc.com intranet


I will probably add to this example at some point, but for now
it contains a session where I start off at my trusted workstation,
ssh to a remote server, then ssh to a futher server on the remote
intranet.  At no time do I have to enter a password or passphrase.
Included in the example are a number of comments that explain what
is going on and a few riddles to test our understanding.


...
Last login: Fri Dec 19 00:46:54 2002 from whereever.com

KeyChain 2.0.1; http://www.gentoo.org/projects/keychain
 Copyright 2002 Gentoo Technologies, Inc.; Distributed under the GPL
 * Found existing ssh-agent at PID 1580

[allard@oceanpark allard]$ # The above output reminds me that I am on
[allard@oceanpark allard]$ # a host that is running keychain.  Since
[allard@oceanpark allard]$ # I was not prompted for a passphrase, that
[allard@oceanpark allard]$ # means this is not the first logon I've done
[allard@oceanpark allard]$ # on this machine since it rebooted.  If this
[allard@oceanpark allard]$ # had been my first logon here as allard since
[allard@oceanpark allard]$ # last reboot, I would have had to enter my
[allard@oceanpark allard]$ # passphrase, which keychain would have then
[allard@oceanpark allard]$ # used to start an ssh-agent on my behalf.
[allard@oceanpark allard]$
[allard@oceanpark allard]$ # Note setting in ssh_config...
[allard@oceanpark allard]$
[allard@oceanpark allard]$ grep ForwardAgent /etc/ssh/ssh_config
ForwardAgent yes
[allard@oceanpark allard]$
[allard@oceanpark allard]$ # Next I ssh to a remote untrusted host
[allard@oceanpark allard]$
[allard@oceanpark allard]$ ssh isis.cs3-inc.com
Last login: Thu Dec 19 02:23:25 2002 from bdsl.66.14.136.144.gte.net
[allard@isis allard]$
[allard@isis allard]$ # Note lack of password prompt here too.
[allard@isis allard]$ # This is because my ssh session forwarded
[allard@isis allard]$ # authentication from oceanpark.com to isis.
[allard@isis allard]$
[allard@isis allard]$ ll .ssh
total 16
-rw-------    1 allard   allard        610 Dec 19 01:24 authorized_keys
-rw-r--r--    1 allard   allard        463 Dec 19 01:41 known_hosts
[allard@isis allard]$
[allard@isis allard]$ # Note lack of private key (id_rsa) in .ssh
[allard@isis allard]$
[allard@isis allard]$ # Note setting in ssh_config here too...
[allard@isis allard]$ # (also note how location of ssh_config here on
[allard@isis allard]$ # isis is not quite the same as on oceanpark.com)
[allard@isis allard]$
[allard@isis allard]$ grep Forward /usr/local/etc/ssh_config
ForwardAgent yes
[allard@isis allard]$
[allard@isis allard]$ # Next I ssh to another host on the CS3 intranet...
[allard@isis allard]$ # and I do so as a different user.  As root!
[allard@isis allard]$
[allard@isis allard]$ ssh -l root router2
Last login: Thu Dec 19 02:25:39 2002 from host1
KeyChain 2.0.1; http://www.gentoo.org/projects/keychain
 Copyright 2002 Gentoo Technologies, Inc.; Distributed under the GPL
 * Found existing ssh-agent at PID 12302

[root@router2 /root]#
[root@router2 /root]# # Note continued lack of password prompt.  This
[root@router2 /root]# # host permits me to logon as root since my
[root@router2 /root]# # private key matches a public key stored in
[root@router2 /root]# # /root/.ssh/authorized_hosts and authentication
[root@router2 /root]# # was performed via authentication forwarding.
[root@router2 /root]#
[root@router2 /root]# # Note, the keychain output above means that this
[root@router2 /root]# # host has keychain installed.  Read the keychain
[root@router2 /root]# # references to understand why I prefer to only
[root@router2 /root]# # install keychain on trusted hosts.
[root@router2 /root]#
[root@router2 /root]# # Having keychain here means that root could logon
[root@router2 /root]# # here without entering a passphrase even outside
[root@router2 /root]# # the context of a forwarded authentication.  But
[root@router2 /root]# # that is a horse of a different color.  You dig?
[root@router2 /root]#
[root@router2 /root]# ll .ssh
total 28
-rw-------    1 root   root       1224 Dec 19 01:55 authorized_keys
-rw-------    1 root   root        744 Dec 14 23:31 id_rsa
-rw-r--r--    1 root   root        237 Dec 16 22:59 known_hosts2
-rw-------    1 root   root        736 Dec 16 15:29 op3-id_rsa
-rw-r--r--    1 root   root        610 May 13  2002 op3-id_rsa.pub
[root@router2 /root]#
[root@router2 /root]# # In the above list of files, id_rsa is root's
[root@router2 /root]# # private key on router2.  Not necessarily
[root@router2 /root]# # the same private key as I have back on
[root@router2 /root]# # oceanpark.com which was used, transitively,
[root@router2 /root]# # to authenticate me here.  However, my public
[root@router2 /root]# # key from oceanpark.com is (and must be) in
[root@router2 /root]# # authorized_keys, as evidenced by:
[root@router2 /root]#
[root@router2 /root]# grep -c "`cat .ssh/op3-id_rsa.pub`" .ssh/authorized_keys 
1
[root@router2 /root]# # The 1 means that one line matches the key.
[root@router2 /root]#
[root@router2 /root]# # Finally, see that that the private keys are
[root@router2 /root]# # different.  Not that they couldn't be identical.
[root@router2 /root]#
[root@router2 /root]# diff .ssh/op3-id_rsa .ssh/id_rsa
3c3
< DEK-Info: DES-EDE3-CBC,82F056EC31980160
---
> DEK-Info: DES-EDE3-CBC,9D4DC582BEFFAE44
5,14c5,14
< 00n2YuTz00w76cNF4hwsutpC4pyzDgZMVJEXWSiXxU3DTroIEZzF0KEuu1NRfhN5
< xW9D+oG20lsNGQZ85arPTe1VFyPHjdTAokMH3rZv/qSCy1BqTt6fWnj7W7DDH7MG
< /+qaw8tw6WO6l0okw7TDZlJo4OGSSk7huWkggLvCXHVAa1kNLdCoMQYxqZxoPO3Q
< ...
< /Tb7R+7Jtvvl8/fB92tkbg==
---
> rmh86isLFIX6kY4+DGyr53U4hYBOrk8NBsbwDNnqwC0yE4vTkosjDVvz6VQpnYin
> 2OGCsBxJ7i998oj8esY+zKtjd6Af4WZhD6g1w+NKiGofIJBgLG90EMiWWNAA3oV1
> 7Fuebx1lc3dTyDv/ywsS7DbXHACT6WMHCZ4uqoRQivLtv7HJPoaV1L/be+YJbvaS
> ...
> dHX+7NCEp6gP6v36Wg3K3bFw1NxClh7y
[root@router2 /root]#
[root@router2 /root]#
[root@router2 /root]# # Just for good measure, demonstrate ability to SFTP
[root@router2 /root]# # back to oceanpark.com as yet a different user!
[root@router2 /root]#
[root@router2 /root]# sftp cs3@oceanpark.com
Connecting to oceanpark.com...
sftp> ls
drwxrwx---    4 cs3      allard       4096 Dec 17 22:21 .
drwxr-xr-x   54 root     root         4096 Dec  8 23:43 ..
drwxrwxr-x    3 cs3      allard       4096 Dec  6 16:16 .kde
-rw-rwxr--    1 cs3      allard        854 Dec  6 16:16 .emacs
-rw-rwxr--    1 cs3      allard         24 Dec  6 16:16 .bash_logout
-rw-rwxr--    1 cs3      allard        191 Dec  6 16:16 .bash_profile
-rw-rwxr--    1 cs3      allard        124 Dec  6 16:16 .bashrc
-rw-rwx---    1 cs3      allard        168 Dec 19 21:27 .bash_history
drwxrwx---    2 cs3      allard       4096 Dec 18 01:24 .ssh
-rw-rwx---    1 cs3      allard         59 Dec 17 21:05 .Xauthority
sftp> bye 
[root@router2 /root]# 
[root@router2 /root]# # Note that I SFTP'd back as user cs3, not as root or
[root@router2 /root]# # allard.  Why was I not prompted for a passphrase?
[root@router2 /root]# # There are two possible explanations.  One is that
[root@router2 /root]# # the identity (private key) for root here has its
[root@router2 /root]# # public key stored in the authorized_keys of user
[root@router2 /root]# # cs3 back at oceanpark.com and keychain is running
[root@router2 /root]# # here for root.  Or, just as we earlier changed
[root@router2 /root]# # from allard to root, it could be that my original
[root@router2 /root]# # allard public key that has been being authenticated
[root@router2 /root]# # here via forwarded authentication is also in the
[root@router2 /root]# # set of authorized_keys for cs3 back home.  Hard to
[root@router2 /root]# # say!  If that seems confusing to you now, just
[root@router2 /root]# # concentrate on the fact that we have done secure
[root@router2 /root]# # FTP without entering a password or passphrase
[root@router2 /root]# # and be happy about that!
[root@router2 /root]# 
[root@router2 /root]# exit
Connection to router2 closed.
[allard@isis allard]$
[allard@isis allard]$ exit
Connection to isis.cs3-inc.com closed.
[allard@oceanpark allard]$
[allard@oceanpark allard]$ # Back where we started.  
[allard@oceanpark allard]$ # Passwordless life on the net.
[allard@oceanpark allard]$ # That's ALL Folks!




6. Running local crontab scripts to control remote scripts 

Recently, I applied these ideas to perform backups from remote untrusted
hosts behind a remote firewall that copy remote files to a local trusted host,
all under the control of a crontab script that runs on the trusted host.
Although this example illustrates a backup technique, its main purpose
is to illustrate the idea of running remote scripts under the control
of a local crontab script.

The backup activity is initiated from a root crontab script running on my
local host that looks something like this:

#!/bin/sh
source $HOME/.keychain/${HOSTNAME}-sh
ssh -l allard firewall.somewhere.com ssh -l root somehost /usr/local/bin/do-backup

The above script runs an ssh command that runs another ssh command in order to
get through the remote firewall in a secure manner to a host named somehost on
the remote intranet which then runs a script named do-backup, which looks like:

#!/bin/sh
rsync -Cavz --delete --rsh ssh /etc root@oceanpark.com:/home/backup/somehost
rsync -Cavz --delete --rsh ssh /root root@oceanpark.com:/home/backup/somehost
rsync -Cavz --delete --rsh ssh /usr/local root@oceanpark.com:/home/backup/somehost/usr

The above script recursively mirrors remote directories back to my local
trusted host beneath directory /home/backup/somehost.  I did have to prep
my local host with a mkdir /home/backup/somehost/usr.

Notice that the intial crontab intiates what ends up being a chain of four
ssh connections:

local trusted host --> firewall --> somehost --> local trusted host

The trusted host is asking the remote untrusted host to copy files back to
the trusted host. 

I run the original script as root since I know that authentication fowarding
will forward root's private key all the way full circle which will enable
the rsync invocation to create directories in the backup directory having
arbitrary uid and gid ownership.  That is necessary to be a true backup that
preserves ownerhip bits.  Of course, I could run the local crontab script as
some user other than root, just as long as /root/.ssh/authorized_keys
on the local trusted host permits that user in as root.  For that matter,
even root would not be able to ssh in as root unless root's private key is
in root's authorized_hosts file.  (This last point is a gotcha when you are
suprised that you cannot ssh in to a machine as a user that you are able
to ssh out of.)  When I have some time (never), I may improve the above
script by generalizing it and using chroot on the server to increase security.
However, I trust my local trusted host, and find that the above technique
is adequate for many purposes of doing backups.

A real backup script would also perform daily, weekly, and monthly offloads
of backed up data.  That could be done incrementally thanks to rsync, but
that is getting us a bit far afield of the present topic, so let's wrap things
up for now.

The above technique is a way to do incremental updating of backed up files at
whatever frequency I choose to run the crontab script on the local trusted host.
During invocation of the crontab script, there is no need for any login or
keychain activity on the remote hosts, thanks to ssh, keychain, authentication
forwarding, and rsync.