The goal is to have random clients connected to random networks using
DHCP getting random addresses behind NAT connected firewalls. People
try all kinds of things to achieve this. They use DynDNS services.
They open and forward ports through their firewalls. Everyone tries
accomplishing this in different ways. This document tells how I am
doing it. Is it better? Is it better for me but not the ultimate.
Is it perfect? No. But it is quite good. I find it more robust than
even OpenVPN and depending upon what I am doing it is better than
OpenVPN. Does it replace OpenVPN? No. I also use OpenVPN in other
environments for a full service solution. How does it compare to
sshuttle? I love sshuttle but that has a different use goal in
mind.
With that introduction let's get on with the documentation. This is what I do for connectivity to remote clients. I have the client connect to my server machine. If the client is online it will connect to my server and use ssh to create a tunnel back to itself. I can then always follow through the tunnel to the remote device. I have many of these remote clients set up in commercial environments at mountain airports and other remote places.
Is this document perfect? No. Far from it. In fact I am rather embarrased by how jumbled this appears to me. I have held off posting this for some time because I wanted to make it more cohesive. In the end I have given up and am just going to let it escape as it is now. Some of you will think this is overly complicated and want to run screaming back to OpenVPN. That's fine. OpenVPN is awesome. I use it too. But I also feel very confident setting up AutoSSH on an already running remote host with no danger of locking myself out of it. With OpenVPN I need to modify routing tables and have locked myself out at times. I have never done that with AutoSSH. In fact I usually set up two connections for redundancy and rewire them one at a time when needed. If this document helps you then great! If not then you got what you paid for it. I am hoping it is useful regardless. Read through the entire document before doing any part of it. Then after you think you have a grip on how things go together then try it on a test instance. I think you will find it a reliable connection method.
AutoSSH is a program that starts ssh and monitors it, restarting it as necessary to keep it runing continuously. This ssh is then used for port forwarding creating a simple VPN service connection between two Internet connected hosts. A typical use of this ssh port forwarding VPN service is for remote access to a host on a dynamic ip address. The remote host can change addresses and will always "call back home" to create the VPN to enable you to log into it. I also use this for secure email transport from a client to a mail server. There are many possible uses. I am using this on everything from a Raspberry Pi to a large rack mounted server.
In order to make use of this for logging into dynamic clients one must have a static IP address somewhere. I have a server with a static address that I use. If you have a server somewhere, anywhere, then that is what you will use. If you don't have a static IP address anywhere then this will not be useful to you. But if you do then you can have your dynamic remote system open up a listening port on your server that you can use to connect to ssh.
I place a Banana Pi at the remote airport hangar. It has a network connection but only at a dynamic IP address behind a firewall. I want to be able to log into it from my home. Or copy files back and forth from it.
rsync -av somefile hangarpi:bin/
ssh hangarpi
It is really this simple to use. Let's talk about how to set it up. Setup is a little involved. But each step is simple and following this recipe guide should help you out. You can read the upstream AutoSSH documentation at http://www.harding.motd.ca/autossh/ for details, and there are many other fine articles, but this is my tutorial on how I do it.
Conceptually I would like to start on either the client or the server and walk through the setup there. Then move to the other and walk through the setup on the other side. That would be great! But it can't be done. We must do a little here and a little there and go back and forth between the server and the client just a little bit.
AutoSSH has two different strategies that can be used to monitor the connection. It doesn't just monitor that the ssh program is alive. Instead it monitors that the ssh connection from end to end is alive and passing data. That is what we really want and much more useful. In order to do this autossh configures its own port forward across the tunnel from the client to the server and back again. It is the full loop that is important. From system A to system B and then back again from B to A.
By default autossh will use two connections. One connection from here to there. Another connection from there to here. This works great. However it means that as soon as you have two autossh systems running the second one collides using the server side port! You the reader haven't even started using it yet and already I am telling you about a problem that needs to be avoided. Sorry. But to avoid that problem I use the alternate strategy of using the echo service on the server. Using the echo service on port 7 only one connection is needed. That is the way to go! It allows using the same configuration on multiple clients.
AutoSSH will start on a local port. It will connect to the remote echo service port. The echo service port will echo back anything sent to it on the same connection. This allows autossh to monitor that the connection is alive and passing data from one end to the other end.
This requires that the server support an echo port service. The easiest way to do this is to install an inetd. I don't use the inetd for anything else these days. Therefore I like to install the BSD version of the daemon.
Some people will read this and go, "I am using xinetd now. It's all about the Red Hat xinetd, baby." For those xinetd people I say, "Knock yourself out! Different drummer, same beat." All that matters is that there is an echo service available on the server.
Install the inetd and the Debian inetd update script.
apt-get install openbsd-inetd update-inetd
On Debian use the one line command to set up the echo port. (As a
small tidbit the --comment-chars '#' option is needed to let the
command know this is a human editing the file and not a package
postinst script. It is designed for package postinst scripts to use
and there it uses a special comment.) This command sets up the
/etc/inetd.conf file for the echo service.
update-inetd --comment-chars '#' --group INTERNAL --add "echo stream tcp nowait root internal"
That command will do all of the work of editing the file and signaling the running inetd to read it after the update. Spiffy!
Install the autossh program on the client.
apt-get install autossh
UPDATE: Instead of using /etc/init.d/foo scripts and mostly due
to the incoming of systemd (me throws salt over my left shoulder) I no
longer put these files in /etc/init.d anymore. Instead I put them
into /usr/local/sbin/foo and modify the Monit setup accordingly. I
hope to get around to updating the following soon but...
Create the init.d startup files, two required, to have autossh start automatically when the system is rebooted. This is the majority of the needed tinkering to get this going.
I like to name the startup script after the connecting system. For
example on my home system joseki automatically connecting to my
server system havoc I name the script /etc/init.d/autossh_havoc to
give it the name of the remote system. This allows me to have several
connections and the file names won't collide. I will name it myserver
from here on.
Download this sample init.d file (see as text in browser) into a working directory so that you can edit it. Change all occurances of the example hostname example.com to the name of your remote host. Feel free to edit it using a text editor. I will show the command line sed command here just for ease of documentation.
mv autossh_example.com.init autossh_myserver
sed --in-place 's/example.com/myserver/g' autossh_myserver
chmod a+x autossh_myserver
Place it into your /etc/init.d directory named using the
/etc/init.d/autossh_myserver name.
mv autossh_myserver /etc/init.d/
chown root:root /etc/init.d/autossh_myserver
Download this sample default file (see as text in browser) into a working directory so that you can edit it. And the same action of changing all occurances of the example hostname example.com to the name of your remote host is needed again. This does NOT need to be executable. Don't chmod it.
mv autossh_example.com.default autossh_myserver
The default file is where all of the configuration needs to take
place. This is where you will need to edit the configuration and make
various changes. All of the capitalized AUTOSSH_* variables are
used by autossh and need to be set in the environment. I use the
lower case args variable to set this up in a way that makes editing
it easier.
args="-N"
args="$args -oBatchmode=yes"
args="$args -oUsePrivilegedPort=no"
args="$args -oExitOnForwardFailure=yes"
args="$args -oServerAliveInterval=200"
args="$args -oSetupTimeOut=120"
args="$args -oUserKnownHostsFile=/dev/null"
args="$args -oCheckHostIP=no"
args="$args -oStrictHostKeyChecking=no"
args="$args -R 2201:127.0.0.1:22"
args="$args clientA@93.184.216.34" # clientA at example.com
DAEMON_ARGS="$args"
unset args
AUTOSSH_POLL=200
AUTOSSH_PORT=21021:7
AUTOSSH_GATETIME=0
AUTOSSH_LOGFILE=/var/log/autossh_example.com.log
AUTOSSH_DEBUG=yes
export AUTOSSH_POLL AUTOSSH_PORT AUTOSSH_GATETIME AUTOSSH_LOGFILE AUTOSSH_DEBUG
Change all instances of example.com to the name of your server.
s/example.com/myserver/
Change the IP address 93.184.216.34 to the IP address of your server. (The 93.184.216.34 is the IP address of example.com.)
Pick a port on your server to forward to the ssh port on the autossh client. I like to start mine at 22something so that when I grep through a listening list the port 22something ports are all grouped together. Therefore I start at port 2201 for the first dynamic client and then 2202 for the next and 2203 for the next and so forth. But the choice is arbitrary. I keep a file with a listing of used ports so that I can allocate the next one for the next one.
This is also the place to forward other ports in either direction. Can server an http web page this way. Can send mail this way. To have the dynamic client be able to tunnel mail to the server use this addition. This sets up port 2501 locally tunneled over the VPN to the remote end. More configuration of the MTA (Mail Transfer Agent) is needed to actually make it transport mail however. Ask if you need this.
args="$args -L 2501:127.0.0.1:25"
Obviously the -R and -L options configure ports on the Remote or
the Local end from the perspective of the dynamic client running the
autossh command. From that dynamic client perspective it will either
listen locally and forward it or it will listen remotely and forward
it.
Move this created file into the /etc/default/autossh_myserver
location.
chmod a+r,u+w,go-w autossh_myserver
chown root:root autossh_myserver
mv autossh_myserver /etc/default/
Create an SSH RSA key for root on the dynamic client. Press Enter
and take the defaults for all entries. Do not set a passphrase.
Root's key does not need a passphrase and will use file system
permissions to protect it. Also the dynamic client usually has no
other logins.
ssh-keygen -t rsa
Note the /root/.ssh/id_rsa.pub file. This will be used in a moment to set up the login on the server.
cat /root/.ssh/id_rsa.pub
It is a good idea to compartmentalize security into layers. Therefore I always use a unique login for every dynamic client. This way if a remote client is compromised I can disable just that one client individually without needing to reconfigure any other dynamic clients that I have connected. I have several.
As a convention I name the user after the dynamic client system name.
Conceptually it is the system that is going to log in using this
account. I will use clientA for this example. Pick the name of
your dynamic client for this user and not clientA from my example.
On the server:
adduser --disabled-password --gecos "clientA System" clientA
Set up the idrsa.pub key from the client into the authorizedkeys file for this user on the server.
cat /home/clientA/.ssh/authorized_keys
I do the copy by cut-n-paste. Be careful not to make any mistakes. End of line errors can be problematic. Edit carefully.
There are many things that might go wrong. It is time to test. Login from the client to the server. If it works then it will work for autossh. If it fails then debug why and fix it. This must work before autossh can function.
ssh -q -oStrictHostKeyChecking=no -oCheckHostIP=no -oUserKnownHostsFile=/dev/null clientA@93.184.216.34
If that works you will get a prompt on the remote machine. If it fails then remove the -q and start debugging. If it works then try it with all of the port forwarding inplace.
ssh -q -oStrictHostKeyChecking=no -oCheckHostIP=no -oUserKnownHostsFile=/dev/null -R 2201:127.0.0.1:22 clientA@93.184.216.34
If that is all working then you are almost done! Just need to set up autossh to start automatically and add a checker to ensure it stays running automatically forever.
Start the service.
service autossh_example.com start
At this point in the configuration everything is working and the tunnel is up and running. However autossh itself sometimes fails and exits under rare circumstances. Since this is my remote access to these remote systems it is useful to set up Monit to monitor the process and restart it if it fails. Therefore we need a watcher to watch the watcher. Monit is perfect for this.
This is a belt and suspenders approach if you have both. AutoSSH will restart ssh tunnels. Monit will restart AutoSSH. A rock solid combination.
apt-get install monit
Then download this monit setup file (see as text
in browser) and download this autossh monitor
file and copy them to the monit config
/etc/monit/conf.d/ directory.
check process autossh_example.com with pidfile /var/run/autossh_example.com.pid
start program = "/usr/sbin/service autossh_example.com start"
stop program = "/usr/sbin/service autossh_example.com stop"
The autossh script needs to be edited for the same example.com change as all of the other examples.
sed --in-place 's/example.com/myserver/g' monitconf.autossh
mv monitconf.Monit autossh /etc/monit/conf.d/Monit
mv monitconf.autossh /etc/monit/conf.d/autossh
chown root:root /etc/monit/conf.d/Monit
chown root:root /etc/monit/conf.d/autossh
Then restart monit.
service monit restart
Check that monit knows about the services.
monit status
Monit is pretty nice. Monit will send email when it detects a problem.
This following will check that autossh is running and restart it if for whatever reason it is not running.
If you wish to temporarily suspect monit from this action for manual maintenance then tell monit to stop monitoring. You can check the status. You can start monitoring back up again. You can unmonitor only a specific service.
monit status
monit unmonitor all
monit status
monit monitor all
monit unmonitor autossh_havoc
And of course you can always stop monit entirely.
service monit stop
Verify that everything is working. Things are tricky enough that frankly the odds are good that there will be a simple mistake somewhere and it won't be working. Therefore take it one step at a time and see if it is working. If not go back and verify each step to make sure that it is correct and work forward again.
The easiest way to verify that ssh is tunneled on the new port is to
connect to it. The typical tool is netcat. Feel free. It has some
problems. A better tool is socat.
apt-get install socat
Verify that you can connect to the port using socat. If you can see the ssh banner then you know the connection is operating. Use Control-C to break out of the connection.
socat - TCP4:localhost:2201
SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3
Control-C
With that working verify that you can log into your dynamic client from the server using the VPN.
Set up your ~/.ssh/config file for this system. This can all be done in one long command line command.
ssh -oProxyCommand='ssh -W 127.0.0.1:2201 myserver.example.com' root@myclient
That can be done on the command line but it is much more convenient to put it in the ssh config file. Here is an example from my config for logging into the remote system through myserver.
Host myclient
ProxyCommand ssh -W 127.0.0.1:2201 myserver
HostKeyAlias myclient
This avoids the need to open any ports in the firewall. Simply hop through the server to get to the dynamic client. These examples are now working trivially.
rsync -av somefile myclient:bin/
rsync -av myclient:bin/somefile .
ssh myclient
This works best when using an ssh-agent. Otherwise you will be prompted for your ssh rsa key passphrase twice. (You do have a passphrase on your ssh rsa key, right?) Once for the server and once for the client. Using ssh-agent allows convenient secure access.
Not covered in this guide is how to set up the Postfix MTA to email the mail sent by monit through to your server. Since monit sends email whenever anything happens this is pretty much a necessity or email will build up on the dynamic client.
Later!
Copyright (C) 2015, 2016, 2017, 2021 Bob Proulx
Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.