Using Reverse SSH tunnels to bypass Stupid ISP restrictions

(skip ahead to the next section header if you just want to see how I did it)

This is yet another post related to our internet connection through 3 (Hi3G). I’ve finally gotten around mounting our new 4G LTE antenna on the side of our house, and I’ve decided to get a second one also, but for now this one will do. The antenna down under it, is a standard DVB-T2 antenna for Danish Terrestrial TV, with an 4G LTE filter of course.

4G LTe Antenna

We are getting 26/15 speeds and 10 ms ping, with this antenna alone, and I expect a 1,5x increase or so, when we get the second one up.

Onto the problem itself: I got a Raspberry Pi running here, and I expect to setup my other ones also for various tasks. For now, it’s running as file storage and small jobs using cron. But what when I’m gone? Usually, you would just portforward the relevant ports and use DynDNS or a static IP address, but things aren’t so simple with 3.

I’ve actually researched the topic before and the things I found suggested you could get a dynamic public routable IP for free which would be fine. They’ve however changed their policies and don’t have any public routable IPs anymore unless you pay 50DKK extra pr. month for a static one. Let me just recap that we are paying 300 DKK pr. month for our entire internet, there is no way I’m going to spend 50 DKK extra just for this as a DigitalOcean VPS is costing me 34 DKK pr. month.

So what’s the solution then?

What are reverse SSH tunnels?

Most of us have probably used a VPN before or at least heard of it (see my tutorial on setting up your own VPN here). You basically connect to some server and send all your traffic through it so you appear to originate elsewhere in the world.

You can actually do the same thing with SSH and make a tunnel. I do this all the time on my various servers to access locally installed MySQL instances. I will forward my own port, typically localhost:8989  or such to the remote servers localhost:3306  over SSH so that I can connect Sequel Pro to localhost on my Macbook and work with the remote server, in a secure and encrypted fashion that doesn’t require any other ports to be open.

Just tell me what a freaking reverse tunnel is!? Ok, a reverse tunnel is just like an SSH tunnel, except you “steal” the port on the remote server, and let it forward all that traffic back to your own machine’s local port. You can also forward the remote port through your own machine to another machine on your LAN, but I only need the simple version (even through they are basically the same in difficulty).

I let my Raspberry Pi SSH into my DigitalOcean server, and have it forward  port 4990 from the DO server’s public IP, back over SSH, to my Raspberry’s local port 80. So then you type in example.com:4990  you will actually be getting the local website on my Raspberry, even though it’s behind a NAT router with no public IP. Why does this work? Because the Raspberry is the one initiating the connection to the outside world, so it doesn’t need any public routable IP to be contacted as it “borrows” the one of the DigitalOcean server.

When idle, the connection is just kept open until a new client connects to the remote server and traffic is sent back to the Pi, but because no traffic is actually sent over it when idle, there is practically no overhead on internet usage or CPU/RAM usage by having it running on either end. I think SSH sends the occasional pings, and if it loses connection for one reason or another, I’ve set it up to reconnect, so this of course uses a couple of KB pr. day but it’s really nothing to worry about.

Sounds cool – How do I setup my own reverse tunnel?

First you will need to allow reverse SSH tunnels. On Debian (Ubuntu and such), you can ssh into your remote server and edit /etc/ssh/sshd_config.

The actual forwarding is really simple and can be done in a single command.

Run this command in your console and login to your server if prompted. Leave it running. You can now access remotehost.com:4990 and actually get back whatever is running on port 80, on your local machine. Please note that if you are using any kind of firewall on your remote machine (which you should), you will need to open up the remote port first, otherwise you will not be allowed access.

But you would be a fool to think that I would stop here. What if the connection hangs? What if somebody hacks my Pi and steals my login credentials for the “real” server?

Securing the tunnel and making it more persistent

Let’s fix the most gaping security hole first. You do not want to use a regular linux user for SSH tunneling. Why? Because you don’t want your private key laying around elsewhere. What if your local machine is hacked? Do you really want the hacker to have a real user on another system of yours?

Add a new user with no shell on the remote machine.

You are now free to setup a new SSH key for this user however you like. I recommend you run the following on your local machine (that needs forwarding)

And then copy the public key over to the remote machine however you like, by editing authorized_keys for tunnel-user og giving him a temporary password to use copy-id or whatever. I personally edited his ~/.ssh/authorized_keys on the remote machine while impersonating him using sudo.

You can now try out the previous reverse tunnel example using a specified private key on the machine that needs forwarding. SSH into it and run the following:

If you can access the local service by visiting remotehost.com on port 9000 you have a success.

What about making the tunnel persistent?

We will use the package “autossh” for this.

You can now run the following:

Try out the connection again and see if it works. Please note how the parameters are basically the same, EXCEPT I added the “-N” argument as this really frustrated me earlier. Why do you need the -N argument? That’s because we specified our user with no login shell earlier, and without that argument, autossh will attempt to actually login, and it will keep exiting as soon as it connects, as we have no shell to login to.

You could run this in the background and add it to rc.local as everyone else apparently does, but I prefer to run all my “background jobs” with centralized control. I absolutely love supervisor for this job! Go ahead and install it.

Supervisor works by having config files specify which programs to run. Without going into too much detail, they have great guides on their website, you can add a new file into /etc/supervisor/conf.d/sshtunnel.conf and it will automatically be loaded. Here is my config file.

I’ve attempted to have it as tolerant to internet connection drops as possible, and it’s also setup to start when my machine starts. All in all, this should be the most robust way to say “keep this running forever goddamnit!”.

Just restart supervisor (you can reload the config also, but I find this easier to remember).

Run this to check the status:

And of course try connecting to your remote server and see if it works.

Thank you for reading my blog and following along. I hope it worked for you, otherwise feel free to message me through hsp.dk.