Field devices behind other people's firewalls
We build devices that run in other people's buildings. Each one is a Raspberry Pi running a custom Buildroot Linux image, and we have a fleet of them in the field.
They are not in a datacentre. They are not on our network. They are on the customer's router, behind a firewall we do not run, on an IP we do not control, and sometimes behind carrier-grade NAT. With CGNAT there is no inbound connection to argue about.
And every so often, one of them needs a shell.
What "needs a shell" looks like
A sensor drifts and the control logic starts fighting itself. A network change at the site breaks the link to the cloud. An update lands badly. Something in the logs would explain it in minutes.
Every alternative to a shell is bad in the same way: it costs a person's day.
Talking a customer through a serial console over the phone works as well as you would expect. Driving to site is the other option. Our own estimate for a call-out in Western Europe is €150 to €400 in labour and travel, before anyone has touched anything. That is a budgeting figure, not a quote.
Everything we tried first
Port forwarding and dynamic DNS. The classic answer, and it starts with a favour. You ask a customer's IT department, or a customer with no IT department, to open an inbound port to a device you supplied. Some will refuse. A site that changes ISP breaks the forward. And a device behind CGNAT has no port to forward at all.
A VPN. Technically fine, commercially hard. A mesh VPN joins two organisations' networks into one flat network, and the customer's IT has to approve that. Our support engineer does not need a route into their building. They need a shell on one machine.
There was a practical problem too. Our root filesystem is read-only, init is BusyBox, not systemd, and the image is built with Buildroot. A VPN client that wants a tun device, a kernel module and a writable state directory is not something you drop into that.
Remote desktop tools. Built for someone sitting in front of a screen. Our devices have no desktop to share.
What we ended up building
One outbound WebSocket connection, from the device to a relay we run, held open. Our own devices run an earlier version of this, against an older relay. What follows is the rebuilt product you can sign up for.
That is the whole trick, and it is not clever. It is the only direction that works. Outbound TLS on port 443 gets out of most networks, because it looks like the browsing traffic the network exists to carry. CGNAT does not care. The exception is a network where web traffic only leaves through a proxy: the agent connects directly and does not use one, so it will not work there.
When an engineer needs a shell, they log in to the relay. The relay checks their role, then asks the device, over the connection the device already opened, to join a session. The device's own OpenSSH server checks the key. From our CLI that is the engineer's own SSH key. In the browser shell, SSH runs in the engineer's browser with their own key, and the relay only forwards encrypted bytes. The agent refuses password sessions. The engineer gets a terminal. For the length of the shell the device opens a second outbound connection to the same relay to carry it. Nobody opened a port. Nobody joined a network.
Three things mattered more than we expected:
Nothing is installed on the customer's network. No client on their side, no route into their LAN, no route from their LAN to us. The blast radius is one shell on one machine we shipped, for one engineer the relay allowed, and a log line saying who, which device and when. That is a sentence a customer's IT department can review.
The agent is one static binary. Go, CGO_ENABLED=0, built for arm64, armv7, armv6 and amd64. No kernel module, no tun device. On a BusyBox init system the installer writes an init script, and if /etc is read-only it prints the start command for you to add to the image instead. It does need an OpenSSH server on the device. Dropbear is not supported.
The operator needs nothing installed either. The relay serves a terminal in the browser. A technician on site can get a shell on the machine in front of them without our CLI.
The reconnect schedule is a distributed systems problem
This surprised us most, and it has nothing to do with firewalls.
When the connection drops, the socket library retries on its own: first after 5 to 7.5 seconds, then doubling up to a minute, each delay randomised by up to half either way. The library clamps the result to the 5 to 60 second range, so about half of the first retries land on exactly 5 seconds. Behind that, the agent checks every sixty seconds. If the connection is still gone, it rebuilds it, up to five times. Then it stops rebuilding until thirty minutes have passed.
Those numbers look like tuning constants. They are not. They are a contract with the relay.
On a small fleet a relay restart is nothing. The cap and the random spread are what stop a relay that is still struggling from being hit by the whole fleet until it falls over again, and the clamp is the part of that spread we would fix first.
So changing any of those numbers is not tuning. It is a load test against your own relay, on every device at once, with no way to abort it. The devices you would need to reach are the ones running the change.
What it costs to run
A connected, idle device holds one WebSocket, pinged every 160 seconds. Over thirty days that comes to about 4.59 MB on the wire, TCP and TLS headers included. That is extrapolated from a short run against a local relay with a network shaper, not measured on a device or over a month.
That traffic leaves Azure from our app host. Azure lists that egress at $0.087 per GB after the first 100 GB a month, and we have no invoice figure to quote. At list price, 4.59 MB is well under a tenth of a cent per device per month. The real cost of this system is the relay process and the database behind it, and both are mostly fixed.
That is why RescueShell costs €0.10 a device a month. Each device pays for its own traffic many times over. The fixed part does not: the relay and the database cost about the same for a small fleet as for a large one, and a small team's yearly minimum does not cover its share of them. We priced for reach, not margin, and we would rather explain the arithmetic than a discount.
The floor comes from the payment fees. Stripe takes about €0.25 plus 5.7% per charge: the card, billing, and Managed Payments, which handles the tax. A small fleet paying monthly would hand more than half of every invoice to fees, so up to 50 devices we bill yearly, at least €6 a year for 5 devices.
RescueShell grew out of the relay we built for our own fleet. Our own devices still run the earlier version; what you sign up for is the rebuilt product. €0.10 a device a month, billed yearly, at least €6 a year, plus VAT. Thirty-day trial, card required, and the clock starts when your first device connects. App and relay in the EU, exceptions listed. Try it