The keys on a jump host outlast the people
Most reverse-SSH setups start the same way. Someone generates one key pair, puts the public half on the jump host, copies the private half onto every device so that each can dial in, and writes down who knows the passphrase. It works on the day.
A year later the list of people has changed and the list of keys has not. This post is about that gap: what it looks like on a fleet, and what we built so that fixing it is a small, signed change on the device and not an edit of a shared file. For the separate argument about what a compromised jump host can do, see our earlier post. Here we stay with people and keys.
How the pile grows
Nobody decides to build a mess. It accumulates one reasonable step at a time.
- A new engineer joins. The quickest way to give them access is to add their key to the jump host's
authorized_keys, or to hand them the shared key. The shared key is faster, so that is what happens. - Someone leaves. Their key is in the file, and possibly in a few backups of it. Removing it means knowing which line is theirs. Comments in that file are written by whoever added the key, on a good day.
- A contractor works from a personal laptop. The key lives on a machine you do not manage, and the engagement ends without anyone closing the loop.
- A laptop is lost. You know a key is exposed. You do not know which devices it opens, because the answer is "all of them", and the shared key is also what every device uses to dial in.
- Nobody rotates. Rotating the shared key means touching every device, in the field, some of them offline. So the rotation sits in a ticket, and the key gets older.
None of this is a failure of any one person. A shared secret on a central host turns every joiner, leaver and lost laptop into a fleet-wide change, and fleet-wide changes get postponed.
What we changed
The first move is to stop having a shared key. In RescueShell each provisioning credential belongs to one team, and the security page says so plainly ("No shared keys"). The second move is to put the list of who may open a shell on the device itself, not on a host in the middle.
Each device keeps its own list of trusted public keys and the accounts those keys may open, under its state directory (trust/keys and trust/users, at most 20 of each). A new install has to be given that list and refuses to run without it. The installer prints each fingerprint and account and asks you to confirm. Our servers cannot add a key or an account to a device (see security).
When someone joins, you add their public key to the devices that should trust it. You can seed a device from an existing account's authorized_keys at install time, which is useful when you are migrating from the old setup and not starting clean.
A change is signed, and it is the device that checks
After install, the list changes in one way: a trust change signed by a key the device already trusts. The payload names the device, the next version number and the challenge that device is holding for its current connection. The signature uses the namespace rescueshell-trust-v1.
The device checks all of it. The signer must be on its current list. The version must be exactly its stored version plus one, so an old change cannot be replayed and a skipped one is refused. The challenge changes on each connection, so a signed change cannot be saved up and applied later. The result has to leave at least one key and one account. The files are written so that a crash leaves each one old or new, never half written.
Our server stores the change and relays it. It never signs. The worst it can do is drop, delay or replay one, and a replay is refused as stale.
For a lost laptop, the runbook example is one command from a key that is still trusted:
rescueshell trust <device> --sign-with ~/.ssh/other_key --add-key new.pub --remove-key lost.pub
For a fleet, the same change can be signed once for a whole group of online devices, from the CLI or from the browser, with one passkey tap or one passphrase. The CLI shows each key fingerprint, each account and the signer before it asks you to confirm, and the browser shows the same review before any unlock. One batch covers at most 50 devices, and the relay counts it as one request. It never signs a change that would remove the key doing the signing, or leave a device with no key or no login account.
That turns rotation from a maintenance window into a command. You add the new key, check that it works, and remove the old one. Nothing has to be copied around, and no file on a shared host is edited.
When a person leaves
Two different things have to happen, and we keep them separate on purpose.
First, remove them from your team. That takes effect at once: they can no longer open a shell through RescueShell on any device, online or not, and their open shells end right away (see the security page, "Your team's data").
Second, take their key off the devices. The runbook calls this hygiene, and we agree with that word. It matters because a key that is still on a device's list is still trusted by that device, whatever happened to the account that used it. Removing it is a signed change, as above.
The audit side is modest. The server records key changes, host key accepts and refused sessions as fingerprints, never key bodies, and each agent keeps its own chained log of the shells it granted or refused (see security). It is a record to read, not an export and not a recording.
Where this does not fit
We would rather list the limits than leave you to find them.
- We cannot know that someone has left a job. Removing a person from the team and taking their key off the devices are things your side has to do. There is no feed from your HR system.
- A key still on the device still works until you remove it. Team removal stops access through us. It does not edit the device's list.
- Offline devices keep their old list. A change names a challenge that exists only while the device is connected, so nothing is queued. You run the change again when the device is back, and the tool shows which devices are still on the old list from what they last reported, a hint and not proof.
- Signing needs a trusted key. If you lose every key a device trusts, no one can sign for it, and the fix is to run the installer again on the device as root.
- A hostile browser frame could sign a bad change with a key you unlock in it. The threat model lists this as a residual, with the protections we rely on.
- Metadata stays visible, and availability is ours to lose. We can see who connected where and when. If our servers are down you cannot get a shell through them. We offer no uptime SLA and no high availability.
- It is more steps than a shared key. For a handful of machines on a bench, one key on a jump host is a reasonable choice, and it is less work. The cost of per-device lists is paid every time you onboard someone, and it pays back when the fleet and the turnover grow.
The goal in our threat model is that full control of our backend and relay opens zero shells. We have not run the red-team round that would test it end to end, so it is a goal and not a result.
RescueShell: €0.10 a device a month, billed yearly, at least €6 a year, plus VAT. Thirty-day trial, no card for your first device, and the clock starts when it connects. Until a card is added, shell traffic is capped at 50 MB in total and a second device needs a card. Hosted in the EU, exceptions listed. Try it