My homelab is fully ephemeral

My homelab has no servers I care about. Every machine in it, the build servers, the workstation, even the CI, is ephemeral: it can be destroyed at any moment, by me or by the provider, and reconstituted from scratch in minutes. The only things that persist are the sources, in git, and the backups, in three places. This post describes the setup and the tenets behind it.

The tenets

Servers are ephemeral, and run by the lowest bidder. Any virtual machine that will do the job is acceptable, so the job goes to whoever rents it cheapest today.
Builds are hermetic, ephemeral, and reproducible, driven by bazel, so a fresh machine is a productive machine after a single command. Nothing about a build needs to be persisted.
The workstation is ephemeral too. A ChromeOS device plus Chrome Remote Desktop in front of a cloud VM. The whole “workstation” can be destroyed and reconstituted in about 20 minutes.
Development runs through my self-hosted Forgejo instance, the one piece of the lab that behaves like a service.
CI runs on the ephemeral workstations themselves. No dedicated build farm.
AI agents work here too. Claude Code runs on the ephemeral workstations, with persistent sessions and remote control, so I can order work from a web interface wherever I am.
Tailscale ties everything together. Every machine joins the tailnet when it is created; the cloud VMs, the ChromeOS devices, the NAS, and the FPGA host see each other by stable names, with nothing exposed to the public internet.
Authentication is OAuth wherever possible, “Login with Google” and its equivalents, so no machine or service in the lab has to keep its own password database.
What must persist is backed up three ways: Google Drive, Dropbox, and the NAS at home, tied into a 3-way synchronized backup setup.

Why?

Because I have owned pet servers before, and I have buried several.

A pet server accumulates state the way a kitchen drawer accumulates cables. Each package installed by hand, each config file tweaked at midnight, each daemon added “temporarily” makes the machine harder to replace. The day the disk dies, and that day always comes, you get to discover which of those tweaks your workflow secretly depended on. The restore then becomes an archaeology project.

The alternative is to make the machine worthless. If nothing on it is precious, then losing it does not matter. You do not harden a worthless machine, you do not nurse it through upgrades, and you do not pay a premium to keep it alive. You rent the cheapest one that works, and when it misbehaves, you throw it away and make another.

This is, of course, the same HER philosophy I apply to builds, promoted from the build directory to the whole lab. A hermetic build treats the machine as an empty vessel; an ephemeral lab treats it as a disposable one. The two reinforce each other: builds that bring their own environment with them are exactly what make disposable machines practical.

Servers, by the lowest bidder

The compute in the lab is rented virtual machines. Because nothing on them persists, the selection criterion collapses to price per unit of work: any provider, any region, any machine shape that is cheap this month is fine. Spot and preemptible instances are fine too: eviction costs nothing but a rebuild, and the rebuild is cheap.

This also removes the sizing anxiety. When I needed a bigger machine for the notoriously CPU and memory hungry place-and-route, I made one; a 32-core machine one day, back to 8 cores the next. I pick the machine shape per task and change it freely.

Builds that bring their own environment

The reason a fresh VM is instantly useful is that every project builds with bazel, hermetically. The build fetches its own compilers, its own simulators, even its own Vivado (for programmable hardware work), from pinned, checksummed sources, many of them served out of my bazel registry. The first build on a new machine downloads the world once; every build after that is incremental. There is no “setting up the dev environment” step, and so there is nothing to lose when the machine goes away.

I have written about this at length, from why I bother with bazel through the migration to bazel modules, so here I will only note the consequence that matters for the lab: no build state is worth persisting. Caches make things faster and are nice to keep warm, but they are always disposable.

The workstation is a tab

My local hardware is deliberately boring: ChromeOS devices, which are stateless by design, in front of Chrome Remote Desktop sessions to a cloud VM. The laptop is a viewport; the workstation is the VM behind it.

The VM itself is built from a scripted recipe. Destroying it and reconstituting it takes about 20 minutes, most of which is waiting: create the instance, run the bootstrap, let bazel warm its caches, reconnect the remote desktop. Twenty minutes is short enough that I do not hesitate to do it, which is the whole point. An environment you can rebuild without worry is an environment you never need to be careful with.

This has pleasant side effects. I can sit down at any ChromeOS device anywhere and be at my desk. A lost or broken laptop costs a few hours and no data. And the workstation can live in whatever region is cheapest, while the FPGA board on my desk at home is reachable over an SSH tunnel for programming, hundreds of miles away from the machine doing the synthesis.

Git at home

Development runs through my self-hosted Forgejo instance. It holds the private repositories, the work in progress, and the release artifacts that public infrastructure should not or cannot hold; the public projects mirror out to GitHub. Of everything in the lab it is the closest thing to a service, and even so it is treated with the same suspicion as the rest: its state is small, well defined, and backed up, so the instance itself can be rebuilt rather than rescued.

CI on the workstations

There is no CI farm. The CI runners ride along on the ephemeral workstations: the same machines, the same hermetic builds, the same lifecycle. When a workstation is destroyed, its runner vanishes with it; when the workstation is reconstituted, the runner registers again as part of the bootstrap. Since a bazel build behaves identically wherever it runs, CI on a workstation runs the same build with a different trigger.

Tied together with Tailscale

Everything above sounds scattered: VMs at whichever provider is cheapest this month, a NAS and an FPGA board at home, ChromeOS devices wherever I happen to be. Tailscale joins all of it into one flat private network. Joining the tailnet is part of every machine’s bootstrap, so a freshly created VM comes up already on the network, and a destroyed VM simply drops off. Machines reach each other by stable names no matter which provider or region they run in, and nothing has to be exposed to the public internet. The SSH connection to the FPGA board, the backup traffic to the NAS, the pushes to the Forgejo instance: all of it runs inside the tailnet.

This is also what makes the lowest-bidder rule workable in practice. A VM at a new provider does not need firewall rules, port forwarding, or a public address plan. It joins the tailnet during bootstrap and is immediately a full member of the lab.

Ephemeral machines should hold no secrets, and that includes passwords. Wherever a service supports it, authentication is delegated to an OAuth provider: “Login with Google” or its equivalent. The services keep no local password database, so there is nothing to steal from a stolen disk, nothing extra to back up, and a rebuilt service comes back with its users intact. Revoking access happens in one place, at the provider.

Ordering work from a web interface

The newest resident of the lab is Claude Code. It runs on the same ephemeral workstations as everything else, in persistent sessions that keep their context across the day. With remote control enabled, those sessions are reachable from a web interface: I can open a browser tab on any device, hand a session a task, and check the result later.

This changes what the lab produces when I am not at a keyboard. I can order a blog post draft, a dependency upgrade, or a round of fixes from a phone, and the work happens on a machine I do not have to think about.

The agent fits the ephemeral model without special treatment. Its work leaves the machine as commits and pull requests, so the session state on the workstation matters only until the work is pushed. When the workstation is destroyed, the sessions go with it, and nothing of value is lost.

What actually persists

Strip away everything ephemeral and what remains is small:

Sources live in git, on the Forgejo instance and mirrored to GitHub. Anything worth keeping is committed; a machine holding only checkouts of pushed branches holds nothing.
Everything else is backed up three ways: Google Drive, Dropbox, and the NAS at home. Three independent parties, three failure domains; any two can disappoint me on the same day and nothing is lost.
The hardware that is genuinely physical, the FPGA board, the NAS, sits at home. The board is on a smart plug and programmed remotely, so even the physical pet is operated like cattle.

What it costs

The setup has real downsides, which I accepted but you might not. The remote desktop adds latency, and you feel it; it is fine for editing and terminals, less fine for anything that needs fast interaction. The 20-minute reconstitution still takes 20 minutes, and the first hermetic fetch of a vendor toolchain on a new machine is measured in hours, once. The lowest bidder occasionally reminds you why they bid lowest. And the whole scheme depends on committing and pushing constantly, because anything that was not pushed is lost when the machine goes away.

For me the trade is clearly worth it. I have not lost work to a dead machine in years, I stopped paying for idle capacity, and every machine in the lab is one command away from being new again.