My homelab is fully ephemeral

#homelab#bazel#HER

In my homelab, no individual server is treated as irreplaceable. Every machine across the laboratory (build nodes, developer workstations, and continuous integration runners) is fully ephemeral: it can be terminated at any moment, by choice or provider eviction, and reconstituted from scratch in minutes. The only state that persists is committed source code in git and backups distributed across three isolated storage domains. This article outlines the architecture of this environment and the operational tenets behind it.

Architectural tenets

  • Compute infrastructure is ephemeral and cost-optimized. Any virtual machine meeting workload requirements is acceptable, allowing compute to be provisioned from whichever cloud provider offers the most economical rates at any given time.
  • Builds are hermetic, ephemeral, and reproducible, orchestrated by bazel, ensuring that a freshly provisioned instance becomes fully productive with a single command. Build state requires no persistent host storage.
  • Developer workstations are ephemeral display endpoints. A lightweight ChromeOS device paired with Chrome Remote Desktop connects to a cloud-hosted development VM, allowing the entire development environment to be destroyed and reconstituted in roughly twenty minutes.
  • Development flows through a self-hosted Forgejo instance, representing the only stateful service in the laboratory.
  • Continuous integration runs directly on the ephemeral workstations, eliminating the operational overhead of a dedicated build farm.
  • Autonomous agents integrate directly into the workflow. Claude Code runs within persistent sessions on ephemeral workstations, allowing tasks to be queued remotely via web interface.
  • Tailscale unifies network topology. Every node joins an encrypted mesh network during bootstrap. Cloud VMs, local ChromeOS devices, network-attached storage, and remote FPGA hardware communicate via stable hostnames without public internet exposure.
  • Authentication delegates to federated OAuth providers. Systems rely on “Login with Google” and equivalent identity providers, eliminating local credential databases across the environment.
  • Persistent data relies on a three-way backup topology. Critical state synchronizes across Google Drive, Dropbox, and an on-premises Synology NAS, establishing three independent failure domains.

Why ephemeral infrastructure?

I adopted this model after years of maintaining traditional pet servers, and suffering the inevitable operational failures that accompany them.

A traditional development server accumulates implicit state much like a drawer accumulates untracked cables. Manual package installations, late-night configuration adjustments, and ad-hoc background daemons gradually make the machine impossible to replicate cleanly. When hardware inevitably fails, you are forced to conduct an archaeology project to discover which undocumented modifications your workflow secretly relied upon.

The alternative is to treat individual machines as entirely disposable. When an instance holds zero unique state, host failures become non-events. You no longer spend weekends nursing systems through operating system upgrades, hardening bespoke configurations, or paying premium hosting rates for high-availability guarantees. Instead, you provision the most cost-effective compute that meets your specifications, and terminate instances the moment they misbehave.

This approach extends the HER philosophy (Hermetic, Ephemeral, Reproducible builds) from individual software build directories to the entire engineering laboratory. A hermetic build treats the underlying host as an empty execution vehicle; an ephemeral lab treats the host as fully disposable. The two concepts reinforce each other: builds that encapsulate their complete environment are precisely what make disposable compute practical.

Compute procurement by spot pricing

The compute capacity in the lab consists of rented cloud virtual machines. Because no host retains persistent state, procurement decisions collapse to cost efficiency per unit of work: any provider, any geographical region, and any instance architecture that offers attractive pricing is viable. Spot and preemptible instances integrate cleanly into this model: an unexpected revocation costs nothing more than re-running a build action, and the build is inexpensive.

This model also eliminates capacity sizing anxiety. When I required substantial compute for memory-intensive FPGA place-and-route runs, I provisioned a 32-core instance, completed the synthesis pass, and returned to an 8-core machine the following morning. Compute capacity can be dynamically matched to the active task.

Builds that provision their own toolchains

A newly provisioned VM becomes productive within minutes because every project builds hermetically with bazel. The build system directly fetches pinned, checksum-verified compilers, simulators, and even Vivado toolchain archives, primarily served from my Bazel registry. The initial build on a fresh instance downloads these artifacts once; subsequent builds leverage incremental caching. Because there is no manual “development environment configuration” phase, no institutional knowledge is lost when an instance terminates.

As I have documented across discussions on build philosophy and the transition to Bazel modules, the architectural implication for lab design is clear: no build state requires persistent host storage. While local caches accelerate execution and are convenient to keep warm, they remain entirely disposable.

The workstation as an ephemeral endpoint

My physical client hardware is intentionally minimal: lightweight ChromeOS devices that maintain no local state, acting as viewports for Chrome Remote Desktop sessions connected to cloud VMs. The physical laptop serves purely as a display; the actual development workstation resides in the cloud.

The cloud VM is instantiated via an automated bootstrap recipe. Tearing down an environment and reconstituting a replacement takes approximately twenty minutes, consisting largely of automated provisioning: creating the cloud instance, executing the bootstrap script, warming Bazel caches, and reattaching the remote desktop session. A twenty-minute recovery window ensures that recreation is never avoided due to operational friction. An environment that can be reconstructed effortlessly requires no defensive maintenance.

This architecture offers significant workflow advantages. I can access my development environment from any ChromeOS client with complete session continuity. Hardware loss or failure represents a trivial hardware replacement rather than data loss. Furthermore, the compute instance can reside in whichever cloud datacenter offers the lowest rates, while an FPGA development board on my desk at home remains directly accessible over an encrypted SSH tunnel for hardware programming.

Self-hosted version control

Primary development flows through a self-hosted Forgejo instance. It manages private repositories, active development branches, and release binaries that are unsuitable for public hosting, while open-source projects mirror upstream to GitHub. Within the lab architecture, Forgejo represents the sole persistent service. Even so, it is managed with strict discipline: its state is compact, explicitly defined, and continuously backed up, ensuring the service can be reconstructed from scratch rather than repaired in place.

Co-located CI runners

The lab avoids maintaining a dedicated, always-on continuous integration cluster. Instead, CI runners execute directly on the ephemeral workstations, sharing the same underlying hardware, hermetic builds, and operational lifecycle. When a workstation instance terminates, the corresponding runner unregisters; upon reconstitution, the runner registers automatically during bootstrap. Because Bazel guarantees identical build behavior regardless of environment, running CI on a development workstation executes the exact same build graph under automated trigger conditions.

Mesh networking via Tailscale

Distributing components across cloud providers, on-premises storage, and physical hardware could introduce significant operational complexity. Tailscale unifies these distributed assets into an encrypted, flat overlay network. Joining the tailnet occurs automatically during machine bootstrap: newly provisioned VMs immediately enter the private network, while decommissioned nodes cleanly depart. Nodes discover and address each other via stable MagicDNS hostnames across heterogeneous cloud regions and local subnets, requiring no public IP addresses or firewall port forwarding. SSH access to benchtop FPGA boards, backup replication to local storage, and git synchronization to Forgejo all route securely across the private tailnet.

This network fabric is what makes cost-driven multi-cloud compute viable in practice. A virtual machine provisioned with a new provider requires no custom network engineering; it connects to the tailnet at startup and immediately participates as a full member of the lab.

Centralized identity via OAuth

Disposable nodes should maintain no local secrets, particularly user credentials. Wherever supported, services delegate authentication to external OAuth providers using Google identity. Services maintain no local password hashes, ensuring that discarded disk volumes expose no confidential credentials and allowing rebuilt services to restore with existing access controls intact. Access revocation is managed centrally at the identity provider.

Remote orchestration with autonomous agents

The most recent addition to the environment is Claude Code. It operates on the ephemeral workstations alongside standard development tools, maintaining persistent sessions that preserve project context across the workday. With remote control enabled, these sessions are accessible through a web interface, allowing tasks to be dispatched from any connected device and reviewed asynchronously.

This extends laboratory productivity beyond active keyboard sessions. Code refactoring, dependency updates, and draft revisions can be initiated remotely, executing against compute instances that require no local supervision. The agent model integrates cleanly with ephemeral infrastructure. Outputs exit the machine as version-controlled commits and pull requests, meaning workstation session state is relevant only until changes are pushed upstream. When an instance is destroyed, the agent session terminates without risking uncommitted work.

The persistent core

Eliminating ephemeral layers reveals a remarkably compact persistent state:

  • Source repositories in git, hosted on Forgejo and mirrored to GitHub. Every meaningful asset is committed; instances holding checkouts of pushed branches contain zero unique data.
  • Triplicated backup storage, distributed across Google Drive, Dropbox, and an on-premises Synology NAS. Three independent providers establish three isolated failure domains; two concurrent provider failures can occur without data loss.
  • Physical hardware assets, specifically the benchtop FPGA board and local NAS, operate on-premises. The FPGA board is connected via smart plug and programmed remotely, allowing physical hardware to be operated with the same operational detachment applied to cloud compute.

Operational trade-offs

This architecture involves deliberate engineering compromises that may not suit every environment. Remote desktop protocols introduce latency that is perceptible; while entirely acceptable for terminal sessions and code editing, it is less suitable for workflows demanding high-frame-rate graphical interactivity. A twenty-minute reconstitution window is fast, but still non-zero, and the initial download of proprietary FPGA toolchains on a clean instance requires substantial bandwidth and storage upfront. Budget compute providers occasionally experience hardware instability or transient network anomalies. Crucially, the entire model requires strict development hygiene: unpushed work on a terminated instance is irrecoverable.

For my engineering workflow, these trade-offs are unquestionably justified. I have eliminated catastrophic data loss from hardware failure, discontinued payments for idle compute resources, and established a development environment where every machine can be restored to a known-clean state with a single command.