Infrastructure Decisions That Separate Hobbyist Agent Demos From Funded Agent Products

Isolation, state, and machine shape decide whether agents survive real users.

· 9 min read
Cover illustration for “Infrastructure Decisions That Separate Hobbyist Agent Demos From Funded Agent Products”
Agent Architecture · November 13, 2026 · 9 min read · 2,039 words

Persistent AI agent infrastructure suits work that needs to keep files, installed tools, authenticated sessions, and unfinished tasks from one interaction to the next. Stateless serverless functions suit requests that stand on their own, but cannot serve as a computer that must pick up where it stopped. The gap between an impressive agent demo and a working product is usually an infrastructure gap, defined by three choices, namely isolation, state, and machine shape.

Why agents break in production despite working perfectly in demos

A demo quietly removes the conditions that test an architecture. One developer runs one task with trusted code, no competing users, no adversarial input, and no need to resume tomorrow. The model can look ready for use in that setting. A second real user changes the test. Execution must stay separate, work must survive interruptions, and the available machine must fit more than the developer’s chosen example.

Del Rosario et al., from SAP, ACM, and Oxford, argue that complex, multi-step agents need robust, secure, predictable architectural patterns. Lost sessions and weak execution boundaries are failures below the model. The infrastructure decisions therefore start with what isolates execution, continue with what preserves state, and end with what machine shape the workload needs.

Why shared-kernel containers are the wrong isolation primitive for agents that run untrusted code

Standard Docker/runc containers separate processes through Linux namespaces and cgroups, but those processes still share the host kernel. That shared kernel is the weak boundary. A kernel vulnerability exposed through one container can put other workloads on the host at risk. An agent that generates Python, installs packages, and manipulates file descriptors exposes a much broader attack surface than a tightly constrained application.

Andronchik & Lokhmakov’s 2026 study places OCI containers at the bottom of its engine-class ordering for host attack surface and defense-in-depth stackability. The practical threat includes indirect prompt injection, where a page or document steers the agent into an attacker’s instructions. An illustrative incident class is a browsing agent encountering hidden instructions, downloading a binary, and executing it inside a container that lets it contact an attacker’s command-and-control server. That does not require a container escape.

Containers are fast, are well known, and work well for trusted internal code. Intent is not the right security test, though. The boundary must withstand what happens when an agent is hijacked or issues a destructive command.

The isolation spectrum — microVMs, user-space kernels, and when each is the right choice

User-space kernels such as gVisor intercept and re-implement system calls rather than exposing the host’s full syscall interface to sandboxed code. They keep code more separate than ordinary containers do, with less overhead than a full VM, though some less common syscalls may not work. Google built gVisor for App Engine workloads before its use in Cloud Run, Cloud Functions, and agent sandboxes. This tier suits generated code that is probably safe but cannot be fully trusted.

MicroVMs such as Firecracker and libkrun give each workload its own kernel through KVM hardware virtualization. A compromised guest kernel is therefore not the shared host kernel. This is the stronger boundary for untrusted execution. Firecracker boots in well under a second with minimal memory overhead, while snapshot restore can prepare new sandboxes in milliseconds. AWS built it for Lambda and Fargate, which also shows why execution isolation and the serverless state model are separate questions.

Isolates offer a narrower, lighter execution environment than a full operating system. Cloudflare’s Dynamic Worker Loader starts in milliseconds with a few megabytes of memory, but requires agents to write JavaScript. That constraint defines where it fits. The sandbox for each code path should match its trust level and what it can do, rather than use the heaviest sandbox for every action. GPU workloads need separate planning because upstream Firecracker has no built-in, officially supported GPU passthrough.

Diagram: The Isolation Spectrum: Matching Sandbox to Trust Level. Visualizes: Show a ranked spectrum of four execution isolation approaches from lowest to highest host protection, with concrete trade-offs for each tier.

What a code sandbox actually gives an agent, beyond security

LLM-in-Sandbox, from Cheng et al. in 2026, found that a model did better at tasks when it had a basic computer environment, without more model training. Terminal access, file management, and code execution produced gains across mathematics, physics, chemistry, biomedicine, long-context understanding, and instruction following. Improvements reached up to 15.5%, while token consumption fell by up to 8×. The gains were not confined to coding.

The environment lets a model run experiments, check the results, and fix its work before giving a tested answer. That makes the sandbox part of the agent’s capability stack as well as a security expense. Restrictions have a performance cost too. A sandbox can get in the way of the task itself if it starts too slowly, lacks tools the agent needs, or makes users share working environments. Isolation and startup design therefore belong in the same architecture discussion, because the computer must be both safe to use and ready when the agent needs it.

Diagram: Sandbox Capability: Up to 15.5% Better Performance, 8× Fewer Tokens. Visualizes: Show a before/after or dual stat callout contrasting agent performance with and without a basic computer environment (terminal, file management, code…

Why do long-running AI agents fail on stateless serverless functions?

A stateless request can finish without preserving its machine for the next request. A multi-step agent often cannot. Its next action may depend on a file it wrote, a package it installed, an authenticated session, or an unfinished form. If execution stops and that environment is lost, restarting the code does not pick up the work where it left off. The state model limits the agent’s ability to resume work.

Consider an agent that signs into Citrix and navigates three screens into a task before its machine is evicted. If the session state is not saved, the login, navigation, and unfinished form can all be lost. Replaying those steps may be costly or impractical. Traditional Kubernetes addresses replica scheduling, but has no native primitive for resuming a specific machine’s memory and disk from its paused position.

The better model is a computer that sleeps. Major cloud providers have announced agent infrastructure built around microVMs for each session, snapshots, and checkpointing, which points to a move beyond cold-started requests that run on their own. For workflows that depend on a live environment, persistent AI agent infrastructure preserves the machine, while a stateless function architecture leaves the application to reconstruct it.

Snapshot and restore as the technical primitive that makes persistence practical

Snapshot and restore makes persistence possible without reserving active compute throughout every pause. A Firecracker microVM can boot, receive its toolchain, and capture its disk and memory state. New sessions can then restore an environment that is already set up in milliseconds, skipping the full setup. Existing machines can sleep between tasks and wake in roughly a second. State survives without continuous execution.

Copy-on-write cloning extends that mechanism to many sessions launched from a common warm snapshot. Each session avoids the full boot cost. The same mechanism supports different lifetimes, including a per-user machine that resumes repeatedly and a throwaway sandbox that exists for one task.

A persistent desktop also supports human takeover, because a person can inspect the actual environment, correct the agent’s work, and return control. A stateless reconstruction cannot provide that same live handoff. When machines sleep, they use no active compute, which makes per-second active-use billing practical. Firecracker’s upstream snapshot model does not solve GPU passthrough, however, so GPU-dependent workloads need a separate plan.

How machine shape decisions determine whether eval pipelines and RL rollouts are viable

Eval and reinforcement learning workloads expose machine mismatch quickly. Multi-turn rollouts take different amounts of time because token generation and tool calls do not finish at the same time. Assigning them statically to fixed compute leaves resources idle while the slowest work completes. An always-on VM wastes time between runs, while a stateless function is a poor fit for the evolving environment within a rollout.

The problem also affects the validity of results. Container resource configuration can shift benchmark scores more than changing the models being compared can. Infrastructure noise can masquerade as model quality.

Polar, a rollout framework described on arXiv in 2026, separates runtime pre-warming, agent execution, trajectory reconstruction, and evaluation across gateway nodes. Its decoupled design improves utilization without tying the framework to one agent harness, training system, or RL algorithm. The suitable machine shape is a sandbox restored from a snapshot, used for one rollout, and deleted with its disk. Running thousands of these environments calls for burst scheduling rather than keeping resources reserved all the time, while per-second billing ties cost to the time they spend running.

Computer-use agents on Windows and Android require OS surfaces that containers cannot provide

Some production workflows need a different operating system, so a larger Linux machine will not meet their needs. Windows-native or Citrix-hosted applications include EHR systems such as Epic and Cerner, loan origination platforms, and dealership management systems. Banking and healthcare apps that run only on phones need an Android environment with touch event simulation. A Linux container cannot take the place of those surfaces.

Anthropic’s computer-use API has a reference implementation built around a Linux desktop in Docker, but the developer controls the environment where calls run. The API does not choose the production OS. Microsoft UFO² combines Windows GUI actions with Win32, WinCOM, and UIA, while UFO³ Galaxy extends orchestration to Linux and Android.

Remote desktops add another layer because the agent must operate Citrix or RDP from inside its own machine, using a remote desktop client or the required nested virtualization support. Regulated workflows also need audit trails, encrypted storage, and HIPAA-compliant logging where applicable. These requirements remain across operating systems. OpenAdapt’s support for browsers, Windows, macOS, Linux, RDP, and Citrix/VDI further illustrates why production computer use cannot be reduced to one Linux image.

Per-user agent architecture and why the unit economics must be designed in from the start

Agent loops process many more tokens than a simple chat exchange because tool calls, retries, and context reloads add work. Infrastructure costs arrive on top of that inference bill. Each user’s isolated machine, required OS, and active execution time shape the cost of serving that user. Those choices are difficult to retrofit once a product depends on shared state.

Sharing machines can seem cheaper in a demo. A runaway process, leaked credential, or corrupted environment can then cross one user’s boundary and affect other users. Dedicated environments narrow that exposure, but always-on reservations make every inactive user a continuing compute expense. Sleeping machines with per-second active-use billing address that mismatch by charging only for active work.

Metering each user matters just as much because a small share of users can account for most of the money spent on compute. Without it, the shared bill hides how much of the usage comes from those few users. Regulated deployments add preconfigured images, specific network topologies, and compliance-ready logging, but repeated sessions can amortize that setup. Isolation, persistence, OS support, and billing should be evaluated against the product’s expected per-user cost outside the cheap conditions of its demo.

The five infrastructure decisions, restated as a checklist for teams moving from demo to product

For teams choosing an architecture, the checklist starts with the isolation primitive. Does the execution boundary match the code’s trust level? Containers fit trusted internal code, user-space kernels or microVMs fit generated or user-supplied code, and isolates fit JavaScript-only ephemeral tasks. Familiarity is not a substitute for that distinction.

The second decision is the state model. Does the workflow need authenticated sessions, installed tools, or unfinished work to survive between interactions? If so, the machine must be persistent and resumable. The third decision is the boot model, with snapshot restore judged on whether it makes separate per-user or per-task environments ready fast enough for the planned experience. Even with the right state model, users still have to wait if startup is slow.

The fourth decision is the OS surface. Can the infrastructure provide the Linux, Windows, or Android environment the actual application requires? The fifth is the billing model, including per-second active use, sleeping without idle charges, and a boundary for measuring each user’s consumption. These are product requirements.

Teams serving legacy desktop workflows, isolated user environments, or large eval and RL pipelines should assess providers against those five decisions. The costly mistake is postponing them until real users expose the defaults. By then, replacing shared execution, disposable state, or the wrong OS can require rewriting the architecture that made the demo look simple.

Sources

  1. Computer Environments Elicit General Agentic Intelligence in LLMs
  2. Architecting Resilient LLM Agents: A Guide to Secure Plan-then-Execute Implementations
  3. AI Code Sandboxes: A Comparative Security Study. Part 1 of 2 -- Engine-Level Properties (Attack Surface, Leakage, Stackability, CVE History, Patch Cadence, Fuzzing)

More in Agent Architecture