Choosing Your Agent Execution Substrate When Persistence and Tenant Isolation Are Non-Negotiable

MicroVMs fit persistent, isolated agents only if snapshots and tenant scoping hold.

· 9 min read
Cover illustration for “Choosing Your Agent Execution Substrate When Persistence and Tenant Isolation Are Non-Negotiable”
Agent Architecture · November 17, 2026 · 9 min read · 1,966 words

Per-user persistent AI agent deployment at scale requires an execution environment that preserves each user’s full session without keeping compute assigned while idle. When strict tenant isolation is also non-negotiable, microVM-backed persistent environments fit that combination, provided the platform implements snapshot restore and tenant-scoped tools and memory. The choice turns on four properties, namely session lifetime, state fidelity, isolation boundary, and idle cost.

Why persistence and tenant isolation are architectural constraints, not feature requirements

Shared mutable state makes persistence easier, but weakens the boundary between users. Destroying an environment at session end simplifies separation, but destroys continuity. Neither shortcut satisfies both constraints. A session that spans hours or days and accumulates tool state does not fit infrastructure designed only for independent, short-lived invocations.

Persistence here means more than saving a conversation. The process state, filesystem, and state associated with in-flight tool calls must survive idle periods and return to a running condition without the agent knowing it was suspended. Tenant isolation must hold through that same cycle, preventing one user’s execution from contaminating another’s tools, memory retrieval, or reasoning chain. Together, these requirements rule out substrates that can preserve only outputs or separate tenants only by convention.

The four substrate properties that determine fit are session lifetime, state fidelity, isolation boundary, and idle cost

Session lifetime is the interval between provisioning an environment and releasing it. That interval need not equal active execution time. An agent may run briefly, wait for inference or an external tool, and then need its full environment back. A sandbox for independent executions and an environment for an evolving session therefore solve different problems, even when both can execute the same code.

State fidelity measures how much survives suspension. An ephemeral container preserves nothing unless state is stored elsewhere, while a filesystem snapshot preserves files but not the running process. Full memory-and-CPU snapshots preserve the execution point itself. That distinction governs what “resume” actually means. Reopening saved files is not equivalent to continuing from the instruction where execution stopped.

Isolation boundary and idle cost complete the framework. Logical tags on shared storage offer a different boundary from a separate guest kernel per session, and are insufficient for untrusted workloads. Idle cost is the resources an agent uses or the charges it incurs while suspended. For intermittently active agents, that can dominate execution cost. Familiar tooling and quick startup are useful only after these four properties satisfy the workload’s constraints.

How hardened containers, gVisor, and microVMs map onto those four properties

In hardened containers, the environment usually lasts only as long as the process. Volumes can preserve filesystem state, but preserving in-process state across restarts requires explicit serialization or application-level checkpointing and orchestration. Keeping the container resident avoids that restart problem at the cost of ongoing resource use. The host kernel remains shared with other containers. A successful escape can expose the host node and potentially adjacent workloads, making this a poor boundary for untrusted code executed for external tenants.

For bounded internal tasks, that tradeoff can be acceptable. Containers offer a viable fit when external state storage is enough and a shared kernel meets the isolation requirement. They do not, by themselves, provide transparent suspension with full state restoration. Stopping a container frees its resident resources but means it needs re-initialization later. Keeping it running preserves continuity by retaining the compute assignment, which is precisely the expense a large, mostly idle fleet needs to avoid.

gVisor intercepts syscalls before they reach the host kernel, reducing the host attack surface. That offers a middle ground between hardened containers and microVMs. The interception adds latency, kernel feature compatibility gaps, and an attack surface of its own. Snapshot support exists in some configurations, but does not provide the same completeness as full memory-and-CPU microVM snapshots. Its isolation model alone also does not establish session lifetime or suspended-session cost, so those properties depend on the surrounding implementation.

With microVMs, each session has its own guest kernel. So a guest kernel compromise does not directly expose the host kernel. With a working snapshot path, memory and CPU state can be restored while idle sessions release their compute assignment. An issue filed August 28, 2026 reports that long-running jobs lose progress during node drains or spot reclaims because Kata Containers does not yet make Firecracker’s snapshot-and-restore capability available. Choosing the underlying model is not enough if the integration omits the feature the workload needs.

How do you give thousands of users persistent agents without keeping every session running?

Production architectures separate session management from compute capacity. The control plane keeps track of session state and assignments, while the execution plane provides workers. A session can then be suspended, persisted, and restored onto an available worker, releasing running compute between turns. The user keeps a distinct environment across turns.

Firecracker’s snapshot-restore path makes this separation practical. Restore memory-maps the snapshot file, loads CPU state, and resumes execution without booting a kernel or running init. For new environments, pre-warmed snapshot pools reduce activation time compared with a full boot. These mechanisms serve different points in the lifecycle. One restores an existing session’s execution state, while the other avoids building a fresh environment from scratch on demand.

The control plane must keep track of session-to-Worker assignments apart from Kubernetes Pod lifecycles. Pod scheduling, reconciliation, and image preparation add too much overhead for very short execution turns. Activation instead restores a session onto a worker that is already there. This permits a fleet to support more persistent sessions than running VMs, because sessions waiting for inference, tools, or another user turn need not occupy active compute. The platform’s lifecycle design provides that capability.

The three-layer isolation model and where each layer can fail under real multi-tenant load

Execution isolation is only the first of three layers. A separate kernel contains execution, but it does not decide which tools an agent may call or whose context it may retrieve. Those decisions belong to the application layer. A microVM can run exactly as designed while an agent inside it receives another tenant’s tool definition or memory result.

Tool isolation starts by limiting the available tools to those for each tenant. Shared HTTP clients and connection pools must be thread-safe or tenant-scoped so that shared infrastructure does not defeat that scope. Tool selection must come from the filtered available-tools dictionary. Otherwise, prompt injection can open a path to another tenant’s tools.

Context isolation requires tenant scoping at query time. Embedding stores and index partitions must be strictly partitioned so that semantic nearest-neighbor searches cannot cross tenant boundaries. Soft tags on shared storage are not enough. Storage backends also need per-tenant encryption keys bound to verified tenant sessions, so access to raw records does not itself grant the ability to decrypt another tenant’s data.

Persistent memory adds an active attack surface beyond accidental leakage. Stateful Agent Backdoors, arXiv 2605.06158, demonstrates that a backdoored model can use persistent memory to connect local sub-backdoor behaviors into a complete cross-session attack program. The attack depends on state surviving between sessions. Persistence therefore preserves more than useful task progress, and isolation controls must account for hostile state as well as unauthorized access. Snapshot fidelity does not distinguish between the two.

How idle cost and runaway spend change the economics of persistence at scale

An always-resident environment incurs cost even when its agent has nothing to do. Across a large fleet of intermittently active sessions, idle cost can dominate the bill. Suspension changes that relationship by releasing compute between turns. Its economic value depends on both state fidelity and restoration latency, because releasing resources is not useful if resumption loses required state or takes too long for the workload. The billing model must also charge differently for a persistent session and an actively running one.

Runaway spend is a separate failure mode. A documented production post-mortem describes a two-agent loop consuming a large sum over an extended period while dashboards were running. Visibility did not stop the loop. The practical control is a per-agent daily budget limit enforced by an infrastructure-level circuit breaker that blocks a call before it is sent. An alert after spending occurs cannot substitute for that gate.

Swarm-scale eval and RL workloads have a different cost profile. Across thousands of rollouts, model tier, cache hit rate, and disposable worker capacity can outweigh the economics of preserving individual sessions. Prices within model families span roughly 100×. Cached input costs roughly 10× less than uncached input, while spot compute can suit rollout workers when losing progress on interruption is acceptable. Persistent sessions, unlike throwaway workers, must keep their unfinished work through a reclaim.

What compliance requirements add to the substrate decision in regulated workloads

For regulated workloads, some non-negotiables are externally defined. The substrate determines which vendors enter the execution path and therefore the compliance chain. Every vendor that touches ePHI, including the model provider, hosting provider, vector store, and observability provider, requires a signed BAA, on a HIPAA-eligible product tier where required. Even with a strong execution boundary, those agreements are still required. Nor does an agreement establish that the runtime preserves state or isolates tenants correctly.

For desktop automation integrated with electronic health record systems and involving legacy Windows apps, Citrix environments, or phone-only apps, every tool call, screen interaction, and data access must be traceable for review. An execution record needs to show what ran, what it produced, and which resources it consumed. This is a property of the execution environment. Adding a general activity log around an otherwise opaque session does not provide the required step-level account.

Strict index partitioning and tenant-bound encryption keys support minimum-necessary access and access controls, although they do not establish compliance on their own. Encryption also should not be presented as an unconditional current HIPAA mandate. HHS’s Security Rule summary distinguishes the minimum-necessary Privacy Rule standard from the Security Rule’s access-management provisions. HHS’s 2025 proposed rule would make encryption mandatory, though it is not yet required in all cases.

A decision framework for matching substrate to workload before committing to an implementation

The decision procedure starts by specifying what must survive and what must remain separate. The next step is to establish how long sessions must exist, what interruption is acceptable, and what idle resources the budget can support. Those answers eliminate unsuitable implementations before startup speed or familiar tooling breaks a tie. Where a middle-ground boundary is acceptable and microVM overhead is prohibitive, gVisor remains defensible, subject to its snapshot and compatibility limits.

For short-lived internal tooling, hardened containers with external state storage fit when stored outputs are enough and the shared kernel is acceptable. External users need a different fit when their full sessions must resume after idle periods. Filesystem-only persistence is insufficient for that requirement. MicroVM-backed environments need snapshots for each session, a control plane that works apart from the compute assignment, and separate checks that enforce tool and memory isolation.

For eval pipelines, RL rollouts, and throwaway batch environments, aggregate pipeline state can remain external while individual workers are disposable. Per-second sandboxes that are deleted with their disks at task completion fit that lifecycle. Spot workers are an option only where interruption losses are acceptable, and each run still needs containment so that a failed or malicious rollout cannot affect others.

For regulated computer-use agents, the fit is a persistent microVM environment with the required Linux, Windows, or Android desktop capability, per-session audit trails, and BAA-eligible hosting. Signed-in credentials and unfinished UI work must still be there when the next turn starts. App images, network configuration, and compliance documentation may require a dedicated or managed setup that goes beyond the self-serve default. Human takeover also belongs in this evaluation as a control for supervised autonomy. The viable substrate is the one whose actual lifecycle, preserved state, tenant boundary, and idle economics match those requirements together.

Sources

  1. Stateful Agent Backdoor

More in Agent Architecture