Scaling From One to One Million Per-User Agents Without Rewriting Your Infrastructure Layer

Per-user persistent AI agent deployment at scale is an infrastructure design problem. Scaling from one to one million agents without replacing the foundation requires VM-level isolation, persistent state, snapshot-restore scheduling, and economics that charge for activity. Standardised tool access and per-agent identity make that foundation manageable as the fleet grows.

Why scaling per-user agents is an infrastructure problem, not an orchestration problem

The rewrite trap starts with a prototype that works. A team builds on stateless serverless functions, reaches thousands of agents, and finds that it must replace the runtime beneath its agent logic to add durable per-user environments. The original choice worked at one agent or ten because those deployments did not expose its structural limits.

Each new user adds more than an inference workload. The agent accumulates context, retrieves memory, and calls tools, with those demands multiplying against each other as the fleet grows. Model routing and orchestration frameworks can tune how work reaches the runtime, but they cannot change what that runtime preserves or isolates. The design prescription is therefore to select the execution environment before optimising the logic above it.

What "persistent state" actually requires at the infrastructure level

Persistent agents need three distinct layers of state management. The first is the in-context working buffer, a sliding-window scratchpad used for reasoning within a session and flushed when that session ends. It is the cheapest layer and the shortest-lived. As the buffer approaches its token limit, summarisation compresses older turns, invalidating the KV cache and causing a latency spike on the next call. A persistent runtime does not remove this tradeoff, so its state model must distinguish session context from durable execution state.

The second layer is execution checkpointing. Graph-based frameworks write variables, history, and workflow position to a durable store after each step, allowing a crashed or paused agent to resume from its saved position. Resumption does not provide exactly-once semantics, however. A node that ran partway before a crash may run again, so side-effecting steps must be idempotent. The runtime must support durable checkpoints, but recovery alone cannot prove that an external action happened only once.

The third layer is semantic memory, which stores facts, preferences, and domain knowledge in an external vector store with metadata filtering. Retrieval injects the relevant records into the prompt at query time. Extraction often adds an LLM call per turn, and when facts from different sessions conflict, the retrieval layer must resolve them. Memory feeds task state at the start, state changes during execution, and selected state returns to memory at the end. Broken state loses the task’s place, while broken memory loses personalisation. Supporting this cycle requires a runtime with persistent disk, state that survives sleep and wake cycles, and a private address space for each agent.

Why containers are not a sufficient isolation boundary for multi-tenant agent fleets

Containers use the same host kernel for their workloads. For agents that install packages, execute code, and hold separate users’ credentials, that shared kernel is the weakness in the isolation boundary. A kernel bug reachable from one container can compromise the host and neighbouring containers. Even with separate filesystems and processes, tenants still share a kernel.

With a micro-VM, the agent has its own guest kernel behind a hardware virtualisation boundary. Firecracker runs one VMM process per micro-VM rather than a single daemon managing all VMs, supporting security and operational simplicity with fast boot, low overhead, and high density. AWS uses Firecracker as the foundation for Lambda. Each function execution environment has one micro-VM, reused across warm invocations of the same function, while the newer Lambda MicroVMs product uses a per-session isolation model. This establishes a production-scale precedent without equating every Lambda invocation with a dedicated persistent agent.

High density makes it practical to give each user that boundary. Because agents spend most of their time idle, oversubscription can make dedicated isolation affordable when sleeping VMs consume only storage. Isolation is not the whole security story, however. A misconfigured network interface can still allow unrestricted internet access, and credentials baked into a root filesystem can leak into every run. Egress policies need separate audits, and secrets should be injected at runtime.

How snapshot-restore scheduling turns an idle fleet from a cost problem into a cost advantage

In a multi-tenant deployment, most agents are idle at any given moment, so keeping each environment running charges the operator for users who are doing no work. Snapshot-restore replaces that idle compute commitment with stored state. When an agent sleeps, the runtime checkpoints its full VM state, including memory, disk, and in-flight context. A later request restores the environment.

AWS reports a P75 snapshot wake time of 1.9–2.0s. The operator pays for storage while the agent sleeps and for compute only while it runs. This supports a fleet that sleeps for a few cents per tenant and is billed for concurrency. The economic distinction is between owning an agent and running it. A large idle population need not imply a correspondingly large active compute bill.

AWS’s microvm-ctl pattern, published September 2026, demonstrates one Firecracker micro-VM per tenant, conversation memory in RAM, and identity injected at launch without a bespoke control plane. A production-grade snapshot-restore scheduler requires sustained infrastructure engineering. Managed platforms that supply it as a primitive remove the need to build that control plane for teams below the utilisation threshold where self-hosting becomes competitive.

MCP as the tool-layer primitive that prevents integration debt from compounding at scale

Compute is not the only source of rewrites. Without a standardised tool protocol, each integration needs its own code, which the team must maintain across the tenant fleet. Model Context Protocol provides a common tool surface, reducing the need to encode every connection as bespoke agent infrastructure. Anthropic donated MCP to the Agentic AI Foundation, a directed fund under the Linux Foundation, in December 2025. Its path from internal experiment to that stewardship took roughly 12–13 months, and the ecosystem now hosts thousands of public MCP servers.

Code execution via MCP can let agents load only the tool definitions needed for the current task. As integrations accumulate, selective loading keeps the tool-definition token surface from growing with the entire catalogue. Standardisation does not settle access control, however. Microsoft Entra Agent ID gives each agent its own governed identity, so identity can be injected at VM launch instead of having agents use shared secrets or generic service accounts.

Google Cloud’s multi-tenant agent architecture shows how that separate security layer works. A Principal Access Boundary (PAB) Policy works with VPC Service Controls to limit which resources each agent can access and block access to other tenant projects or unauthorised services. Model Armor with Sensitive Data Protection inspects and dynamically masks PII before the agent processes it. These controls set rules for tool access and data handling, which are separate from guest-kernel separation. A standard tool interface thus needs its own policies for identity and resources even when every agent runs inside a VM.

The autonomy and guardrail decisions that determine whether a scaled fleet is safe to operate

Scale amplifies the consequences of poorly bounded autonomy. A fleet needs reversibility checkpoints and hard resource boundaries for every agent, rather than protection only for workloads a developer happened to flag. VM isolation provides a policy boundary on which those controls can be enforced, but it does not decide which actions deserve approval. An isolated agent can still take an authorised action with an undesirable result. Safe operation requires both containment and a policy for consequential actions.

Bounded autonomy preserves automation while keeping failure recoverable. Agents can draft high-value messages without sending them, or commit to branches but need approval for merges. The objection that human gates erode autonomy is valid when every action faces the same interruption. Gates should instead depend on reversibility and value thresholds, with enforcement in the infrastructure so the policy applies consistently across tenants. Google Cloud’s resource boundaries demonstrate the complementary principle that an agent’s inability to reach an unauthorised resource should not depend on its own reasoning.

A fleet also needs clear roles, deterministic execution paths, observability, and rollback from the start. Per-agent kernels provide isolation, while hard runtime resource policies constrain runaway work and protect neighbouring tenants. Above that boundary, egress controls, network interface audits, and runtime secret injection address risks that VM separation cannot resolve. Designed together, guardrails compose with isolation and snapshot-restore scheduling.

The pricing model question and why per-seat licensing breaks for agent fleets

Agent work does not scale in proportion to seats. Its cost grows with autonomy and volume, so a pure per-seat model can over-provision light users or impose restrictive limits on heavy users. Charging for every agent creates a related problem by making adoption expensive even when those agents are idle. The commercial unit must distinguish access from activity.

Microsoft Agent 365 went generally available May 1, 2026, with per-user licensing. One licence covers every agent associated with that user, avoiding a governance charge that rises with each additional agent in the account. That distinction addresses agent-count growth, but it does not make all users equally costly to serve. The most active users consume a disproportionate share of inference budget. A product price that ignores that concentration risks losing margin on the users most engaged with the service.

A base subscription plus usage or outcome components is the dominant structure among successful AI companies, and it maps naturally to an idle-storage, active-compute architecture. The base can pay for the storage that persistent agents need, while variable charges cover the work they do. Outcome components give companies another way to charge, but the infrastructure bill still depends on what runs. Snapshot-restore makes that distinction tangible by separating the low cost of retaining an agent from the cost of putting it to work.

How do you deploy thousands of persistent agents without rebuilding infrastructure?

The foundation decisions belong at agent one. A team should select VM-level isolation, persistent disk and memory across sleep cycles, snapshot-restore scheduling, MCP-compatible tools, and identity injected at launch. Isolation provides an enforceable boundary, scheduling makes that boundary economical, and standardised tools keep integration work manageable. Per-launch identity then makes access attributable without requiring manual security configuration for every agent.

Buying those primitives can be more economical than building them. Self-hosted Firecracker infrastructure typically becomes cost-competitive only at sustained high volume, with the crossover driven primarily by operations labour. Depending on those costs, the threshold can range from a few hundred to several thousand sandbox-hours per month. Below it, total cost of ownership favours a managed platform that provides isolation, scheduling, and persistence.

The practical sequence is to choose infrastructure first, connect MCP-compatible tools second, design roles and guardrails third, and select orchestration patterns fourth. ReAct, plan-and-execute, routing, reflection, and parallelisation sit above the runtime and are largely portable. Choosing the foundation preserves the freedom to change the agent’s reasoning strategy without replacing the layer responsible for state, tenancy, and operating cost. The rewrite trap is avoided by making those obligations native to the first deployment.