--- title: "75,000 AI agents in a Raspberry Pi's memory: what is AWS's Infinity?" description: "AWS open-sourced Infinity under Apache 2.0, part of the Hydro project with UC Berkeley: a runtime that suspends an agent at a tool call like async/await, so about 75,000 idle agents fit in 8 GB." url: https://ihsanalapsi.dev/articles/aws-infinity-agent-runtime lang: en date: 2026-10-24 updated: 2026-10-24 tags: ["AI", "Developer tools"] author: Ihsan Alapsi (إحسان العبسي) --- # 75,000 AI agents in a Raspberry Pi's memory: what is AWS's Infinity? If you have AI agents that spend most of their time waiting on a tool, in most harnesses each of them holds a process and its memory for the whole wait. On October 23, 2026, AWS published an [official post](https://aws.amazon.com/blogs/opensource/introducing-infinity-a-runtime-for-extreme-agent-concurrency/) saying that 75,000 agents fit in the 8 GB of a Raspberry Pi 5, a number calculated from a benchmark they ran on a server. What makes this possible is Infinity: a runtime that suspends an agent like an async function when it calls a tool, and wakes it when the result arrives. By the end of this article you'll know how it works, whether it fits your agents, and how to try it now. ## What exactly is Infinity? Infinity is a research runtime for AI agents, part of the Hydro project that AWS works on with UC Berkeley, and it's on GitHub under the Apache 2.0 license. Its idea comes from the way programming languages suspend asynchronous functions and resume them later. The problem it solves is that agents spend most of their time waiting on tools. A coding agent may start a build that runs for twenty minutes, a triage agent may wait hours for the next pull request, and an assistant may wait until the next morning for an approval. Most harnesses run the agent loop as a series of steps, and each step stops until the tool returns, so a waiting agent holds a process and its memory and can't respond to new events. With Infinity, when the agent calls a tool, the runtime dispatches the call, saves the conversation, and suspends the agent. When the result arrives, it resumes the agent from the state it saved. Meta also released the weights of Muse Spark Open this month [under a license with conditions](/articles/meta-muse-spark-open-licence), but that's a model. Infinity is infrastructure that runs fleets of agents. ## Why does a tool call look like await? A tool call looks like await because in both cases the work stops until a result arrives from outside, and everything needed to continue afterwards is saved. For a function, that's the continuation. For an agent, it's the conversation history. Most developers know async and await from Rust, JavaScript, or Python. When a function waits on an asynchronous operation like a network request, it's suspended, and the thread moves on to other work. When the operation completes, the runtime resumes the function from where it stopped. To do that, it has to save the function's continuation, which is the state it needs to continue. Languages save this state in one of two ways. OS threads and stackful coroutines keep the whole call stack in memory while they wait. Stackless coroutines, which Rust uses for `async fn`, have the compiler turn the function into a state machine that holds only the values needed at each await. That uses less memory, so an executor can run a large number of them. AWS's point is that an LLM agent can be suspended the same way: at the moment it calls a tool, its conversation history holds everything it needs to continue once the result arrives. If the runtime saves that history, it can drop everything in memory, and any process with access to the store can resume the agent later. So Infinity runs each agent as a sequence of bounded slices. In each slice, the runtime loads the history from a store you can swap out, runs one completion, saves the output, and dispatches at most one tool call. Then the slice ends. Tool calls are the await points: a slice never waits on a tool, a timer, or any other outside operation. Everything that can wake an agent, from user text to tool results, arrives as a message on a shared input queue. Because the queue holds events until they're processed, the runtime doesn't need to keep the agent alive while it waits: when it pulls a message, it starts a slice that loads the agent's state and processes the message. ## Where did the 75,000 number come from? The 75,000 number is a calculation from a benchmark AWS ran: each idle agent took about 100 KB of memory, and 8 GB, the memory of a Raspberry Pi 5, holds about 75,000 agents at that rate. The benchmark itself ran on a server CPU, not on a Raspberry Pi. A suspended agent holds only its history, so the memory it needs is close to the size of that history. To measure this, AWS wrote a benchmark that launches waves of agents in one system with a simulated model. Each agent runs twenty turns, and each turn makes a full round trip through the runtime: a completion of about 1.2 KB calls a tool, the result comes back through the queue, and a second completion ends the turn. They ran it with 80,000 agents, 3.2 million completions in total, on a single thread of an AMD EPYC 9R14 with in-memory stores. Memory grew linearly, at about 100 KB per idle agent, and most of it goes to conversation histories, because an idle agent has no task, stack, or connection. ## What does zero compute while waiting mean? Zero compute while waiting means that a waiting agent on a serverless platform has no running invocation. A slice never waits on outside operations and loads its state from storage, so a function is invoked per unit of work and exits as soon as it's done. Infinity supports this with step mode: the platform calls `step` once per batch of messages, and nothing stays in memory between calls. The reference deployment AWS built uses an SQS FIFO queue as the input queue, with one `MessageGroupId` per thread. Each delivery invokes a Lambda function that runs one step and exits. Histories live in Aurora DSQL, deduplication state in DynamoDB, and the `sleep` tools use SQS delivery delays and EventBridge Scheduler for timers. The repository includes CDK constructs that set up these resources. Running on that kind of platform puts five requirements on the runtime: - Bounded execution time: a slice is one completion plus a few storage and dispatch calls, and it never waits on an operation with no time limit. - No process affinity: all state is stored by thread ID, so any invocation can process the next message for any thread. - At-least-once delivery: the queue may deliver a message again, so each message has a stable ID, and the runtime skips one it already processed, which makes processing a slice idempotent. - Saved turns: streamed model output is saved only when the turn completes, and when a tool call ends a turn, the runtime saves the turn before it dispatches the call, so no result can arrive for a call that isn't recorded. - Ordering without locks: FIFO message groups line up each thread's slices one after another, so there's no need for distributed locks. Traditional runtimes that stop at a tool call can also run on serverless, but each invocation has to stay open while its tools run. In Infinity, a waiting agent has no running invocation, so an agent waiting on a three-day CI pipeline uses no compute until the pipeline returns its result. ## How do events reach a waiting agent? Events reach a waiting agent as synthetic tool calls: the runtime adds a tool call to the history that the model never made, and its result is the event. That way the event enters the conversation in order, in a form the model knows how to read. An agent waiting on a tool needs to hear about things that happen while it waits: a long command that produces output over time, or a repository subscription that produces an event for every new pull request. In both cases one call produces a sequence of results, and the runtime has to present each one to the model in a form it understands. Model APIs have no way to represent an asynchronous event: each tool call must be answered by exactly one result. A second result breaks that rule, and models usually reject the conversation. If you add the event as a user message, the API accepts it but attributes it to the user. If you put it in the system prompt, it leaves the order of the history that the model relies on to know what happened when. So when an event arrives, the runtime adds an assistant tool call that the model never made, followed by a tool result that holds the event: ``` ▶ Synthetic: receive_event__injected({ original_tool_name: "subscribe_github_events", original_tool_call_id: "call_abc123", original_args: { owner: "acme", repo: "api" } }) ■ Result: {"event_type": "pull_request", "action": "opened", "number": 42} ``` Because the event arrives as a tool call, the model uses what it learned in training to read tool results and decide what to do with them. The arguments identify the earlier tool call that produced the event, so the model knows which of its requests it belongs to. The dedicated name `receive_event__injected` lets the model tell an event apart from a new call to the same tool, and the `__injected` suffix says the runtime created it. The runtime never offers this name to the model as a tool, and it rejects the call if the model tries to use it. A tool produces events the same way it produces its first result: by sending messages to the agent's input queue. It marks the first result as the start of a subscription, so the runtime knows more results for the same call may follow. The agent stays idle between events, and each event wakes it on arrival, in a local process or through a new Lambda invocation, so a subscription keeps nothing running while it waits. Where an event gets processed is a separate choice. If a subscription sends a hundred events to one agent, they take up a large part of its context window, so the tool picks one of two strategies for each event. The default: each event is handled by a temporary child thread that inherits the parent's context. The child acts and reports to the parent, or closes without a report if the event doesn't matter, in which case the parent's history doesn't change, and events that arrive close together are handled in parallel. The other: an event marked associative goes straight into the history of the thread that subscribed, which is meant for events that update an operation the agent is following, like the output of a test run it started in the background. ## How do agents talk to each other? Agents talk to each other through the same tool calls and events: each agent has a built-in `spawn_thread` tool that creates a child thread with its own input queue. For the parent, `spawn_thread` behaves like a subscription: the call returns as soon as the child thread is created, the parent carries on with its work, and the child's reports reach it later as events on that same call. When the child calls `report_to_parent`, the parent sees the report as a synthetic tool call tied to the original `spawn_thread` call, so it knows which child sent it, even when many are running at once. A child can send several reports while it works, and the last one arrives when it calls `close_thread`. So the runtime doesn't need a separate messaging system for agents. A child thread is created with an ordinary tool call, and so is a report to the parent, and their results and events travel through the input queues, and the same ordering guarantee (each thread's messages are processed in turn, one slice at a time) applies to them. The parent and its children run in parallel as separate threads in the same agent system. For example, an agent reviewing a pull request can spawn a child thread per file: each child checks its file and sends its report, and the parent collects the reports and writes the review. Several agents on the same change is the same pattern that tools like Claude Code Review use ([AI code review on GitHub](/articles/ai-code-review-github-claude-ga)), and Infinity lets the waiting agents in that pattern use no compute. ## Can I try it now? Is it production-ready? You can try Infinity now, but don't depend on it in production yet: AWS calls it an early research release, shared to get early feedback. AWS also lists open questions: how models reason about interleaved events, scheduling policies, compaction in a large tree of agents, and how the runtime proves where an event came from. To try it, open the [quickstart](https://infinity.hydro.run/docs/infinity-runtime/quickstart/launching-your-first-agent): according to AWS, you build a simple agent on Infinity there in a few minutes. The library is written in Rust. Start in local in-memory mode before you think about a serverless deployment, because the tools, model providers, and tests you write against the core library run in both modes without changes. If you run agent fleets, like CI triage, PR bots, or coding agents, this architecture is worth studying. And if you care how AI search agents read your site, see [what the Cloudflare and OpenAI pilot taught us about ChatGPT search](/articles/chatgpt-search-cloudflare-discovery-pilot). If you had 75,000 agents that sleep and wake on demand, using no compute while they sleep, what would you build with them first? ## FAQ **How does Infinity suspend an agent during a tool call?** When the agent calls a tool, the runtime dispatches the call, saves the conversation history to its store, and ends the slice without waiting. A suspended agent holds only its history. When the result arrives on the input queue, a new slice loads the history and continues from where it stopped. **What does a waiting agent cost?** On a serverless platform, zero compute: the Lambda function exits as soon as the slice ends, so an agent waiting days on a CI pipeline uses no compute. What remains is the conversation history saved in storage. In local mode, an idle agent takes about 100 KB of memory, according to AWS's benchmark. **How do I try Infinity?** Open the quickstart at infinity.hydro.run: according to AWS, you build a simple agent there in a few minutes, and the library is written in Rust. Start in local in-memory mode before you think about a serverless deployment, because the same tools and tests run in both modes without changes. **Is Infinity production-ready?** No. AWS calls it an early research release, shared to get early feedback. Open questions remain: how models reason about interleaved events, scheduling policies, compaction in a large tree of agents, and how the runtime proves where an event came from.