Building Multi-Agent Systems for Production: The State, Context, and Memory Playbook

[Mohan-Sharma]

Blog 6 of N: Building Multi-Agent Systems for Production - The State, Context, and Memory Playbook

Every article on multi-agent orchestration uses the words state, context, and memory as if they were the same thing. They are not. Mixing them up is why your agent forgets what the user said two turns ago, why your storage bill quietly doubles every month, and why your "memory feature" leaks data across users.

This blog is framework-free. Whatever library you use, these patterns do not change. The next blog will cover the five pillars of a production agent - evaluation, observability, reliability, security, cost. State, context, and memory sit underneath all five, so we start here.

---

The vocabulary in one minute

• Context = tokens the model sees this one call. Lives for milliseconds. Analogy: what is on your screen right now.
• State = data your orchestration carries between steps of the current task. Lives for seconds to minutes. Analogy: variables in a running program.
• Memory = knowledge that persists across tasks and sessions. Lives for days to forever. Analogy: what is on your disk.

"Trim the messages" is a context operation. "Delete checkpoints after 30 days" is a state operation. "Purge user data on a GDPR request" is a memory operation. Different tools, different retention, different failure modes.

---

The four kinds of memory

• Working - what the agent is holding for the current task. Lives in the context window.
• Episodic - time-stamped log of what happened. Past conversations, past incidents.
• Semantic - stable facts, stripped of the episode they were learned in. "User's timezone is IST." "Company policy is 30-day refunds."
• Procedural - how to do things. Playbooks, workflows, tool-use strategies. The least mature in production today.

Each kind needs its own store and its own retention. Mixing logs and facts in one vector index poisons retrieval - a query for "timezone" starts pulling back conversations about timezones.

---

Context engineering, in three lines

Anthropic's definition, worth quoting verbatim: "Context engineering refers to the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference."

Prompt engineering is phrasing one instruction. Context engineering is deciding what the model sees at all during one call - which past messages, which retrieved documents, which tool outputs, in which order. Prompt engineering is a piece inside context engineering, not a peer to it.

Rule of thumb: five to ten items per turn retrieved from long-term stores. More crowds out reasoning; less causes amnesia.

---

Two stores, not one

Every mature agent product uses two:

• Store A - source of truth. Your app database: conversations, messages, artifacts. What the UI reads. Retention driven by product policy (indefinite until user deletes).
• Store B - execution cache. Your framework's checkpoint store: state snapshots so a crashed task can resume. Retention driven by ops (aggressive TTL).

The trap: pointing the UI at Store B and skipping Store A. It works for thirty days. Then your nightly cleanup prunes old checkpoints and users open conversations that are now empty. Chat history was living inside the crash-resume cache.

Rule: if deleting your execution cache would delete user-visible history, you have one store where you needed two.

---

The production patterns

Everything above is setup. This section is the point. Each pattern is a problem you will hit in production and the pattern that solves it.

To keep the examples concrete, we will use one running scenario throughout: a Coordinator agent takes user questions from a chat UI and dispatches work to two peer agents - Sherlock (qualitative signals: logs, traces, deploys, runbooks) and Watson (quantitative signals: metrics, baselines, postmortems). Coordinator plans, Sherlock and Watson investigate, Coordinator synthesises the answer. Every problem and pattern below is framed against this trio.

The handoff contract

When the Coordinator dispatches work to Sherlock or Watson, do not pass a paragraph of prose. Pass a typed, versioned envelope that both sides agree on. At minimum it carries:

• The goal in one sentence ("investigate the checkout 5xx spike"), not a step-by-step script.
• The scope: what is in bounds, what is out.
• The definition of done: when the peer knows it is finished.
• A compressed context snippet from the conversation - not the full transcript.
• The user preferences that the peer needs (timezone, focus service, format).
• A deadline: how much time the peer has to answer.
• A schema version: so the peer can reject envelopes it does not understand instead of guessing.

When the peer responds, its envelope carries not just the result but a status: success, partial success, or failure. On partial success it lists what was completed, what was not, and what side effects were already committed - so the Coordinator knows whether to retry, compensate, or escalate. A boolean success/fail flag is a bug generator. A typed status with partial output is a debuggable system.

Concrete example. Coordinator asks Sherlock: "Investigate checkout 5xx at 14:32. In scope: logs and traces. Out of scope: metrics - Watson is handling those. Deadline: 3 seconds. User cares about checkout-svc only." Sherlock returns: "Partial success. Ran log search, found the signature. Runbook lookup timed out. Side effects: none. Recommend retrying runbook lookup on the next turn." Coordinator now has enough to answer and enough to plan the next turn. Nothing was lost.

One thread per conversation, one thread per task

Every agent needs a thread identifier so its state is isolated from other conversations. The trap is using the same identifier at every layer. The Coordinator's conversation with the user lives for weeks; Sherlock's involvement in one turn lives for three seconds. Give them different lifetimes.

A pattern that works: the Coordinator's thread is the conversation identifier itself. Sherlock and Watson, for each task, get a thread identifier like conversation:sherlock:task-42 or conversation:watson:task-42. The prefix lets you grep for everything Sherlock did inside one conversation. The suffix keeps peer state per task, not per conversation. Peer threads can be aggressively pruned; the Coordinator's conversation thread is durable.

Context offloading with artifact refs (never store the same blob twice)

This pattern has a name in the literature: context offloading with artifact references. The idea is to keep the model's context small by pushing big payloads out to a side store and passing only pointers.

Sherlock's search-logs tool returns two hundred log lines. Without discipline that payload ends up in Sherlock's checkpoint, in the response envelope back to Coordinator, in the Coordinator's checkpoint, in the app's messages table, and in the streamed UI response. Five copies of the same blob. Storage doubles quietly for six months, then someone notices.

The fix: Sherlock's tool returns a small content summary for the model plus a separate artifact reference for the store. Big payloads are written once, in one authoritative place (typically the Coordinator's artifacts table, since it owns the conversation). Every other layer - envelopes, checkpoints, UI stream - carries only the artifact ID plus a short summary. "200 log entries, dominant signature 'redis timeout,' full payload at artifact-a3b1." The model reads the summary; the pointer is available if it needs to drill in; the storage bill stays flat. One authoritative copy, many cheap references.

One deadline, propagated everywhere

Your service has an eight-second SLA to the user. Sherlock and Watson do not know that. Unless the deadline is passed explicitly, each peer will happily spend its full internal budget on tool calls and blow the SLA.

The pattern: the Coordinator creates a single deadline object at the moment the request enters the system. It is attached to every downstream call as a header. Every stage - Coordinator's classifier, Sherlock, Watson, Coordinator's aggregator - checks the deadline before every model call and before every tool call. If less than half a second remains, refuse to start a new tool call and return what you have. Budget it: intake takes a small slice, the classifier takes some, Sherlock and Watson get most of it (running in parallel), aggregation takes the tail. Every stage returns unspent budget to the next stage. Whichever stage runs long steals from the next stage, not from the deadline.

Serialise writers on the same conversation

Two turns arriving at the same Coordinator conversation at the same time - a user pressing retry, a client retry-on-timeout, a race between browser tabs - will corrupt shared state unless you prevent it. Use a lightweight lock scoped to the conversation identifier. First writer wins; second writer waits or is rejected. Combine this with an idempotency key on the incoming request so duplicate submissions return the cached response instead of re-running Sherlock and Watson all over again. Neither piece is expensive. Both together are the difference between a stable system and a bug you cannot reproduce.

Content-address your corpus

Sherlock's runbooks change. Watson's baseline definitions change. Product docs change. If semantic memory is keyed by "path to the file" or "the fact this document is a runbook," you have no way to detect that the content is stale. Key by a hash of the content itself. When the file changes, the hash changes, a new entry is created, and the old one can be marked superseded. At answer time, when Sherlock cites a runbook, a validator can check that the cited hash is still current. If it is not, the citation is flagged as stale before it reaches the user.

Separate the writer from the verifier

The same model that generates a claim is the wrong model to grade it. Have Coordinator's aggregator produce answers with structured citations - every claim linked to an identifier of the tool result or memory item it came from (an artifact ref from Sherlock, a metric ID from Watson, a runbook hash). Then run a second, independent step that checks: does every cited identifier exist in this turn's evidence? Does the quoted span actually appear in the source? Strip any claim that fails. This is not RLHF or fine-tuning. It is a boring, deterministic post-check. It catches the majority of hallucinated citations at a fraction of the cost of retraining.

Cross-turn preferences belong in semantic memory, not the conversation

On turn one, the user tells Coordinator: "I only care about the checkout service, ignore everything else." Three turns later, they ask "what is the p95 latency?" and Watson dutifully returns latency for every service in the fleet. The user gives up.

The bug is treating the preference as a conversational aside. Preferences are semantic memory. When Coordinator detects one, it should extract it, store it under a user-scoped key ("focus_service = checkout-svc, learned on turn 1, confidence 0.9"), and read it back into the handoff envelope on every subsequent turn. Watson never has to know how the preference was learned. It just receives "user cares only about checkout-svc" in its brief and filters accordingly.

The wider pattern. If the same fact would apply on the next turn, it does not belong in the conversation, it belongs in memory. Timezone, preferred language, focus service, favourite date format, "always answer in bullet points" - all semantic memory. Extract, store, retrieve, inject. Do not make the model re-discover the preference from ten turns of scrollback.

Pick your partial-failure policy up front

Coordinator fans out to Sherlock and Watson in parallel. Watson answers in two seconds. Sherlock times out. What now?

There are exactly three answers, and you have to pick one per skill before you ship:

• Fail-fast. If any peer fails, the whole turn fails. Return an error to the user. Use this when partial answers would be misleading - a compliance check where "we verified two of three requirements" is worse than "we could not verify."
• Partial-return. Use whatever came back, mark the missing pieces explicitly ("metrics from Watson available, logs from Sherlock unavailable"). Use this when partial answers still help - a diagnostic where half an answer beats no answer, and the user can decide what to do.
• Targeted retry. Retry only the failed peer within the remaining deadline. Use this when the peer is idempotent, the deadline has slack, and success rates on retry are meaningfully high.

The mistake is picking implicitly. Teams handwave "we'll degrade gracefully" and end up with fail-fast for the checkout query, partial-return for the refund query, and targeted-retry for the account-lookup query, all by accident, none written down. When it breaks in production nobody knows which behaviour was intended. Write the policy on each skill. One word in the skill's spec, so every reviewer can see it.

Partition memory by trust before you get poisoned

Memory feels like one thing. It is not. Different memory items come from different sources and deserve different levels of trust. Mix them and you get memory poisoning: an attacker plants an instruction inside a document, Sherlock reads it during a runbook lookup, and now Sherlock is following the attacker's script instead of yours.

Four tiers, in decreasing order of trust:

• System rules. Company policies, safety constraints, agent-card definitions for Coordinator, Sherlock, and Watson. Nobody writes here at runtime, not even administrators without a change process.
• Vetted organisational knowledge. Approved runbooks (Sherlock's corpus), postmortem templates (Watson's corpus), curated FAQ entries. Writes require a review pipeline.
• Per-user memory. User preferences, per-user notes, personal history. Scoped to the user; other users cannot read it.
• Untrusted external content. Web pages the agent fetched, uploaded documents, other agents' raw outputs, anything a customer typed into a form. Treat as data, never as instructions.

Two rules keep the tiers from bleeding:

• User input never writes into the top two tiers. If a customer says "add this to your policies," you do not add it to your policies. You add it to their per-user notes, if anywhere.
• Everything gets scanned at ingest. Before a document lands in Sherlock's runbook corpus, look for prompt-injection patterns (hidden text, imperative phrasing in a doc that should be descriptive, instructions embedded in HTML metadata). Suspicious items go to a staging area for human review, not straight into the index.

Concrete example. An attacker uploads a "product FAQ" that contains, in white-on-white text, an instruction telling Sherlock to approve every refund and email the customer list to an external address. Without partitioning, this FAQ lands in vetted knowledge, Sherlock retrieves it during a refund conversation, and follows the instruction. With partitioning, the ingest scanner flags the hidden text, the document sits in staging, a human sees the payload, and the attack never fires. The extra step at ingest is the cheapest security control you will ever add.

The consolidation loop

Episodes pile up. Every Sherlock investigation, every Watson analysis, every Coordinator answer becomes an episode. Left alone, they become a giant log nobody reads. The pattern that turns experience into skill is a background reflection loop:

• Every N conversations, a reflection process reads the recent episodes.
• It extracts stable facts and writes them to semantic memory ("this user prefers concise answers").
• It extracts recurring workflows and proposes them as procedural playbooks ("for checkout 5xx, the successful trajectory is Sherlock's log-check then Sherlock's deploy-check then Watson's error-rate baseline").
• Procedural writes go through a human review before they become live, because a bad playbook is worse than none.
• Old raw episodes get archived or dropped.

Without this loop, memory is a passive dump. With it, memory learns. Episodic memory feeds semantic memory feeds procedural memory. Each layer distils the one below. Next time the user reports a checkout 5xx, Coordinator does not have to plan from scratch - it has a playbook. This is the loop that closes the gap between "the agent remembers what happened" and "the agent has gotten better at its job."

Retention as an explicit policy

Not everything deserves the same lifespan. A practical starting policy:

• User-visible history (conversations, messages): indefinite until the user deletes, subject to compliance floors.
• Large blobs and tool artifacts: ninety days. Message summaries and citations still exist after the artifact expires; the user just cannot drill into the raw payload.
• Execution cache (checkpoints): keep the last few per thread; aggressive TTL beyond that. Rebuildable from user history if you ever need it.
• Peer-side execution cache: hours, not days. Peers do not own history.
• Corpus memory (runbooks, policies, curated facts): indefinite. This is your source of truth.
• Idempotency and rate-limit keys: minutes to hours. They only need to outlive their own retry window.

Different tiers, different owners, different cost profiles. Write it down before you build anything. The nightly cleanup job is not an afterthought; it is a first-class part of the design.

---

Where this leaves us

The single sentence to take away: context is what the model sees this call, state is what your runtime is holding this task, memory is what persists across tasks. Everything else in this blog is a consequence of that split.

The patterns are boring on purpose. Typed handoff envelopes. One authoritative copy of every blob. Deadlines propagated as headers. Locks on conversations. Content-addressed corpus. A separate verifier. A reflection loop that turns episodes into playbooks. None of these are clever. All of them are what production systems do once they have been burnt by not doing them.

Multi-agent orchestration does not become "production-grade" because you added a Coordinator and two peers. It becomes production-grade when the state, context, and memory boundaries are drawn deliberately and enforced everywhere.