Agent Interview Deep Dive: Part 1: Runtime and Context · Part 2: Memory, Tools, and Concurrency · Part 3: Evaluation, Safety, and Viability

Part 1 established the runtime skeleton: a state machine drives the loop, checkpoints make recovery possible, and each context window is assembled under a token budget. Now put that system in a real multi-user environment.

This is much harder than connecting a vector database or an MCP server. Long-term memories become duplicated, contradictory, and stale. Tools time out, cross permission boundaries, and create irreversible side effects. Two workers may execute the same step at the same time. Part 2 focuses on these consistency and concurrency problems.

Long-Term Memory Needs More Than Insert and Search

Embedding every conversation and retrieving the top K by similarity does not produce long-term memory. It quickly produces duplicates, conflicts, stale facts, permission leaks, and unbounded growth.

Separate the Memory Layers

Layer	Lifetime	Example
Working memory	One model call	Current prompt and recent observation
Session state	Current run	Plan, step, checkpoint, and budget
Episodic memory	Across runs	A deployment failure and its cause
Semantic memory	Stable facts	User preferences, system constraints, domain knowledge
Procedural memory	Long-lived rules	Skills, operating steps, acceptance criteria

LangChain’s context-engineering documentation similarly distinguishes runtime context, session state, and cross-session stores, and uses lifecycle middleware for summarization, tool selection, and state updates. Context Engineering

The value of these layers is operational, not terminological. Each type has different admission and retention rules. A tool response is an observation; similarity to the user’s question should not automatically promote it into a permanent fact. One successful run should not immediately become a timeless procedure.

Design a Forgetting Policy

A real Memory Service must support writing, retrieval, merging, revision, and forgetting:

• Deduplicate facts about the same entity.
• Let newer facts supersede older versions while preserving an audit trail.
• Attach TTLs to temporary information.
• Decay memories by importance, confidence, and usage frequency.
• Consolidate multiple episodes into semantic memory.
• Move long-unused items to cold storage.
• Set capacity quotas per tenant and user.
• Let users inspect, correct, and delete their memories.

Deletion must cover more than the vector index. The source object, derived summaries, embeddings, caches, and search indexes all need to follow the same deletion workflow. Otherwise a user can delete a memory and still see the Agent reconstruct it from an old summary.

Apply Permission Filters Before Similarity

Each memory should carry provenance, time, tenant, user, permission scope, confidence, and version. Retrieval cannot be based on vector distance alone:

WHERE tenant_id = :tenant_id
  AND permission_scope && :allowed_scopes
  AND valid_from <= :now
  AND (expires_at IS NULL OR expires_at > :now)
ORDER BY relevance_score DESC, confidence DESC, last_used_at DESC
LIMIT :budgeted_k

Apply hard filters first and semantic ranking second. A highly similar memory from another customer is exactly the kind of result that creates a serious data leak. Retrieval must also respect the current token budget: top K describes a candidate set, not a requirement to place every candidate in the prompt.

Decouple the Tool Layer from the Model

A production tool cannot be just a Python function name plus a description. It needs an execution contract:

name: orders.refund
version: 2
input_schema: RefundRequest
output_schema: RefundResult
timeout_ms: 10000
retry_policy: no_automatic_retry
idempotency: required
permission: orders.refund
side_effect_level: irreversible
approval: required
error_taxonomy:
  - validation_error
  - permission_denied
  - provider_timeout
  - business_rejected

The Tool Registry describes and discovers tools. The Tool Gateway controls their execution. The model submits a structured call; it does not receive third-party credentials or connect directly to a database.

The boundary is easy to blur. The registry answers, “Which tools exist, and what are their arguments?” The gateway answers, “Can this tenant call the tool, does this invocation require approval, and may this failure be retried?” Combining them on the model side exposes descriptions, credentials, and policy to an uncertain decision-maker.

Do Not Return Large Results Verbatim

A tool can produce four representations of one result:

• Raw result: stored in object storage for auditing and download.
• Structured result: consumed by deterministic program logic.
• Model summary: only what the next decision requires.
• Typed error: drives retry, argument repair, replanning, or termination.

If a database query returns 200,000 rows, the model does not need all of them. It needs the schema, row count, aggregate statistics, anomalous samples, and a reference to the original artifact.

Anthropic’s work on tool engineering emphasizes clear definitions, composability, careful use of context, and repeated improvement through evaluation. Writing Effective Tools for AI Agents

MCP standardizes how tools are exposed to clients. It does not automatically provide tenant isolation, approval, idempotency, execution sandboxes, or enterprise audit. Those remain harness responsibilities.

Handling Multi-User Concurrency

Once an Agent supports multiple users, it encounters three kinds of conflict:

1. One user opens two sessions that modify the same run or document.
2. Different users operate on one business object, such as a ticket, branch, or order.
3. Two workers both believe they own one run and duplicate an external side effect.

“One session per user” does not solve these conflicts. A session isolates conversational context. It does not isolate shared business objects, run ownership, or external side effects.

Carry Tenant Identity Through Every Layer

Runs, steps, messages, memories, tool calls, traces, and artifacts all need a tenant boundary:

tenant_id / user_id / session_id / run_id / step_id / tool_call_id

Checking the tenant only at the API gateway is not enough if internal queues or memory stores later drop tenant_id. Isolation must survive the complete path through entry points, state, queues, tools, storage, and logs.

Use Versions for State Conflicts, Not Long Locks

Update a run with an expected version:

UPDATE agent_runs
SET state = :new_state,
    version = version + 1
WHERE id = :run_id
  AND tenant_id = :tenant_id
  AND version = :expected_version;

If zero rows are updated, the state has changed. Reload it and decide whether to replan instead of overwriting somebody else’s result.

A pessimistic database lock should enclose a millisecond-scale atomic update, never a model call or network request. Holding a transaction while a model thinks for thirty seconds will exhaust the connection pool and destroy concurrency.

Give Workers Leases and Heartbeats

A worker acquires a time-limited lease, not permanent ownership. It renews the lease while running. If it crashes, another worker may take over only after the lease expires. The new worker resumes from the last committed checkpoint instead of guessing how far the old worker progressed.

The lease must tolerate normal heartbeat jitter without delaying recovery for too long. A practical starting point is three to five heartbeat intervals. After renewal fails, a worker should stop taking new steps. Immediately before an external side effect, it should verify that both the lease and state version are still valid.

Make Side Effects Idempotent

idempotency_key = run_id + step_id + tool_call_id

A refund, message, creation, or write with the same key may succeed only once. Exactly-once execution is rarely achievable across an entire Agent system. The practical design is at-least-once scheduling plus business-level idempotency.

When a database state update and an event publication must either both happen or neither happen, use a transactional outbox. Write business state and the outbox row in one transaction, then let a separate relay publish the event.

Planning Is More Than a Polished Checklist

A static plan can become obsolete after its first action. An effective plan contains:

• An observable final objective.
• Dependencies among steps.
• Acceptance evidence for each step.
• Current state and unresolved blockers.
• A recovery policy for failure.

Replan when tool output contradicts an assumption, a dependency is unavailable, the remaining budget is clearly insufficient, or two consecutive actions produce no new evidence. Replanning should explicitly revoke invalid assumptions, preserve valid evidence, and explain why the route changed.

Prefer One Agent by Default

Subagents are useful for parallel retrieval, independent review, context isolation, and separate domains. Use multiple Agents only when the task is genuinely decomposable, roles are materially different, and outputs can be verified.

Common multi-Agent failures include:

• Duplicate searches and tool calls.
• Divergent state versions.
• Agents reinforcing one another’s mistakes.
• Communication tokens exceeding useful work.
• No single component owning the final result.

A delegation contract should define the input and output schemas, budget, deadline, and completion conditions. A subagent should not return “research complete.” It should return a verifiable result package.

The State Part 2 Leaves Behind

The Agent now has memory lifecycles, tool contracts, versioned multi-tenant state, worker leases, and idempotency keys for external side effects. Retries and concurrency are far less likely to corrupt data.

But “did not corrupt state” is not the same as “completed the task correctly.” Part 3 asks how to reconstruct a run, test a nondeterministic trajectory, place safety boundaries, and decide which Agent products deserve investment.

Continue: Part 3: Evaluation, Safety, and Product Viability

Agent Interview Deep Dive: Part 1: Runtime and Context · Part 2: Memory, Tools, and Concurrency · Part 3: Evaluation, Safety, and Viability

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.