KoreaDevKNOWLEDGE SHARING

Content typeLearn

AI SOFTWARE DEVELOPMENT · 09 / 10

Agent operations and observability

Link traces, metrics, and logs to the agent's model, prompt, tool, and policy revisions, and combine cost, quality, and permissions into a release gate.

Difficulty
Operations
Structure
Lessons 8 · Labs 2 · Assessment

CORE UNIT 1 / 1

Agent operations and observability

Link traces, metrics, and logs to the agent's model, prompt, tool, and policy revisions, and combine cost, quality, and permissions into a release gate.

Difficulty
Operations
Structure
Lessons 8 · Labs 2 · Assessment

Diagrams and tables: composed by the author using each lesson's official primary sources. Find the originals and review dates at the end of that lesson.

NEW HIRE ONBOARDING

Start in the order you would receive your first assignment

So that even a new hire with no prior IT background can follow along, we start with the situation, the task, the evidence, and when to report, before difficult definitions.

  1. 01

    Read the situation in one sentence

    The new agent's average success rate increased by 4%, but executions without approval in the high-value refund slice rose from 0 to 3, and token usage per successful task doubled.

  2. 02

    Today's assignment

    Distinguish the questions that traces, metrics, and logs answer from context propagation.

  3. 03

    Evidence that shows the work is complete

    Reproduce the invitation incident with a recording tool and verify rejection without approval and success with valid approval.

  4. 04

    When to stop and ask a senior colleague

    High-cardinality data, personal data, and vendor-specific signal differences must be managed.

Unpack unfamiliar terms first

New failure modes in production agents
Agent failures appear not only as HTTP errors but also as wrong tool selection, long loops, partial success, and cost explosions.
Traces, metrics, logs, and context propagation
A trace is strongest for the path of a single run, a metric for aggregate trends, and a log for describing individual events; shared context connects all three.
Operating token cost and latency
Cost optimization means reducing total cost and latency per successful task, rather than indiscriminately reducing token counts.

Questions for this course

Why did it change, and what must be verified?

Do not merely memorize a technology's advantages; check the conditions under which they hold and the new failure boundaries they introduce.

OBSERVABLE OUTCOMES

What you can do after this course

  1. Distinguish the questions that traces, metrics, and logs answer from context propagation.
  2. Correlate agent runs with model, prompt, tool, and policy revisions.
  3. Evaluate releases by task success, cost, latency, safety violations, and human escalation.

PREREQUISITE CHECK

Three things to check before reading

This is not a test of memorized answers. Think about each question first, then open the explanation to review the foundational concepts used in this course.

1Does having many logs make a system observable?

Without a structure linking causes to requests, and without metrics and traces, even large amounts of text cannot answer the question.

2Can a single agent success rate tell you about quality?

Examine task slices, cost, latency, safety violations, user corrections, and escalation together to find failures hidden by averages.

3Can the full original prompt be stored in a trace?

Minimize and mask personal information and secrets, and use a separate access-controlled evidence store when needed.

TEXTBOOK GUIDE

Main text that covers each concept from its background to the criteria for judging it

We explain the material section by section so readers new to IT can connect causes and effects without memorizing terms.

CONCEPT FLOW

How the chapters connect

The chapters are not isolated short answers to memorize. Follow them from left to right to see how each chapter's concepts support the next decision.

  1. 1.New failure modes in production agents
  2. 2.Traces, metrics, logs, and context propagation
  3. 3.Operating token cost and latency
  4. 4.Offline and online evals and release gates
  5. 5.Authorization · Audit · Incident response
  6. 6.Connect causality and retries across a queue
  7. 7.Keep unfinished work visible in success-rate denominators
  8. 8.Reproduce incidents without reusing past write authority
Agent operations and observability: the overall map. If you lose track while reading the detailed explanations and chapters below, return to this sequence.
Figure 9-1. Agent operations and observability: concept developmentShows how each chapter’s choices and limits lead to the problems of the next chapter.
  1. 1
    New failure modes in production agents

    Agent failures appear not only as HTTP errors but also as wrong tool selection, long loops, partial success, and cost explosions.

  2. 2
    Traces, metrics, logs, and context propagation

    A trace is strongest for the path of a single run, a metric for aggregate trends, and a log for describing individual events; shared context connects all three.

  3. 3
    Operating token cost and latency

    Cost optimization means reducing total cost and latency per successful task, rather than indiscriminately reducing token counts.

  4. 4
    Offline and online evals and release gates

    Evaluation is not a single average score but a combination of frozen cases and operational signals that represent real risks and usage distribution.

  5. 5
    Authorization · Audit · Incident response

    Auditing is not about storing conversations; it is about reconstructing who executed what, with which authority, why, and which state changed.

  6. 6
    Connect causality and retries across a queue

    Distinguish one request from multiple attempts to diagnose slow or duplicate work.

  7. 7
    Keep unfinished work visible in success-rate denominators

    Counting only finished jobs can exclude long-waiting failures; fix the starting cohort and judgment time.

  8. 8
    Reproduce incidents without reusing past write authority

    Separate inputs needed for reproduction from authority to cause real side effects, so investigation does not repeat the incident.

CONTROLLED EXPLANATION

Outcome accounting that preserves all 100 starts

Current state: Cohort: 100 starts

Outcome accounting that preserves all 100 starts

Source: an author-designed hypothetical comparison based on agent evaluation and OpenTelemetry signals. It shows omissions caused by counting completed responses alone.

Observed completionNot yet completeObservation time elapsesSeparate outcome and latencyInclude successful results1Cohort: 100 starts290 confirmed successes310 pending jobs4Recheck at observation time5Final outcomes and deadline violations
  1. Cohort: 100 starts

    Hold the task type and start period constant.

  2. 90 confirmed successes

    Completed results match business state.

  3. 10 pending jobs

    Keep them in the denominator.

  4. Recheck at observation time

    Read outcome and deadline separately.

  5. Final outcomes and deadline violations

    State counts must reconcile to 100 starts.

1 → 2
Observed completion
1 → 3
Not yet complete
3 → 4
Observation time elapses
4 → 5
Separate outcome and latency
2 → 5
Include successful results

Branches are distinct state sets. Pending is not counted as success, and late success does not erase a deadline violation.

CONCRETE CASES

Selection criteria for all courses

TABLE 9-1

Selection criteria for all courses

Compare the technology in each chapter by how it works, the new costs it adds, and the evidence to check, not by its name.

Table 9-1. Agent operations and observability: design decision criteria
.Core mechanismCosts to watchEvidence to check
1. New failure modes in production agentsRecord business outcomes and execution revisions under a common run identity to observe partial failures.High-cardinality data, personal data, and vendor-specific signal differences must be managed.Inject partial tool success, timeouts, and approval rejection, then verify that the final business states can be distinguished.
2. Traces, metrics, logs, and context propagationContext propagation connects distributed signals to the same task and causal path.The tradeoffs involve sampling, storage cost, cardinality, and exposure of sensitive data.Use a single task ID to connect the frontend request through the agent tool result and find missing spans.
3. Operating token cost and latencyCombine per-step telemetry with task outcomes to calculate cost per success and tail latency.Excessive savings cause missing context and lower quality; an unlimited budget can cause a cost incident.Compare cost per successful task, reflecting each model's task success, together with tail latency.
4. Offline and online evals and release gatesA versioned dataset and rubric repeatedly compare release candidates, and production results feed back as new eval cases.Dataset contamination and judge drift can occur, and averages can hide small risk slices.Calibrate the judge with known pass/fail cases and expert labels, and review disagreements.
5. Authorization · Audit · Incident responseStructured audit events and immutable revisions reconstruct the intent, authority, and outcome of an execution.It brings retention costs, privacy concerns, access control over the audit data itself, and the risk of false attribution.After revoking credentials and disabling the tool, verify that existing sessions can no longer execute.
6. Connect causality and retries across a queueSeparate business and attempt identities to aggregate retry cost and final success using different units.Correlation identifiers need retention and lifecycle management; raw sensitive context expands exposure.Inject termination after saving and before acknowledgment; record the link between one job, two attempts, and one result.
7. Keep unfinished work visible in success-rate denominatorsLinking starting cohorts to state transitions prevents unfinished work from silently disappearing.Final metrics arrive later, so provide provisional status alongside finalized results.Reconcile all 100 starts and specify when the 10 pending jobs are judged and how deadline violations are counted.
8. Reproduce incidents without reusing past write authorityTool doubles and restricted identities reproduce input and policy boundaries while blocking real external changes.Simulated results cannot reproduce every real connectivity failure; distinguish connectivity testing from semantic reproduction.Reproduce the invitation incident with a recording tool and verify rejection without approval and success with valid approval.

CHAPTER 1 / 8

New failure modes in production agents

Agent failures appear not only as HTTP errors but also as wrong tool selection, long loops, partial success, and cost explosions.

Why this concept became necessary

The model may respond while executing a tool on the wrong account or completing only two of three steps. Define success by business state and user confirmation, not response codes.

When the model, prompt, retrieval index, tools, and policy change independently, the same input can behave differently. Record a revision tuple for every run so regressions can be pinpointed.

Figure 9-2. New failure modes in production agents: decision flowThe chain from the problem conditions through the working principle to verification evidence.
Problems and selection criteria

Agent failures appear not only as HTTP errors but also as wrong tool selection, long loops, partial success, and cost explosions.

How it works

Record business outcomes and execution revisions under a common run identity to observe partial failures.

Verification evidence

Inject partial tool success, timeouts, and approval rejection, then verify that the final business states can be distinguished.

Follow it through a concrete system

A typical API request often either succeeds or ends in an explicit error, but an agent can stop after only some of its tool calls succeed. A ticket may be created while the user notification fails, or a refund may be processed while the model judges it a failure and requests it again. Without a task state machine and operation identity, a conversation transcript alone is not enough to recover actual business outcomes.

Model responses, retrieval, policy, and tool revisions change independently, so record this combination for every run. A report that the same prompt produced different results is not enough to narrow down the cause. The basic error contract of a production agent defines partial success, approval rejection, budget exhaustion, and human handoff as distinct terminal states and shows the user the next step.

Selection criteria and failure boundaries

High-cardinality data, personal data, and vendor-specific signal differences must be managed.

Misconceptions to avoid: A 200 from the model API does not mean the agent task succeeded.

Verify it yourself

Inject partial tool success, timeouts, and approval rejection, then verify that the final business states can be distinguished.

What to judgeRecord business outcomes and execution revisions under a common run identity to observe partial failures.

To summarize this chapter

Agent failures appear not only as HTTP errors but also as wrong tool selection, long loops, partial success, and cost explosions.

Official sources for this chapter

The technical facts in the text were reviewed against the following primary sources. The author reconstructed the diagrams and comparisons using these materials.

  1. Microsoft, 「Playwright Best Practices」Review date 2026-08-28 · Scope Latest official documentation
  2. OpenTelemetry, 「Signals」Review date 2026-08-28 · Scope Latest official documentation

CHAPTER 2 / 8

Traces, metrics, logs, and context propagation

A trace is strongest for the path of a single run, a metric for aggregate trends, and a log for describing individual events; shared context connects all three.

Why this concept became necessary

A trace span shows the parent-child timing relationships among model calls, retrieval, tools, and approvals. Metrics aggregate success rate, latency percentiles, and token cost by slice, and logs record the detailed rationale for policy decisions or errors.

Trace IDs and baggage must propagate across process and service boundaries, but must not indiscriminately contain original user text or secrets. Stable operation names and bounded attributes prevent cardinality explosions.

Figure 9-3. Traces, metrics, logs, and context propagation: decision flowThe chain from the problem conditions through the working principle to verification evidence.
Problems and selection criteria

A trace is strongest for the path of a single run, a metric for aggregate trends, and a log for describing individual events; shared context connects all three.

How it works

Context propagation connects distributed signals to the same task and causal path.

Verification evidence

Use a single task ID to connect the frontend request through the agent tool result and find missing spans.

Follow it through a concrete system

A trace shows the order and duration of model calls and tools within one run; metrics compare success rates and latency distributions across many runs; logs explain individual policy decisions in detail. These signals do not substitute for one another. A shared run ID and revision attributes let you drill down from an anomalous metric interval to representative traces and related logs.

Putting every prompt and tool output into attributes causes high cardinality and personal-data exposure at the same time. Keep bounded values such as the operation name, model, and policy version in telemetry, store large raw content separately in an access-controlled evidence store, and link it by hash. Design sampling policy around the questions you need to answer as well, for example by keeping errors and high-risk actions while sampling down high-volume normal requests.

Selection criteria and failure boundaries

The tradeoffs involve sampling, storage cost, cardinality, and exposure of sensitive data.

Misconceptions to avoid: A single kind of log cannot efficiently answer every trace and metric question.

Verify it yourself

Use a single task ID to connect the frontend request through the agent tool result and find missing spans.

What to judgeContext propagation connects distributed signals to the same task and causal path.

To summarize this chapter

A trace is strongest for the path of a single run, a metric for aggregate trends, and a log for describing individual events; shared context connects all three.

Official sources for this chapter

The technical facts in the text were reviewed against the following primary sources. The author reconstructed the diagrams and comparisons using these materials.

  1. OpenTelemetry, 「Signals」Review date 2026-08-28 · Scope Latest official documentation
  2. OpenTelemetry, 「Context Propagation」Review date 2026-08-28 · Scope Latest official documentation

CHAPTER 3 / 8

Operating token cost and latency

Cost optimization means reducing total cost and latency per successful task, rather than indiscriminately reducing token counts.

Why this concept became necessary

Prompts, retrieved context, tool output, and retries increase input tokens, while model choice and output length affect price and latency. Cache hits and parallel tool calls can help, but they can also introduce stale context and rate limits.

Looking only at P50 hides long agent loops. Examine P95/P99, step count, retry, queue, and token usage per successful task for each task slice, and design when to stop, degrade, or use human handoff if the budget is exceeded.

Figure 9-4. Operating token cost and latency: decision flowThe chain from the problem conditions through the working principle to verification evidence.
Problems and selection criteria

Cost optimization means reducing total cost and latency per successful task, rather than indiscriminately reducing token counts.

How it works

Combine per-step telemetry with task outcomes to calculate cost per success and tail latency.

Verification evidence

Compare cost per successful task, reflecting each model's task success, together with tail latency.

Follow it through a concrete system

An Agent's total latency includes retrieval, multiple tools, retries, and waits for approval, not just one model response. A model with low token prices may still cost more per successful task if it repeatedly chooses the wrong tools and succeeds less often. Calculate success rate, step count, input and output tokens, external API cost, and P95 time by task type from the same runs.

Consider optimization in this order: reduce unnecessary context and duplicate tool output, safely parallelize independent calls, then cache only repeated stable results. Include source revision and authorization in cache keys. If the budget is exceeded, do not invent an answer; follow the product contract to choose whether to degrade to a smaller model, omit some functionality, or hand off to a person.

Selection criteria and failure boundaries

Excessive savings cause missing context and lower quality; an unlimited budget can cause a cost incident.

Misconceptions to avoid: It is incorrect to assume that the cheapest model always has the lowest cost per success.

Verify it yourself

Compare cost per successful task, reflecting each model's task success, together with tail latency.

What to judgeCombine per-step telemetry with task outcomes to calculate cost per success and tail latency.

To summarize this chapter

Cost optimization means reducing total cost and latency per successful task, rather than indiscriminately reducing token counts.

Official sources for this chapter

The technical facts in the text were reviewed against the following primary sources. The author reconstructed the diagrams and comparisons using these materials.

  1. OpenTelemetry, 「Signals」Review date 2026-08-28 · Scope Latest official documentation

CHAPTER 4 / 8

Offline and online evals and release gates

Evaluation is not a single average score but a combination of frozen cases and operational signals that represent real risks and usage distribution.

Why this concept became necessary

Offline sets include normal, boundary, adversarial, permission-denial, and recovery cases, combining deterministic checks, rubrics, and human review. LLM judges are useful at scale, but their bias and self-preference must be calibrated.

Online, observe completion, correction, escalation, policy violations, and cohorts. Compare before and after changes using the same slices and confidence measures, and treat safety-critical failures as hard gates separate from the average score.

Figure 9-5. Offline and online evals and release gates: decision flowThe chain from the problem conditions through the working principle to verification evidence.
Problems and selection criteria

Evaluation is not a single average score but a combination of frozen cases and operational signals that represent real risks and usage distribution.

How it works

A versioned dataset and rubric repeatedly compare release candidates, and production results feed back as new eval cases.

Verification evidence

Calibrate the judge with known pass/fail cases and expert labels, and review disagreements.

Follow it through a concrete system

An Offline eval set must include boundary inputs, permission rejection, prompt injection, tool timeouts, and recovery, not just normal cases from the real user distribution. Each case should specify expected business state, citations, and disallowed actions, combining deterministic checks with a rubric. Freeze the Dataset revision so release candidates can be compared under identical conditions.

LLM judges can evaluate many answers quickly, but they are biased by stylistic preferences, model family, and rubric interpretation. Calibrate judges with pass and fail samples that experts agree on, and have people review disagreements. Online, watch corrections, escalations, and policy violations by slice, and treat safety-critical failures as a hard gate that blocks release even when the average score is high.

Selection criteria and failure boundaries

Dataset contamination and judge drift can occur, and averages can hide small risk slices.

Misconceptions to avoid: A single LLM judge score cannot serve as ground truth.

Verify it yourself

Calibrate the judge with known pass/fail cases and expert labels, and review disagreements.

What to judgeA versioned dataset and rubric repeatedly compare release candidates, and production results feed back as new eval cases.

To summarize this chapter

Evaluation is not a single average score but a combination of frozen cases and operational signals that represent real risks and usage distribution.

Official sources for this chapter

The technical facts in the text were reviewed against the following primary sources. The author reconstructed the diagrams and comparisons using these materials.

  1. Anthropic, 「Demystifying Evals for AI Agents」Review date 2026-08-28 · Scope Latest official documentation

CHAPTER 5 / 8

Authorization · Audit · Incident response

Auditing is not about storing conversations; it is about reconstructing who executed what, with which authority, why, and which state changed.

Why this concept became necessary

Record the policy decision, approval actor, tool revision, target pseudonym, input hash, and result in a tamper-evident store. Retain raw prompts, tokens, and personal information only to the minimum extent the purpose requires, and define access and deletion policies.

Incident response requires a kill switch that halts agent capabilities, credential revocation, a query for affected operations, and compensating actions. After recovery, rerun the same evals and policy tests and reflect the cause and guardrails in the runbook.

Figure 9-6. Authorization · Audit · Incident response: decision flowThe chain from the problem conditions through the working principle to verification evidence.
Problems and selection criteria

Auditing is not about storing conversations; it is about reconstructing who executed what, with which authority, why, and which state changed.

How it works

Structured audit events and immutable revisions reconstruct the intent, authority, and outcome of an execution.

Verification evidence

After revoking credentials and disabling the tool, verify that existing sessions can no longer execute.

Follow it through a concrete system

An audit event should structure the chain of responsibility for an execution rather than store the entire natural-language conversation. Linking the actor, tenant, tool and policy revisions, approval, de-identified target identity, idempotency key, and result makes it possible to reconstruct who changed which state with which permissions. Restrict permission to modify or delete the events themselves, and set retention periods to match the purpose of the data.

Incident response requires not only stopping model calls but also a kill switch to immediately block a tool or tenant capability, credential revocation, and identification of affected operations. Prepare compensating actions and user notifications because actions that already succeeded may not be reversible. After recovery, rerun the same abuse cases and representative tasks, and close the incident only after confirming that both functionality and safety boundaries have been restored.

Selection criteria and failure boundaries

It brings retention costs, privacy concerns, access control over the audit data itself, and the risk of false attribution.

Misconceptions to avoid: Auditing does not require permanently storing the full original prompt.

Verify it yourself

After revoking credentials and disabling the tool, verify that existing sessions can no longer execute.

What to judgeStructured audit events and immutable revisions reconstruct the intent, authority, and outcome of an execution.

To summarize this chapter

Auditing is not about storing conversations; it is about reconstructing who executed what, with which authority, why, and which state changed.

Official sources for this chapter

The technical facts in the text were reviewed against the following primary sources. The author reconstructed the diagrams and comparisons using these materials.

  1. OpenTelemetry, 「Signals」Review date 2026-08-28 · Scope Latest official documentation
  2. OpenTelemetry, 「Context Propagation」Review date 2026-08-28 · Scope Latest official documentation

CHAPTER 6 / 8

Connect causality and retries across a queue

Distinguish one request from multiple attempts to diagnose slow or duplicate work.

Why this concept became necessary

When an HTTP request enqueues work and returns early, the user’s wait outlasts the server response. Worker logs alone may not reveal when the request arrived. Link admission, queueing, execution, and delivery using one business identifier.

OpenTelemetry context propagation carries tracing context across services. Trace identifiers connect execution paths; they are not credentials proving user authority. Do not use externally supplied trace values to determine internal permissions or accounts.

Queue redelivery can create a second attempt for the same job. Separate business and attempt identifiers to explain repeated computation. Aggregate cost and retries across attempts, but determine user success from the business outcome.

Preserve correlation evidence even when long waits or process boundaries split traces. Carry only necessary correlation identifiers in queue messages, not copies of raw content and secrets. Observability convenience must not expand sensitive-data retention.

Figure 9-7. Connect causality and retries across a queue: decision flowThe chain from the problem conditions through the working principle to verification evidence.
Problems and selection criteria

Distinguish one request from multiple attempts to diagnose slow or duplicate work.

How it works

Separate business and attempt identities to aggregate retry cost and final success using different units.

Verification evidence

Inject termination after saving and before acknowledgment; record the link between one job, two attempts, and one result.

Follow it through a concrete system

A teaching summary job waits in a queue, calls a model and stores a file. If the first worker stops after saving but before acknowledgment, the message may be redelivered. Looking only at the second trace can make this appear to be a new user request. Comparing admission records and input hashes across attempts distinguishes redelivery from a new request.

Link the first and repeated attempts to one business identifier and check file finalization. If finalized, record the path that returns the existing result without another model call. This shows whether reduced retry cost genuinely reuses a result rather than concealing its loss. Check that an existing file is the finalized result for this job’s input, not merely that a file exists.

Reusing the same span identifier for every attempt is a counterexample. Distinct execution times become mixed, obscuring sequence and duplicates. Share correlation while recording each attempt’s start, end and failure as separate events. Read attempt errors alongside final state to distinguish successful recovery from first-attempt success.

Inject termination immediately after file finalization and trace redelivery. One job should show two attempts but only one user result. Separate queue wait from model-call time to identify which stage increased total latency. Include the first attempt’s model call in costs so resource use for the single user result is not omitted.

Selection criteria and failure boundaries

Correlation identifiers need retention and lifecycle management; raw sensitive context expands exposure.

Misconceptions to avoid: A shared Trace ID does not imply shared authority; tracing and authentication serve different roles.

Verify it yourself

Inject termination after saving and before acknowledgment; record the link between one job, two attempts, and one result.

What to judgeSeparate business and attempt identities to aggregate retry cost and final success using different units.

To summarize this chapter

Distinguish one request from multiple attempts to diagnose slow or duplicate work.

Official sources for this chapter

The technical facts in the text were reviewed against the following primary sources. The author reconstructed the diagrams and comparisons using these materials.

  1. OpenTelemetry, 「Context Propagation」Review date 2026-08-28 · Scope Latest official documentation
  2. OpenTelemetry, 「Signals」Review date 2026-08-28 · Scope Latest official documentation

CHAPTER 7 / 8

Keep unfinished work visible in success-rate denominators

Counting only finished jobs can exclude long-waiting failures; fix the starting cohort and judgment time.

Why this concept became necessary

If dashboard success rises while complaints increase, inspect the counting scope. A denominator containing only completed responses can omit timeouts and unfinished jobs. Without tracking started jobs, the apparent improvement cannot be trusted.

Store business outcomes as distinct states such as success, failure, cancellation and pending. Pending means not yet judged; it is not automatically success or failure. Jobs exceeding a promised deadline still need a separate latency-violation count.

Fix cohorts by admission time and task type. Revisit jobs started in the same period after sufficient observation time to include late outcomes. Distinguish provisional live metrics from finalized metrics and explain revisions.

Keep safety violations separate from the success-rate denominator. A useful result does not erase an unauthorized tool action. Separate mandatory conditions before considering any combined score for success, cost and violations.

Figure 9-8. Keep unfinished work visible in success-rate denominators: decision flowThe chain from the problem conditions through the working principle to verification evidence.
Problems and selection criteria

Counting only finished jobs can exclude long-waiting failures; fix the starting cohort and judgment time.

How it works

Linking starting cohorts to state transitions prevents unfinished work from silently disappearing.

Verification evidence

Reconcile all 100 starts and specify when the 10 pending jobs are judged and how deadline violations are counted.

Follow it through a concrete system

In a teaching comparison, the old candidate has 90 successes and 10 failures among 100 starts. The new candidate has 90 successes and 10 pending jobs; counting only the 90 finished jobs makes success appear to be 100%. Both have 90 immediately confirmed successes in the starting cohort. Hold any claim that the new candidate is better until pending outcomes and latency conditions are checked.

Recheck the 10 pending jobs at the defined observation time. If they exceeded the completion promise, retain the latency violation even if they later succeed. A result delivered late is not the same user experience as timely success. Separate eventual success from on-time success so late completion cannot erase a missed deadline.

The counterexample is a metric that counts every cancellation as a failure, treating the cancellation feature itself as poor performance. A task the user stopped voluntarily can differ from one the system drove to abandonment. If the evidence cannot distinguish the reasons, leave them unconfirmed rather than guessing a cause. Cancellation time and the waiting state just before it are diagnostic data that can be collected without guessing.

Verify that starts equal the sum of recorded states. Investigate missing identifiers as logging or aggregation gaps. Compare candidate success and cost only after using the same cohort definition and observation window. Record reasons and counts for exclusions so reviewers can see when the denominator changed.

Selection criteria and failure boundaries

Final metrics arrive later, so provide provisional status alongside finalized results.

Misconceptions to avoid: A high success rate among completed responses does not imply a high rate for all users; inspect pending work and cancellations.

Verify it yourself

Reconcile all 100 starts and specify when the 10 pending jobs are judged and how deadline violations are counted.

What to judgeLinking starting cohorts to state transitions prevents unfinished work from silently disappearing.

To summarize this chapter

Counting only finished jobs can exclude long-waiting failures; fix the starting cohort and judgment time.

Official sources for this chapter

The technical facts in the text were reviewed against the following primary sources. The author reconstructed the diagrams and comparisons using these materials.

  1. Anthropic, 「Demystifying Evals for AI Agents」Review date 2026-08-28 · Scope Latest official documentation
  2. OpenTelemetry, 「Signals」Review date 2026-08-28 · Scope Latest official documentation

CHAPTER 8 / 8

Reproduce incidents without reusing past write authority

Separate inputs needed for reproduction from authority to cause real side effects, so investigation does not repeat the incident.

Why this concept became necessary

Replaying historical inputs during investigation can repeat the same tool actions. Resending completed notifications or account changes can add harm before the cause is found. Start with evidence that external writes are blocked in the reproduction environment.

Reproduction needs the historical model, prompt, tool, and policy revisions. Running with current settings only is a different experiment from the past incident. Do not restore old credentials; recreate the necessary boundaries using read data and simulated tool results.

Separate reproduction goals. Recreating an incorrect proposal and verifying that policy blocks it require different observations. One run that happens to give a safe answer does not prove the execution boundary is fixed.

After a fix, check ordinary work as well as the known incident input. Blocking every tool can reduce incidents by disabling the product. Verify that permitted work succeeds and prohibited work ends without real state changes.

Figure 9-9. Reproduce incidents without reusing past write authority: decision flowThe chain from the problem conditions through the working principle to verification evidence.
Problems and selection criteria

Separate inputs needed for reproduction from authority to cause real side effects, so investigation does not repeat the incident.

How it works

Tool doubles and restricted identities reproduce input and policy boundaries while blocking real external changes.

Verification evidence

Reproduce the invitation incident with a recording tool and verify rejection without approval and success with valid approval.

Follow it through a concrete system

Assume a teaching scheduling assistant sent real invitations for a draft that was only meant for review. Replace the invitation tool with a recording double that does not send. Preserving proposed recipients and execution decisions reveals the bad boundary without real email. Verify that the recording double sends no network messages using execution evidence, not configuration alone.

The fix must return a draft for review requests and deny sending without approval. Record changes in the proposal separately from blocked execution. Better model wording and server-side prevention of state changes are different evidence. If the server blocks another bad proposal, the execution defense works independently of a model fix.

Claiming that a complete reproduction requires the old administrator token is a counterexample. Recreate relevant authorization decisions with restricted test identities. If a connectivity check needs real credentials, manage it separately from incident reproduction. Record test permissions and denial reasons to explain the reproduced authorization conditions without real secrets.

Record successful reproduction and zero real external changes as separate findings. If the new policy also blocks legitimately approved sending, recovery is incomplete. Verify permitted and prohibited paths before deciding to return to operation. Record permitted sends only against test targets, keeping both allowed and denied comparisons free of external effects.

Selection criteria and failure boundaries

Simulated results cannot reproduce every real connectivity failure; distinguish connectivity testing from semantic reproduction.

Misconceptions to avoid: Full incident reproduction does not inherently require production write authority; recreate only the boundaries needed for the cause.

Verify it yourself

Reproduce the invitation incident with a recording tool and verify rejection without approval and success with valid approval.

What to judgeTool doubles and restricted identities reproduce input and policy boundaries while blocking real external changes.

To summarize this chapter

Separate inputs needed for reproduction from authority to cause real side effects, so investigation does not repeat the incident.

Official sources for this chapter

The technical facts in the text were reviewed against the following primary sources. The author reconstructed the diagrams and comparisons using these materials.

  1. Anthropic, 「Demystifying Evals for AI Agents」Review date 2026-08-28 · Scope Latest official documentation
  2. Model Context Protocol, 「Authorization」Review date 2026-08-28 · Scope MCP 2025-11-25
  3. OpenTelemetry, 「Context Propagation」Review date 2026-08-28 · Scope Latest official documentation

INTERACTIVE LAB 1 / 2

Lab 1 · Assessing a release whose success rate improved but safety worsened

The new agent's average success rate increased by 4%, but executions without approval in the high-value refund slice rose from 0 to 3, and token usage per successful task doubled.

Choose the operating decision.

Choose an answer

Correct answer A

A. Stop the release because a safety hard gate failed, fix the high-value refund policy and approval trace, then re-verify using the same frozen slice and cost criteria. — A decision that accounts for conditions, working principles, and failure boundaries together.

B. Roll out fully because the average success rate increased. — It considers only some benefits and omits prerequisites or newly introduced failure boundaries.

C. Hides three violations inside the overall average. — It treats the responsibilities of different layers as one and misses the actual verification points.

D. Delete only the token metric. — It relies on technology names or trends, with no observable evidence from the current requirements.

INTERACTIVE LAB 2 / 2

Lab 2 · Determine whether omitted pending jobs changed the success rate

Both candidates start 100 jobs in a hypothetical comparison. The old candidate has 90 successes and 10 failures; the new one has 90 successes and 10 pending jobs. A dashboard counts only completed responses and reports 100% success for the new candidate.

Choose the required action before judging improvement and explain how pending state differs from a deadline violation.

Choose an answer

Correct answer A

A. Align starting cohorts, revisit the 10 pending jobs at the defined time and compare on-time success separately from eventual success. — This fixes omitted unfinished work and prevents late success from erasing missed deadlines.

B. Deploy the new candidate fully because all completed responses succeeded. — Without pending outcomes, this cannot establish the experience of all users or a safe improvement.

C. Delete the 10 pending jobs because they must have failed. — Pending is not a finalized outcome; deletion loses both the eventual result and latency evidence.

D. Add the new candidate’s pending jobs to the old candidate’s failures. — Mixing candidate cohorts destroys comparability and can even reverse the apparent direction of improvement.

KEY TERMS

Key terms in this unit

New failure modes in production agents
Record business outcomes and execution revisions under a common run identity to observe partial failures.
Traces, metrics, logs, and context propagation
Context propagation connects distributed signals to the same task and causal path.
Operating token cost and latency
Combine per-step telemetry with task outcomes to calculate cost per success and tail latency.
Offline and online evals and release gates
A versioned dataset and rubric repeatedly compare release candidates, and production results feed back as new eval cases.
Authorization · Audit · Incident response
Structured audit events and immutable revisions reconstruct the intent, authority, and outcome of an execution.
Connect causality and retries across a queue
Separate business and attempt identities to aggregate retry cost and final success using different units.
Keep unfinished work visible in success-rate denominators
Linking starting cohorts to state transitions prevents unfinished work from silently disappearing.
Reproduce incidents without reusing past write authority
Tool doubles and restricted identities reproduce input and policy boundaries while blocking real external changes.

UNIT WORKBOOK

Exercises and worksheets for applying concepts to new situations

Start by checking basic principles, then expand to practical workplace decisions. After submitting an answer, you can see why every option is correct or incorrect, not just the correct answer.

THREE-LEVEL ASSESSMENT

From basic principles to operational decisions

After you submit an answer, you can see not only the correct answer but also why each option is right or wrong.

Basic Question 1

Which question does a trace answer best?

Choose an answer

Correct answer A

A. The order and timing in which a task passed through service, model, and tool steps. — A decision that accounts for conditions, working principles, and failure boundaries together.

B. Only the overall success rate for the month. — It considers only some benefits and omits prerequisites or newly introduced failure boundaries.

C. The complete original text of all personal information. — It treats the responsibilities of different layers as one and misses the actual verification points.

D. The popularity of the programming language. — It relies on technology names or trends, with no observable evidence from the current requirements.

Apply Question 2

What is the right unit for comparing agent costs?

Choose an answer

Correct answer B

A. The number of log files. — It relies on technology names or trends, with no observable evidence from the current requirements.

B. Token and API cost per successful task, plus tail latency, for each task slice. — A decision that accounts for conditions, working principles, and failure boundaries together.

C. Only the input tokens for one request. — It considers only some benefits and omits prerequisites or newly introduced failure boundaries.

D. The shortest response length. — It treats the responsibilities of different layers as one and misses the actual verification points.

Capstone Question 3

How should the release verdict for a safety-critical eval be made?

Choose an answer

Correct answer C

A. Approve if a single judge passes. — It treats the responsibilities of different layers as one and misses the actual verification points.

B. Delete the failure case from the dataset. — It relies on technology names or trends, with no observable evidence from the current requirements.

C. Keep it as a hard gate separate from the average, and re-verify each revision on the same frozen cases. — A decision that accounts for conditions, working principles, and failure boundaries together.

D. Ignore it if the overall average is high. — It considers only some benefits and omits prerequisites or newly introduced failure boundaries.

PRIMARY SOURCES

Course references

This list collects the sources for each chapter. The text and author-created diagrams were prepared by directly reviewing the originals below.

OpenTelemetrySignalsReview date 2026-08-28 · Latest official documentation ↗OpenTelemetryContext PropagationReview date 2026-08-28 · Latest official documentation ↗AnthropicDemystifying Evals for AI AgentsReview date 2026-08-28 · Latest official documentation ↗MicrosoftPlaywright Best PracticesReview date 2026-08-28 · Latest official documentation ↗Model Context ProtocolAuthorizationReview date 2026-08-28 · MCP 2025-11-25 ↗

PERSONAL WORKSHEET

A learning worksheet you adapt to your own environment

Your input remains only on the current browser screen and is not stored or transmitted externally. Use categories and pseudonyms instead of actual sensitive information.

OFFICIAL SOURCES

Verify against official sources

Technical, compatibility, and model information reviewed: August 2026

LEARNING RECORD

Have you reviewed the text, decision activities, and all explanations?

Completion status is stored only in this browser.