WP-01

Design Document: Agentic AI in the Audit

An implementation framework for firm methodology, quality control, and engagement teams

Author
Cici Liu, CPA
Version
1.0
Date
July 2026
Status
Draft for comment
Regulatory baseline
PCAOB standards current as of July 2026; draft 2026-2030 Strategic Plan (Release No. 2026-006)

1Purpose and Scope

1.1Purpose

To specify how a firm should design, validate, deploy, and monitor agentic AI systems used in the conduct of financial statement audits, such that the resulting work satisfies PCAOB requirements and survives quality control inspection.

This is a design document, not a survey. It assumes the reader has decided to implement and needs to know how.

1.2Current state assumption

Agentic AI in audit is at pilot and early-rollout stage. Task-level agents supporting substantive procedures — expense vouching, searches for unrecorded liabilities and accrued expenses — are in production within at least one Big Four audit platform. Orchestration agents, which direct other agents and select which is suited to a given task, entered pilot in mid-2026 against asset valuation and revenue process testing.

The frequently cited figures of tens of thousands of deployed agents describe tax and consulting functions, not assurance. Audit is the slowest-adopting function in these firms. This document treats that as appropriate rather than remedial, and is written for the window before broad deployment rather than after it.

1.3In scope

1.4Out of scope

1.5Regulatory context

The PCAOB's draft 2026–2030 strategic plan sets six goals across three requests for comment issued March, June, and July 2026. Two bear directly on this document.

Goal 2, modernizing inspections, states the Board's intent to inspect firms' quality control systems, supplemented and reinforced by individual engagement reviews. This inverts the historical emphasis and is, as Jeffrey Johanns of UT Austin has argued in his reading of the three RFCs, the most disruptive of the six goals.

The design consequence is direct. Under engagement-focused inspection, agent governance is a workpaper matter. Under quality-control-focused inspection, it is a system matter: how the firm approves agents, monitors them across engagements, handles model version changes, and maintains an inventory. This document therefore places the majority of required controls at the firm level, not the engagement level.

Goal 5, modernizing oversight through technology and data, reflects what Johanns frames as two distinct problems: the company's use of technology in its accounting and internal control systems, and the auditor's use of technology to audit the company's use of technology. This document addresses the second.

Data and Technology survived the new Board's clearing of the prior standard-setting and research agenda and appears across all three RFCs. QC 1000 has been reopened for narrow revisions. Comments on the draft strategic plan are accepted through September 4, 2026.

2Design Principles

These are constitutional. Every specification below derives from one of them, and any proposed deviation should be resolved against them rather than against convenience.

P1 — An agent's output is a procedure, not evidence. If an agent extracts terms from 400 contracts, the contracts are the evidence and the extraction is a procedure. Its reliability must be established independently, as with any instrument relied upon. Agent output is a measurement to be validated, not a work product to be reviewed.

P2 — Autonomy is bounded at the procedure-design boundary. The auditor designs the response to assessed risk. The agent executes it. An agent may select among pre-approved paths; it may not invent a procedure, change scope, or determine that different evidence would be adequate.

P3 — Populations are defined deterministically. Population boundaries are established by query against the system of record and reconciled to control totals before any agent operates. Population definition is never a semantic or retrieval operation.

P4 — Every evidentiary procedure must be re-performable under a defined protocol. Re-performance is the mechanism by which review, engagement quality review, and inspection all function. A procedure whose re-performance protocol has not been specified is not deployable.

P5 — Skepticism is architected, not assumed. Language models are optimized toward helpfulness and agreement. This is the inverse of the disposition AS 1015 requires. Countermeasures are structural.

P6 — Accountability is named. Every agent has a single named individual accountable for its performance. "The firm" is not an answer, and neither is a committee.

P7 — Governance precedes deployment. The registry, approval gate, validation protocol, and trace infrastructure exist before the first engagement use, not after the pilot succeeds.

3Reference Architecture

Ten components. A firm may implement these differently, but each function must exist somewhere and be identifiable to an inspector.

Component Function Design requirement
Population service Extracts populations from systems of record; reconciles counts and control totals Deterministic. No model involvement. Produces a population ID referenced by all downstream steps.
Orchestration layer Sequences agents; enforces autonomy bounds Must log every path-selection decision. Must refuse actions outside the approved procedure envelope.
Task agents Execute defined procedure steps Scoped narrowly. One agent, one task class, one validation record.
Challenge agents Construct the case against the assertion Separate objective. No visibility into the producing agent's reasoning.
Verification service Confirms citations, source references, and quoted content against source Deterministic string and document matching. Never a second model.
Trace store Immutable execution record Write-once. Retained per documentation retention requirements. Schema in §6.
Agent registry Firm-level inventory of every audit-use agent Schema in §8.2. Single source of truth for approval status.
Validation harness Runs golden datasets and adversarial sets against agents Reproducible. Version-controlled datasets.
Monitoring service Drift detection, exception-rate monitoring, revalidation triggers Runs on a defined cadence independent of engagement timing.
Human checkpoint interface Presents output for review; captures reviewer action Must capture what the reviewer independently verified, not merely that they approved.

The verification service deserves emphasis. Using a model to check another model's citations reproduces correlated error rather than detecting it. Citation and source verification must be deterministic.

4Deployment Tiers and Gates

Implementation proceeds through tiers. Each gate must be evidenced before advancing. Gates are firm-level determinations, not engagement-level ones.

Tier 0 — Non-evidentiary support

Scope: Engagement administration. Prior-year workpaper comparison. Standards research with citation retrieval. Disclosure checklist completion against a filing. Draft assembly from human-reached conclusions.

Rationale: Nothing here produces evidence. Failure is visible and cheap. The real object of this tier is organizational: learning how reviewers behave when handed fluent output.

Gate to Tier 1: - Agent registry populated and current - Named accountable owner per agent - Trace capture operating and queryable - Seeded-error program running with documented reviewer catch rates (see §9) - At least two quarters of operation

Tier 1 — Evidence gathering within human-defined populations

Scope: Journal entry extraction and risk scoring. Contract, lease, and revenue term abstraction. Confirmation orchestration and exception tracking. ITGC evidence collection. Analytical procedure preparation.

Boundary: The agent gathers and structures. A human defines the population, designs the procedure, and concludes.

Gate to Tier 2: - Stratified validation complete per §5, all strata above threshold - Precision and recall reported separately by stratum - Documented re-performance protocol exercised successfully by a reviewer not involved in the original run - Trace reviewed by an experienced auditor with no prior connection to the engagement, who confirms they can follow the work performed, evidence obtained, and conclusions reached - Reviewer catch rate stable or improving

Tier 2 — Multi-step procedures with orchestration

Scope: Sequenced procedures where the agent identifies what requires testing, ingests source documentation, extracts data, generates the workpaper, and executes the test.

Boundary: The agent may select among pre-approved paths. Path selection must be logged and must remain within the procedure envelope the auditor designed.

Why this tier is harder than it appears: when an agent selects which sub-agent handles a step, procedure design has become partly the agent's. This is the point at which P2 is under real pressure and where AS 2301 concerns become live.

Gate to expanded Tier 2 scope: - Bounded variance demonstrated: procedure run n times (minimum 20) against identical inputs, with variance in conclusions — not phrasing — documented and within a pre-set tolerance - Escalation triggers verified to fire under test conditions designed to provoke them - Trace survives an engagement quality reviewer actively attempting to find a gap - Independent revalidation following any model version change

Tier 3 — Judgment

Scope: Sufficiency of evidence. Materiality application. Scope determination. Fraud risk conclusions. Going concern. Estimate reasonableness. Evaluation of identified misstatements.

There is no gate. The standards reserve these to the auditor. Agents may inform; they may not conclude.

Design obligation: the escalation architecture must make this boundary enforceable rather than aspirational. Under deadline pressure, aspirational boundaries move. Specify the technical mechanism by which an agent is prevented from producing a sufficiency or materiality conclusion, and test it.

5Validation Protocol

Vendor accuracy figures do not satisfy this requirement. The following is the minimum protocol before an agent operates at Tier 1 or above.

5.1Why stratification is mandatory

Statistical sampling produces error that is random with characterizable properties, which is what makes extrapolation defensible. Model error is correlated and semantic: an agent that misinterprets a clause construction misinterprets it consistently across the entire population.

This produces a conclusion that runs against a decade of full-population marketing and should be stated directly:

Testing 100% of a population with a biased instrument is not categorically safer than testing a sample with an unbiased one. Coverage substitutes for assurance only when the instrument's error structure has been characterized.

Reported pilot results — full analysis of revenue and expense line items against a typical 5–10% sample, surfacing duplicate-invoice schemes missed in prior cycles — are genuine gains and should be acknowledged as such. They do not remove the requirement to characterize error structure; they raise its importance, because a systematic error now propagates across the whole population rather than a sample of it.

Aggregate accuracy is therefore near-meaningless as an acceptance metric.

5.2Golden dataset construction

5.3Stratification dimensions

At minimum: transaction or instrument type; document form and template; jurisdiction and language; entity and business unit; period; and a dedicated edge-case stratum.

Acceptance requires every stratum to clear its threshold. Aggregate performance is not a substitute.

5.4Metrics

5.5Adversarial set

Constructed deliberately, not sampled: ambiguous contract language; unusual or non-standard structures; incomplete or partially illegible documents; descriptions worded to mislead; documents containing embedded instruction-like text (see §7); and near-miss cases just inside and just outside the target definition.

5.6Thresholds

Set before testing, tied to the risk of the assertion the procedure supports, approved by the accountable owner and firm methodology. A threshold set after results are seen is not a threshold.

6Trace Schema

AS 1215 requires that an experienced auditor with no previous connection to the engagement can understand the work performed, the evidence obtained, and the conclusions reached. A prompt and a response do not meet this.

The following fields constitute the minimum trace for an agent-executed procedure. Retrofitting this is expensive and usually incomplete; build it before Tier 0.

6.1Execution identity

6.2Inputs

6.3Execution record

6.4Verification

6.5Human interaction

6.6Outcome

Field 6.6's third element matters disproportionately. An agent that never reports insufficient information is not being careful; it is being fluent.

7Threat Model

Threat Mechanism Control
Hallucinated citations and authorities Model generates plausible references to standards, clauses, or documents that do not exist or do not say what is claimed Deterministic verification service (§3). Never model-checks-model. Failed verifications block completion.
Prompt injection via client-supplied evidence A document within the evidence population contains text that functions as instructions to the processing agent Treat all client-supplied content as untrusted. Separate instruction and data channels architecturally. Include injection cases in the adversarial validation set. Log any anomalous instruction-like content encountered.
Silent population incompleteness Retrieval recall limits cause items to be omitted without signal P3: deterministic population definition and reconciliation before agent execution
Correlated extraction error Consistent misinterpretation of a document form or clause construction across the population Stratified validation (§5.3); edge-case stratum; periodic champion/challenger against full human performance
Context truncation Long documents silently truncated; agent analyzes what it received Explicit truncation logging as a blocking event; document length pre-checks
Tool-call error Agent queries wrong entity, period, or ledger and returns confident, well-formatted, wrong output Parameter validation at the orchestration layer; reconciliation of agent-retrieved totals to independently extracted control totals
Compounding multi-step error Early misreading propagates through all downstream steps Intermediate output checkpoints at Tier 2; independent verification of step outputs before dependent steps execute
Agreement bias Model optimized toward helpfulness leans toward supporting management's position §10 architected skepticism
Model version drift Vendor updates model; behavior changes without notice Contractual version notice and pinning; version change is a mandatory revalidation trigger (§8.3)
Automation complacency Reviewers review fluent output less rigorously over time §9 reviewer instrumentation — the single most likely cause of an actual audit failure

8Firm-Level Control Framework

Per §1.5, the majority of required controls sit at firm level. These map to the firm's system of quality control and should be documented within it rather than in engagement files.

8.1Control domains

Domain Control objective Owner
Governance Every audit-use agent is inventoried, risk-rated, and approved through a defined gate before deployment. A single named individual is accountable for each. Firm QC
Design Autonomy bounds are specified and technically enforced. Procedure design, materiality, scope, sufficiency, and conclusions remain with the auditor. Firm methodology
Validation Documented stratified validation against human-adjudicated ground truth, with thresholds set ex ante. Firm QC and methodology
Execution Populations defined deterministically and reconciled. Human checkpoints enforced. Configuration pinned and logged. Engagement team
Documentation Trace per §6, sufficient for an experienced auditor with no prior connection. Engagement team
Monitoring Drift detection, revalidation on trigger events, exception-rate monitoring, periodic champion/challenger. Firm QC
Human factors Reviewer catch rates measured and monitored. Firm QC and L&D

Four of seven domains are firm-level. A firm with strong engagement documentation and nothing above it has prepared for the inspection regime the Board is moving away from.

8.2Agent registry — required fields

Agent ID and version · task class · tier classification · accountable named owner · approval date and approver · underlying model and version · validation record reference and date · performance thresholds by stratum · last revalidation date · next scheduled revalidation · engagements deployed on · known limitations · escalation triggers configured · deprecation status.

8.3Revalidation triggers

Revalidation is mandatory upon any of: underlying model version change; prompt template material revision; retrieval index rebuild; orchestration policy change; expansion of task scope; drift detection threshold breach; adverse inspection or internal monitoring finding; annual cycle, whichever is soonest.

9Human Factors and Reviewer Instrumentation

The failure mode most likely to cause an actual audit failure is not technical.

Reviewers presented with fluent, well-formatted, confident output review it less carefully than they review a junior's rough work. This is a documented human tendency, it worsens with exposure, and no amount of model validation addresses it.

Required program:

There is a related structural issue this document flags without solving. The judgment that senior auditors apply was historically acquired as a byproduct of high-volume procedural work. If agents absorb the volume, judgment has to be developed deliberately rather than incidentally. That is a talent architecture problem, and it should be owned by someone.

10Architected Skepticism

AS 1015 requires professional skepticism. Language models are optimized toward helpfulness and agreement. The default disposition of the instrument is the inverse of the disposition the standard requires. This is a design problem, not a training problem.

Required patterns:

  1. Evidence-first prompting. Withhold management's conclusion from the agent. Ask what the evidence supports on its own terms. Compare afterward.
  2. Adversarial framing. Require the agent to construct the strongest available case against the assertion before evaluating it.
  3. Independent challenge agents. Different objective function, no visibility into the producing agent's reasoning or output rationale.
  4. Mandatory contradictory-evidence search. AS 1105 defines audit evidence to include information that contradicts management's assertions. Make this an explicit, logged procedure step rather than emergent behavior.
  5. Uncertainty surfacing. Require explicit reporting of what could not be determined, with the absence of such reporting treated as a signal rather than a clean result.
  6. No conclusion-shaped output at Tier 1. Agents at Tier 1 produce findings and exceptions, not conclusions. The grammatical form of the output should make the boundary visible.

11Open Regulatory Dependencies

The following are unresolved in current standards. Each is stated with the interim design posture this document recommends pending guidance, and each is a candidate for comment before September 4, 2026 (PCAOB Release No. 2026-006).

11.1 Does re-performance require determinism? If an agent can reach a defensible conclusion by more than one path, what does re-performance mean and what variance is acceptable? Interim posture: pin configuration for all evidentiary procedures; document bounded variance where pinning is impracticable.

11.2 What does AS 1215 require of an agent trace? Full reasoning capture, or a summary of the procedure and its basis? The cost difference is substantial. Interim posture: capture per §6 and reduce later if guidance permits. Retrofitting is not feasible.

11.3 Is an agent a tool or a specialist under AS 1210? This determines the entire evaluation and documentation architecture. Interim posture: apply the specialist analogue — assess competence via validated task-class performance, assess objectivity via design review, evaluate the work. It is the better fit and is not currently sanctioned.

11.4 What validation supports full-population coverage using model-based extraction? Given correlated rather than random error, does the profession need a stratified-validation requirement analogous to sampling standards? Interim posture: §5 in full.

11.5 Where in QC 1000 does agent governance sit? The standard has been reopened for narrow revisions, which presents an opportunity to state this explicitly. Interim posture: document under the firm's existing QC components and map explicitly.

11.6 Should "agentic" be a defined term? Distinguishing systems that select their own procedure path from those executing a designed one would carry real regulatory weight. Interim posture: adopt the tier taxonomy in §4 internally and use it consistently in firm documentation.

11.7 What is the auditor's responsibility for the client's own agents? Where client agents are embedded in transaction processing or generate accounting estimates. This is Johanns' first technology problem, and it is arguably moving faster than the second. Interim posture: extend AS 2110 risk assessment procedures to inventory client AI systems affecting financial reporting; treat model-generated estimates as requiring specialist-equivalent evaluation.

11.8 What is a foundation model provider? Service organization, specialist, tool vendor, or a category the standards do not yet have. Interim posture: obtain version-change notice contractually; document provenance and data boundaries as an independence matter as well as an IT matter.

12Pre-Deployment Checklist

No agent operates at Tier 1 or above until every item is evidenced.

Three questions the accountable owner must be able to answer without preparation:

  1. Who is accountable by name if this agent produces a material error?
  2. What is this agent categorically not permitted to decide?
  3. If an inspector asked us to re-perform this procedure tomorrow, what exactly would we hand them?

If these lack crisp answers, the implementation is not ready regardless of pilot results.

13Closing Position

Audit is early. Not behind — early. The pilots are running now, the standards are being drafted now, and both remain changeable. That alignment is rare and it will not last.

Every specification in this document reduces to a single obligation that predates all of it. The auditor obtains sufficient appropriate evidence and reaches a conclusion. Nothing about an agent changes who signs.

It changes only how far the signer stands from the work — and whether that distance was engineered, validated, and documented, or quietly assumed.

Appendix A — Source Material

All standards references require verification against current PCAOB text prior to circulation. QC 1000 component mapping in §8 should be conformed to the standard as revised.