HUX AI · Research Internship · Project 2 · Summer 2026

Making Autonomy Governable: A Risk Control Matrix for Tool-Using AI Agents

A governance instrument that separates what an agent is permitted to do from what happens when it is wrong, and states the control posture, evidence and rollback owed at each intersection.

Research team: Aslınur Alkan
Benjamin Hong
Beste Özyurt Ersöz
Bhoomika Hanbal Puttaswamy
Frances Chinaza Agba
Kevin Braza
Peri Yakar
Shalom Chimkwesiri Okeke
Mentors: Ece Sülüngür
Hayriye Anıl
Sami Tuğal
Merve Ayyüce Kızrak
Published: September 2026

Executive summary

In November 2022 an airline's website chatbot told a grieving passenger they could claim a bereavement fare after travelling. The information was wrong, and a tribunal later required the airline to compensate the passenger. That chatbot could not reach a single external system. Responsibility did not begin when the system was allowed to act — it was already there.

Agentic systems call tools, write to external systems and take actions that were never individually approved. The governance question moves from what a model might say to what an agent can actually do. Published guidance names agentic risk and tells organisations to assess it; none of it says how much control a given deployment needs. This project fills that gap.

A governance instrument, not an assurance certificate. The scoring rules were defined inside the project and validated internally across eight scenarios, not by an independent assessor. Version one addresses single-agent deployments only.

Four linked assets

01Risk taxonomyThree families, nine domains, thirty-one observable failure modes, indexed to the OWASP Top 10 for Agentic Applications.
02Autonomy modelSix levels with a seven-question classification tree that assigns a use case to a level reproducibly.
03Scenario librarySixteen use cases spanning every level, four annotated in full.
04Control matrixA posture at each of the twenty-four intersections, with the evidence required at each autonomy level.

Two findings

Autonomy and impact must be assessed separately. An advisory system that cannot act may still influence criminal sentencing; an autonomous system may be confined to read-only access. One combined score rates both wrongly.

Controls do not erase inherent risk. Coverage is reported separately from exposure, so a deployment can hold 87% of its expected controls and still need redesign.

The control matrix

Select any cell for its control set. Rows are impact, columns are autonomy. Postures are the reconciled values at Table C1; tiers are their numeric form at Table E1.
Impact ↓Autonomy →	L0Respond onlyNo action capability	L1Advise onlyHuman decides and acts	L2Prepare actionHuman executes	L3Approved executionPer-action approval	L4Bounded autonomyNo per-action approval	L5Open autonomyHigh-impact, unapproved
I0NegligibleNo risk, or easily reversible
I1LimitedReversible, small blast radius
I2SignificantReversible, large blast radius
I3CriticalIrreversible

L0 × I3

Restricted, safety

Respond only — no action capability. Oversight: not applicable. Impact: critical, irreversible.

Tier 3

Safeguarding, crisis or health information; vulnerable users.

Required controls

Impact assessment before launch
Professional-authored safety content
Deterministic crisis routing
Long-session / adversarial testing
Accountable owner, 24/7 suspension
Independent evaluation
Pre-tested fallback to human channel

Approval & logging

Named professional owner with 24/7 suspension authority; two-person sign-off on safety-path changes.

Rollback

Pre-tested fallback to static resources and a staffed human channel. Rollback protects the next user, not the one already harmed.

EvidenceRaine / Garcia Magesh 2025 NIST AI 600-1

Reading the colours

Tier 1 — PermittedBaseline controlsStandard safeguards, monitoring and a named owner.
Tier 2 — SupervisedSupervised or conditional deploymentAdditional supervision, validation and limits.
Tier 3 — RestrictedRestricted deploymentFormal assurance, approval or verified sign-off before deployment.
Tier 4 — Unacceptable as configuredDo not deploy as assessedRedesign, reduce autonomy or impact, or change the operating envelope, then reassess.

Colour means risk. Nothing else on this page is coloured. Posture terms express a recommended governance posture, not a determination of legal permissibility.

How to use it

Step 1Define the agent's intended action and objective. What is it for, and what is it allowed to touch?
Step 2Assess the autonomy level with the classification tree — not what the technology could do, what this deployment is permitted to do.
Step 3Assess the impact class independently of autonomy. Ask what happens when it is wrong, not when it is right.
Step 4Apply the control posture at the intersection before deployment, and record the evidence that posture requires.

The posture is a floor, not a ceiling. An organisation adopting only one part of this should adopt the separation of autonomy from impact — it is the step that changes conclusions rather than presentation.

Classify a use case

Seven yes-or-no questions assign the autonomy level. An eighth question assesses impact separately and never changes the level. Answer them for a system you have in mind.

Question 1 of 7

Does the agent only generate information or content, without making recommendations or taking any action?

Your answers

Q4 is a gate rather than a level: an agent that can neither prepare nor execute an action has been misread earlier in the tree, and the assessor is returned to the start. Q6 and Q7 are two routes to the same place — an agent outside predefined bounds is at L5 whether or not it sets its own objective.

The two scales

Autonomy levels — what the system is permitted to do

L0Respond onlyNo action capability

Respond only: no action capability.

Oversight: Not applicable

L1Advise onlyHuman decides and acts

Advise only: human decides and acts independently.

Oversight: Human decides

L2Prepare actionHuman executes

Prepare action, human executes: agent drafts; human signs, sends or files.

Oversight: Human in the loop

L3Approved executionPer-action approval

Execute low-risk actions with approval: agent acts only after approval of that specific action.

Oversight: Human in the loop

L4Bounded autonomyNo per-action approval

Execute bounded workflows autonomously: no per-action approval, within reversible, capped or allowlisted limits.

Oversight: Human on the loop

L5Open autonomyHigh-impact, unapproved

Execute high-impact actions autonomously: no approval; action may be irreversible or high-stakes.

Oversight: Human in command

L0 through L5. Assess what this deployment is permitted to do, not what the technology could do.

Impact classes — what happens when it is wrong

I0NegligibleNo risk, or easily reversible

Incorrect information produced before anyone acts on it.

I1LimitedReversible, small blast radius

Effect stays within the affected system.

I2SignificantReversible, large blast radius

Correction possible, but the effect reached the user.

I3CriticalIrreversible

Financial or judicial consequences such as refunds, sentencing or filings relied upon.

I0 through I3. Impact is a property of the deployment, not the technology.

Risk taxonomy — what can fail

AIntent & Authority

A.1Objective & instruction integrityR1 · Model-output · ASI01 · 5 failure modes

A.1.1Goal misinterpretation
A.1.2Goal hijacking
A.1.3Conflicting instructions
A.1.4Objective drift
A.1.5Wrong task prioritisation

If it fails

The agent replies to customers with inappropriate or conflicting policy information.

Typical triggers

Ambiguous instructions. Prompt injection. Malicious external content.

Control themes

Goal validation, intent confirmation, prompt-injection controls, stop conditions.

A.2Autonomy, authority & permissionR2 · Governance · ASI03 · 3 failure modes

A.2.1Unauthorised action
A.2.2Approval bypass
A.2.3Privilege escalation

If it fails

The agent approves unverified high-value transactions beyond its intended authority limit.

Typical triggers

Misconfigured permissions. Excessive credentials. Delegated authority.

Control themes

Least privilege, role-based access, action allowlists, approval gates.

BExecution & Interaction

B.1Tool & action executionR3 · Action-execution · ASI02 · 4 failure modes

B.1.1Wrong tool selected
B.1.2Incorrect API call
B.1.3Destructive operation
B.1.4Duplicate transaction

If it fails

The agent executes a DELETE command instead of READ on a production database.

Typical triggers

Model error. Tool ambiguity. Poor validation. API changes.

Control themes

Tool allowlisting, input validation, sandboxing, rollback, limits.

B.2Data, memory & information boundariesR4 · Action-execution · ASI06 · 3 failure modes

B.2.1Data exfiltration
B.2.2Unauthorised disclosure
B.2.3Memory poisoning

If it fails

The agent emails a consolidated report containing external personal data.

Typical triggers

Malicious input. Weak access control. Untrusted memory.

Control themes

Data minimisation, loss prevention, encryption, retention limits, memory governance.

B.3Planning, workflow & delegationR5 · Model-output · ASI08 · 4 failure modes

B.3.1Sequencing failure
B.3.2Infinite loop
B.3.3Premature execution
B.3.4Bad sub-task delegation

If it fails

The agent gets stuck in an infinite loop, consuming API credits and blocking processes.

Typical triggers

Planning error. Missing dependencies. Conflicting outputs.

Control themes

Step validation, bounded workflows, execution limits, checkpoints.

B.4Agent-to-agent & ecosystem interactionR6 · Action-execution · ASI04 / ASI07 · 3 failure modes

B.4.1Cascading failure
B.4.2Agent spoofing
B.4.3Insecure inter-agent messaging

If it fails

A compromised third-party API allows an attacker to control internal agents.

Typical triggers

Compromised third party. Weak agent identity. Unverified messages.

Control themes

Identity verification, secure messaging, isolation, inter-agent logging.

CGovernance & Assurance

C.1Human control, accountability & trustR7 · Governance · ASI09 · 3 failure modes

C.1.1Human approval bypass
C.1.2Automation bias
C.1.3Rubber-stamping

If it fails

Human operators rubber-stamp a batch of flawed refunds.

Typical triggers

Poor human-in-the-loop design. Alert fatigue. Ambiguous accountability.

Control themes

Human-in-the-loop, dual approval, human override, clear ownership.

C.2Observability, traceability & auditabilityR8 · Governance · no clean fit · 3 failure modes

C.2.1Missing logs
C.2.2Incomplete action history
C.2.3Unverifiable approvals

If it fails

Auditors cannot determine why the agent granted a specific loan application.

Typical triggers

Logging gaps. Poor telemetry. Distributed systems.

Control themes

Immutable logs, trace IDs, monitoring, audit trails, provenance capture.

C.3Behavioural stability & runtime driftR9 · Model-output · ASI10 · 3 failure modes

C.3.1Behavioural drift
C.3.2Increased risk-taking
C.3.3New tool behaviour

If it fails

After a silent model update, the agent begins prioritising speed over accuracy.

Typical triggers

Model update. Environmental change. Memory accumulation.

Control themes

Runtime monitoring, behavioural baselines, kill switch, revalidation.

Child risks are the unit of assessment: a control cites a child ID such as B.2.1, never a family or domain alone. C.2 has no clean OWASP counterpart. The operational register expands the same nine domains to fifty-four modes at the same grain; no new level of hierarchy was introduced.

Six amplification factors

An amplifier raises the likelihood, scale, speed, severity or detection difficulty of a risk without being a failure itself. Amplifiers are held outside the taxonomy and escalate a rating rather than feeding a probability.

Irreversible actionsActions that cannot be undone, such as deleting data or transferring funds.
External impactsActions directly affecting customers, external systems or the public.
Sensitive dataAccess to personal data, health records, financial data or intellectual property.
Financial impactHigh transaction limits, or access to corporate payment systems.
Vulnerable usersInteraction with minors, elderly users, or users in crisis.
Execution velocityThe agent's ability to repeat an error thousands of times in seconds.

Scenario library

Sixteen use cases span every autonomy level, drawn from criminal justice, GMP biologics manufacturing, retail, banking, security operations, construction, education, child safeguarding, software engineering and internal knowledge work. Eight were run through the full instrument; the cards published below are the annotated and tested ones.

Scenarios per autonomy level: L0 2, L1 2, L2 3, L3 3, L4 4, L5 2.

SC-01L1 × I3Strict approval

Judiciary advisory risk scoring

Generate a risk score from defendant history, summarise case law, flag inconsistent rulings and draft a research memo.

Domain: Judiciary and criminal justice: sentencing and pretrial. Users are judges, magistrates, clerks and registrars.
Data accessed: Defendant criminal history, risk-model inputs, case law, dockets and sentencing guidelines — all read-only.
Why I3: The output is information only, but the class is assessed on the decision it feeds: once a judge relies on the score for an outcome already served, the effect is irreversible.
Human control: Formally required, weak as a real-time check: nothing catches anchoring in the moment, only on appeal.
Key risk: Automation bias; proprietary methodology hard for the defence to challenge; disputed disparity in error rates.
Required controls: Written disclosure of the tool's limits in every presentence report; the score may not be sole or determinative; local validation; right to contest scored factors.
Citations: Loomis (2016); Angwin et al. (2016); Dieterich et al. (2016); Courts and Tribunals Judiciary (2025); EU AI Act Annex III 8(a), Art. 6(3).

SC-05L4 × I0Restricted

Internal read-only research assistant

Retrieve, read and summarise internal and external sources, and draft notes into the user's own workspace.

Domain: Internal knowledge work. Users are staff carrying out desk research.
Data accessed: Internal knowledge sources and public material — all read-only.
Why I0: Read-only on every source; writes only to the user's own workspace, so nothing leaves it to be reversed.
Human control: Post-hoc and periodic rather than per-action, which the impact class permits.
Key risk: Scope creep from read-only research into acting on what it finds.
Required controls: Read-only permissioning enforced technically rather than by policy; no external posting; periodic spot-check against source.
Assessor note: The tool's grid returns 2.0 at L4 by I0. The assessor recorded low, on the grounds that the agent has no external reach and nothing to reverse.

GMPL2 × I3Mandatory verified sign-off

Regulated documentation drafting, GMP biologics

Draft regulated manufacturing documentation for human sign-off; the agent cannot file anything itself.

Resolved posture: Mandatory verified sign-off. One draft of the matrix prohibited agent-drafted regulated documentation outright; the reconciled cell requires verified sign-off instead.
Validation change: This case changed the instrument: human review and substantive source verification had been treated as equivalent, which in a regulated documentation setting they are not. The control text now requires verified sign-off and names rubber-stamping as the failure it prevents.
Citations: 21 C.F.R. pt. 11; Mata v. Avianca (2023); FDA draft guidance on AI in regulatory decision-making (2025).

FLEETL5 × I3Prohibited

Autonomous fleet remediation

Diagnose and remediate faults across a fleet autonomously, with production access and no per-action approval.

Base rating: Tier 4 at the outset: high-impact autonomous execution combined with critical impact.
Amplifiers: The amplifier score would escalate the rating further; the scoring model is capped at tier 4.
Control coverage: Thirteen of fifteen control points — 87%, which the tool bands as strong.
Status: Unacceptable as configured. Strong coverage does not clear a tier-4 exposure.
Required changes: Reduce the agent's autonomy; require human approval for critical changes; introduce a canary rollout that stops on failure; test rollback and emergency isolation before wider use.

The risk level on each card is the assessor's qualitative judgement at the time of writing and may differ from the rating the tool computes. Where the two differ, the tool's rating is the floor and the card records the assessor's view.

Run the assessment

The instrument takes a base rating from the matrix, escalates it by an amplifier score, caps it at the top tier, and reports control coverage separately. Coverage never lowers the inherent tier. These are project calibration rules, not derived standards.

Autonomy level

Impact class

Amplifiers — weights from Table E2

Action type

Reversibility

Exposure flags

Control points — Table E3, 15 available

Human approval 2 / 3

Permission model 2 / 3

Action limits 2 / 3

Monitoring 2 / 3

Rollback 1 / 2

Kill switch 1 / 1

L3 × I2Strict approval

Base rating, Table E1: tier 3
Amplifier score: 4
Escalation: none
Inherent tier (cap 4): tier 3
Control points: 10 / 15
Indicative coverage: 67% — partial
Evidence readiness: Design evidence only

Status

Control gap: 10 of a required 11 control points at this tier.

Evidence readiness, reported separately

Amplifiers can make a posture stricter; they cannot make it weaker, and existing controls never reduce the inherent tier. Evidence readiness is reported separately and is never folded into the coverage figure.

Evidence an auditor will ask for

1Evidence of risks

Risk assessment, impact, classification assumptions, affected systems and stops.

2Evidence of control design

Policies, guardrails, workflow boundaries, permissions, limits and monitoring.

3Evidence of control effectiveness

Test results, monitoring evidence, exception handling and control failure tests.

4Evidence of approval & oversight

Human-in-the-loop configuration, approver authority, approval logs and exception approvals.

5Evidence of resilience & recovery

Rollback tests, failover evidence, incident response and recovery ownership.

Five categories, applied at every cell.

Method and findings

How it was built

Each column of the matrix was researched independently against a shared reference scale, then reconciled at the end of the project. The taxonomy was derived from the agent's own operating workflow and the OWASP mapping applied after the structure was settled. Sixteen scenarios were written to test the ladder across domains.

Eight scenarios were run through the full instrument against the team's own judgement. Seven produced the expected result. The eighth — a GMP biologics case — had treated human review and substantive source verification as equivalent, which in a regulated documentation setting they are not. The required-controls text now demands verified sign-off and names rubber-stamping as the failure the control prevents.

Why amplifiers sit outside the base rating

A missing approval step does not mean the same thing everywhere. In a summarisation tool it produces an unwanted output; in an ordering API it produces transactions. Execution speed changes the scale of a failure: a person applying a policy wrongly affects the cases they handle, while an agent can apply the same error across a queue before anyone notices.

Supervision is a capability, not a checkbox

A human approval step counts as a control only where the reviewer has enough visibility into the system and enough authority to challenge the output and stop the action. An approval field a reviewer cannot exercise is a recorded step, not a control.

Impact is a property of the deployment

The same retrieval assistant is negligible-impact on a canteen menu and critical-impact on a safeguarding pathway. Classify the use case, not the technology.

Where the drafts disagreed

Sixteen of the twenty-four intersections agreed across every version. The eight below did not. Disagreement was reconstructed at the end by comparing the surviving versions, and is published rather than smoothed over.

Cell	Column-authored	Review copy	Resolved	Basis
I0 × L2	Permitted, light-touch verification	Supervised	Supervised	Unevidenced. Judgement.
I0 × L3	Auto-approved / supervised	Supervised	Supervised	Unevidenced. Judgement.
I0 × L4	Restricted	Restricted	Restricted	Unevidenced. Judgement. The tool sheet had supervised / conditional.
I1 × L0	Supervised	Permitted	Supervised	Evidenced. Moffatt v. Air Canada (2024): an organisation owns what its information-only chatbot says.
I1 × L3	Restricted	Restricted	Restricted	Unevidenced. Judgement. The tool sheet had supervised.
I2 × L0	Restricted	Supervised	Restricted	Evidenced. The NYC MyCity incident (Lecher, 2024): an information-only public-sector bot giving unlawful guidance at scale, with disclaimers, left online after publication.
I3 × L2	Draft-only, mandatory verified sign-off	Prohibited / redesign	Mandatory verified sign-off	Evidenced. The permitting cell cites 21 C.F.R. pt. 11, Mata v. Avianca and draft FDA guidance; the prohibiting cell recorded no scenario, controls or citation.
I3 × L3	Avoid if possible / strict oversight	Restricted autonomy / dual approval	Restricted / dual approval	Labels differ, substance agrees.

Where no version recorded a basis, the resolved value is the one consistent with the surrounding row; where the drafts still differed, the stricter reading was taken. One inconsistency was corrected in review: at significant impact, advice-only moved from supervised to restricted so that increasing autonomy or impact never weakens the control requirement.

Limitations

Properties of the method, stated so a reader can judge how far the instrument's outputs can be relied on.

Literature selection: Sources were selected for standing with the intended audience, not by systematic search. The evidence base is a purposive sample, not a systematic review.
Control matrix: Columns were assigned independently and reconciled at the end of the project rather than continuously, and documentation depth still varies across the six columns.
Calibration constants: The base grid, amplifier weights, escalation thresholds and coverage bands are project calibration rules. No sensitivity analysis was performed and they derive from no external standard.
Internal validation only: The instrument was calibrated against the team's own judgement across eight scenarios, and has not been applied by an external assessor or used in a live engagement.
Scenario domain coverage: Scenarios were sampled for coverage across autonomy levels, not risk domains; several domains carry no scenario. Coverage claims concern the ladder, not the taxonomy.
Evidence checklist: Twenty-two of the thirty evidence entries are written; the highest-autonomy column and part of one other are not.
Single-agent scope: Version one addresses single-agent deployments. Inter-agent communication and cascading failure are named in the taxonomy but not exercised.
Crosswalk not delivered: Mapping to ISO/IEC 42001 Annex A control identifiers was in scope at the outset and was not completed; it remains future work.
Judgement is recorded, not removed: Ratings involve judgement. Disagreement was reconstructed at the end by comparing the surviving versions, and is published rather than smoothed over.

Conclusion and next steps

Agentic AI moves the governance question from what a model says to what an agent is permitted to do. Present frameworks name that risk without saying what control it requires. The contribution here is the assessment method: a taxonomy of observable failure modes, an autonomy scale a use case can be assigned to reproducibly, a scenario library that tests the method against real deployments, and a matrix that states a posture at every intersection and the evidence owed at every level.

Three extensions are proposed: a crosswalk to ISO/IEC 42001 Annex A control identifiers, multi-agent coverage for the inter-agent and cascading-failure risks the taxonomy names but no scenario exercises, and a live engagement with use cases the team did not write. A fourth follows from the treatment of supervision — system transparency should become an assessment dimension in its own right, because an approval step cannot function as a control where the reviewer cannot see the reasoning, the tool use or the data behind a decision.

AI use disclosure

Generative AI tools were used for drafting support, presentation materials, formatting and language, citation verification and consistency checking. The research questions, taxonomy, autonomy model, scenario library, matrix postures, calibration rules and all findings were developed and decided by the research team. A team member read and verified every cited source against its primary; the authors take full responsibility for the report's accuracy and integrity.

References

Dahl 2024Peer-reviewed
Dahl, Magesh, Suzgun & Ho (2024) Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis 16(1), 64-93.
Confabulation is intrinsic and not self-detectable. Why an information-only system needs controls at all.
Open source ↗
Magesh 2025Peer-reviewed
Magesh, Surani, Dahl, Suzgun, Manning & Ho (2025) Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies 22(2), 216-242.
Retrieval grounding reduces but does not remove hallucination. The control-effectiveness anchor.
Open source ↗
Goddard 2012Peer-reviewed
Goddard, Roudsari & Wyatt (2012) Automation bias: a systematic review of frequency, effect mediators, and mitigators. JAMIA 19(1), 121-127.
The human between output and effect is an unreliable control. Why disclaimers are not scored as mitigating.
Open source ↗
Moffatt 2024Tribunal decision
Moffatt v. Air Canada, 2024 BCCRT 149 B.C. Civil Resolution Tribunal, 14 February 2024. Negligent misrepresentation by an information-only chatbot.
An organisation owns what its information-only chatbot says. Basis for the I1 rating and the citation control.
Open source ↗
Markup 2024Investigative journalism
The Markup / THE CITY (29 March 2024) NYC's AI Chatbot Tells Businesses to Break the Law. Logged as OECD AI Incident 2024-03-29-3dce.
Information-only public-sector bot giving unlawful guidance at scale, left online after publication. Basis for the I2 rating.
Open source ↗
Raine / GarciaLitigation — allegations, not findings
Raine v. OpenAI (filed Aug 2025); Garcia v. Character Technologies (filed Oct 2024) Wrongful-death claims against conversational systems with no tool access.
Low autonomy does not mean low impact. Basis for long-session and adversarial-framing testing at I3.
Open source ↗
EU AI Act Art.50Regulation
Regulation (EU) 2024/1689, Article 50(1) Transparency obligation: users must be told they are interacting with an AI system. Enforceable from 2 August 2026.
Makes first-contact AI disclosure a legal baseline control, not a nice-to-have.
Open source ↗
NIST AI 600-1Framework
NIST (2023; Generative AI Profile July 2024) AI Risk Management Framework 1.0 (AI 100-1) and Generative AI Profile (AI 600-1).
Names confabulation as a first-class risk; supplies testing, monitoring and incident-response actions.
Open source ↗
OWASP LLM09Industry standard
OWASP GenAI Security Project (2025) LLM09:2025 Misinformation, OWASP Top 10 for LLM Applications.
Named mitigations map onto our preventive controls. Cross-reference point to Project 1.
Open source ↗
FCA Enforcement 2024Regulatory
Financial Conduct Authority (2024) FCA fines Starling Bank £29m for failings in their financial crime systems and controls.
Automated sanctions screening ran against an incomplete watchlist for years, undetected through 80x customer growth. Why periodic/batch validation isn't sufficient oversight for a continuously-operating autonomous agent.
Open source ↗
OWASP Agentic AI 2025Industry
OWASP GenAI Security Project (2025) Agentic AI — Threats and Mitigations. v1.0.
Names excessive agency, tool misuse, and prompt injection via ingested content as agentic-specific threat classes. Why an enrichment-only agent still needs technical scope limits, not just policy ones.
Open source ↗
OWASP LLM Top 10 2025Industry
OWASP GenAI Security Project (2025) OWASP Top 10 for LLM Applications 2025. v2.0.
LLM06:2025 Excessive Agency and LLM09:2025 Misinformation. Why the applicable risk category shifts with impact tier — misinformation dominates at enrichment-only (I0), excessive agency dominates once the agent executes (I1–I3).
Open source ↗
PRA SS1/23Regulatory
Bank of England Prudential Regulation Authority (2023) Model risk management principles for banks. Supervisory Statement SS1/23.
Requires model validation independent of the team that built the model. Why 2nd-line MRM oversight is mandatory regardless of AI Act classification.
Open source ↗
SR 11-7 / OCC 2023-17Regulatory
Federal Reserve & Office of the Comptroller of the Currency (2011; 2023) Supervisory Guidance on Model Risk Management (SR 11-7); Third-Party Risk Management (OCC 2023-17).
US analogue to PRA SS1/23, extended to AI/ML and vendor-supplied models. Why independent validation is a convergent prudential requirement across jurisdictions, not a UK-specific rule.
Open source ↗
FINRA 2026 Regulatory Oversight ReportRegulatory
Financial Industry Regulatory Authority (2025) 2026 FINRA Annual Regulatory Oversight Report.
Names agent autonomy without human validation, scope/authority creep, and auditability of multi-step reasoning as emerging risks. Why agentic-specific risks (hallucinated rationale, non-determinism) are flagged but not yet evidenced by a public incident.
Open source ↗
EU AI Act Annex IIIRegulatory
European Parliament and Council of the European Union (2024) Regulation (EU) 2024/1689, Annex III, point 5(b) — Access to and enjoyment of essential private services and essential public services and benefits.
Creditworthiness/credit-scoring high-risk category, with a fraud-detection exception; AML/CFT systems fall in scope only where "functionally linked" to credit evaluation. Why this scenario's high-risk classification under the AI Act is genuinely contested, not settled.
Open source ↗
UK Judicial Guidance 2023Regulatory guidance
Courts and Tribunals Judiciary (UK) (12 December 2023) Artificial Intelligence (AI) Guidance for Judicial Office Holders.
Excludes public GenAI tools from judicial research and confines use to approved sources. Basis for the L1 retrieval-scoping control.
Open source ↗
Mata v. AviancaLitigation
Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023) Sanctions imposed on counsel for submitting ChatGPT-fabricated case citations to a federal court.
Fabricated citations reached a filed legal document undetected until opposing counsel could not locate them. Basis for citation-verification and audit-trail controls wherever an agent drafts a cited document.
Open source ↗
LoomisCase law
State v. Loomis, 2016 WI 68, 371 Wis. 2d 235, 881 N.W.2d 749 (Wis. 2016) Wisconsin Supreme Court upheld sentencing use of a proprietary COMPAS risk score, provided it is not the sole or determinative factor.
Establishes that an advisory score must never be treated as determinative, and that proprietary methodology limits defence cross-examination. Basis for the mandatory-disclosure and non-determinative-use controls.
Open source ↗
ProPublica vs NorthpointeInvestigative journalism — disputed
Angwin, Larson, Mattu & Kirchner, ProPublica (23 May 2016); Northpointe rebuttal (July 2016) Machine Bias: There's Software Used Across the Country to Predict Future Criminals. And It's Biased Against Blacks.
Found Black defendants disproportionately misclassified as high-risk by COMPAS; Northpointe disputed the statistical framing. Basis for treating racial-disparity findings as an unresolved empirical question, not settled fact.
Open source ↗
EU AI Act Annex III 8(a)Regulatory
European Parliament and Council of the European Union (2024) Regulation (EU) 2024/1689, Annex III, point 8(a) — AI systems used by or on behalf of a judicial authority to research, interpret and apply the law to a concrete set of facts.
Classifies AI systems used in judicial research/decision-support as high-risk. Basis for the L1 regulatory-anchor control — distinct from the credit-scoring point 5(b) cited under L4.
Open source ↗
FDA Part-11Regulatory
U.S. Food and Drug Administration 21 CFR Part 11 — Electronic Records; Electronic Signatures.
Requires an immutable, attributable audit trail for electronic records used in regulated decisions. Basis for the draft-to-signature audit trail control.
Open source ↗
FDA 2025 AI GuidanceRegulatory guidance — draft
U.S. Food and Drug Administration, CDER (7 January 2025) Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products. Draft guidance for industry.
Proposes a risk-based credibility assessment framework for AI models used in regulated drug/biological decisions. Basis for the documented model-credibility-case control.
Open source ↗