jackprior.ai · AI-in-CMC Regulatory Map · Job aid

Manage AI risk in five steps

FDA, EMA, ICH, Annex 22, BioPhorum and GAMP each grade AI in their own vocabulary. Underneath, they all ask the same two questions: how much does the AI influence the decision, and what happens if it is wrong. Grey tags say who each item comes from.

The five steps in one table, then the detail each step leans on: the risk grid, the failure modes, the caveats and the sources the image leaves out.

Manage AI risk in 5 steps infographic: define the question of interest and context of use, grade on influence times consequence, identify and mitigate failure modes, layer the controls, monitor and improve across the lifecycle — The job aid as a single image. Open full size (PNG, 2400 × 2800) to save or share.

Step	Why	How
1Define the Question of Interest and Context of Use	FDA's credibility framework and ICH M15 both start here: state the question the AI answers, then the context in which the answer is used. Five questions get you there. ASME V&V 40 → FDA 2025 steps 1–2 · ICH M15 · EMA+FDA principle 4	What question does the AI answer? Which decision depends on the answer? Does a person or a system act on the answer? sets the influence row; EMA workshop: oversight vs human-in-the-loop What action can the AI drive? for an agent, every action it is able to take, not just the ones intended; EMA workshop agentic use case What harm follows if the answer is wrong? sets the consequence column; ICH Q9
2Grade on Influence × Consequence = Risk	FDA 2025 and ICH M15 call the product "model risk" and grade it low, medium or high, following ASME V&V 40. BioPhorum calls it composite risk. The risk grade sets how much validation evidence, control and oversight is required, and no more. ICH Q9(R1) formality · EMA+FDA principle 2 · FDA 2025 steps 4–7 · ICH M15 · Annex 22	Influence: how much does the AI drive the decision, given other evidence and controls? Advisory → contributing → determinative. FDA 2025, ICH M15: model influence · ICH PtC 2011: low / medium / high impact · PMF REF: model responsibility level · BioPhorum: "influence" = maturity (see caveats) Consequence: how bad if the decision is wrong? Business → data integrity → product quality → patient safety. ICH Q9: harm · FDA 2025, ICH M15: decision consequence · EMA RP: patient risk · Annex 22: critical / non-critical · BioPhorum: severity · PMF REF: potential HARM Risk = influence × consequence, read off the grid below. The labels are FDA's; the cell placement is my interpretation, not from any guidance.
3Identify and mitigate failure modes	The grade says how much evidence is needed. The failure modes say what kind. Same grade, different mitigation, depending on how the AI is built and behaves. These five are on the image; the full list below adds the ones regulators also name.	Unrepresentative data: provenance and coverage review before use Fails silently: in-model checks, out-of-distribution detection Hallucinates: bind outputs to source and verify against it Overconfident when wrong: uncertainty estimates, confidence gates Acts on its own: permission and audit every action; human gate on consequential ones
4Layer the controls and name what they cover	Place every mitigation at one or more layers, so no single control is load-bearing, and name the failure mode each one covers. Unanticipated failures are not covered. EMA Annex 22 workshop report, Oct 2026 ("Swiss cheese"; likely to enter the annex as "control mechanisms")	Prevent at input: context-of-use checks, out-of-distribution detection, rule limits Detect in the model: source binding, uncertainty estimates, conformal prediction Contain at output: hard limits, confidence gates, human gates, monitoring, fallback to a validated state or stop
5Monitor and improve across the lifecycle	Validation is a lifecycle, not an event. Every 2026 document says so. Improvement is welcome but controlled: a model does not update itself in GMP; every change goes through the change plan and is re-tested. EMA+FDA principle 9 · Annex 22 §10 · FDA SaMD action plan 2021 · GMLP 10 · EMA workshop topic 4 · ICH Q10 continual improvement	Monitor performance and drift in use Annex 22 §10.4 · BioPhorum §8.7 change triggers Change plan: pre-agreed updates and re-test triggers FDA PCCP · ICH Q12 PACMP Periodic review: keep the qualification dossier current EMA workshop ("AI qualification dossier") · ICH Q10

2The risk grid

Proportionate effort: low risk earns a light touch; high risk earns the full dossier.
	Consequence if wrong →
AI influence ↑	Business only	Data integrity, compliance	Product quality	Patient safety
Determinativesole basis of decision	Not GxPno GxP decision	Mediumrisk	Highrisk	Highrisk
Contributingone control of several	Not GxPno GxP decision	Mediumrisk	Mediumrisk	Highrisk
Advisoryhumans decide	Not GxPno GxP decision	Lowrisk	Lowrisk	Mediumrisk

Risk grade	Example	Equivalents
Not GxP	Cell-line ranking in development	PtC low · FDA n/a unless it enters a submission · Annex 22 out of scope
Low	SOP drafting, QA-approved	PtC low · FDA/M15 low · Annex 22 non-critical
Medium	Anomaly alerts acted on by operators	PtC medium · FDA/M15 medium · Annex 22 critical
High	Chemometric release assay, autonomous feed control	PtC high · FDA/M15 high · Annex 22 critical

3Failure modes, the full list

Failure mode	Mitigation	Source
Unrepresentative data	Provenance and coverage review before use	FDA 2025 step 4 · EMA+FDA principle 6 · GMLP 2–3 · Annex 22 §4
Fails silently	In-model checks, out-of-distribution detection	ICH Q9(R1) detectability · EMA workshop
Hallucinates	Bind outputs to source and verify against it	EMA workshop (RAG with source binding) · NIST AI 600-1 · GAMP AI Guide
Overconfident when wrong	Uncertainty estimates, confidence gates	FDA 2025 · ICH M15 · EMA workshop (uncertainty quantification, conformal prediction)
Acts on its own	Permission and audit every action; human gate on consequential ones	EMA workshop agentic use case · BioPhorum ("AI must not execute e-signatures")
Cannot explain its answer	Explainability sized to the decision; reviewer able to challenge	EMA RP · EMA workshop topic 3
Same input, different answer	Fix settings, or characterise the spread	Annex 22 draft probabilistic gate · EMA workshop
Supplier owns the model	Contract for evidence, metrics and change notice	Annex 11 · EMA workshop topic 6
Humans stop checking it	Periodic challenge; reviewer must understand the model	EMA workshop topic 3 · BioPhorum (HITL as risk source)

Also check

Regulatory impact: does the output become submission evidence? Business-only output that enters a filing is not "Not GxP" in EMA's scheme. EMA RP, second axis
EU AI Act tier: high-risk only if a device safety component Reg. 2024/1689 Art. 6
GAMP category: the computerised-system lens ISPE GAMP 5 · GAMP AI Guide

Caveats

Annex 22 is still a draft. The July 2025 text says dynamic models and LLMs should not be used in critical GMP applications. The October 2026 workshop report records the drafting group discussing a broader scope under a documented control strategy, but no revised text has been published. An EU inspector can still apply the draft.

A human in the loop lowers influence but does not fix a bad model. The workshop report says HITL is one control among several and cannot compensate for inadequate validation; BioPhorum treats it as a risk source as well as a control.

Impact ≠ impact. ICH PtC "high-impact model" and ICH M15 "model impact" are different things. M15's equivalent is model risk.

Influence ≠ influence. BioPhorum's model influence means maturity (autonomy × adaptiveness), not weight of evidence. Re-derive it before using it in an FDA credibility plan.

Five steps compress FDA's seven. Steps 1 and 2 here are FDA's steps 1 to 3; FDA's steps 4 to 7 (plan, execute, document, assess adequacy) sit behind "validation evidence required". Steps 3 to 5 come from EMA and Annex 22, not FDA.

Sources

FDA 2025: FDA draft guidance, Considerations for the Use of AI to Support Regulatory Decision-Making for Drug and Biological Products, Jan 2025 (7-step credibility framework)
ASME V&V 40: ASME V&V 40-2018, credibility of computational models; origin of FDA's model-risk logic
ICH M15: ICH M15 General Principles for Model-Informed Drug Development, adopted Jan 2026
ICH PtC: ICH Q8/Q9/Q10 Points to Consider, 2011, model section
ICH Q9(R1): ICH Q9(R1) Quality Risk Management, 2023
EMA RP: EMA Reflection Paper on the Use of AI in the Medicinal Product Lifecycle, final Sept 2024
EMA+FDA: EMA and FDA, Guiding Principles of Good AI Practice in Drug Development, Jan 2026 (ten principles)
Annex 22: EU GMP Annex 22 Artificial Intelligence, draft for consultation Jul 2025
EMA workshop: EMA report, Multistakeholder workshop on expert contributions to AI guidance development (Annex 22), Oct 2026
BioPhorum: BioPhorum AI risk guidance for the pharmaceutical industry, Jun 2026
GAMP: ISPE GAMP 5 (2nd ed., 2022) and GAMP Guide: Artificial Intelligence, Jul 2025
GMLP: FDA, Health Canada, MHRA Good Machine Learning Practice guiding principles, 2021
NIST AI 600-1: NIST Generative AI Profile, Jul 2024
AI Act: Regulation (EU) 2024/1689
PMF REF: Pharma Manufacturing Forum industry proposal (unpublished, 2026): Risk Evaluation Framework