This is the fact check of the episode as published. Each factual claim was extracted from the script and verified by an AI agent that saw only the claim and the primary document (the PDFs in the corpus). Verdicts: SUPPORTED (the document says it), PARTIAL (supported with a difference, noted), NOT-IN-CORPUS (the source is not among the primary documents on disk), NOT-CHECKABLE (an estimate or a characterisation). Opinions voiced by the hosts are listed but not verified. This report is itself AI output and can be wrong; corrections: chat@jackprior.ai.
Treat these lines with caution.
These statements stand, but the primary documents on disk do not themselves confirm them: they come from the landscape reference (a working index whose rows are checked against the web), are estimates, or are hedged in the episode as such.
| Line | Speaker | Claim | Verdict | Evidence |
|---|
| 0 | HOST | The EMA workshop report is fifteen pages long. | SUPPORTED | Running header on every page, e.g. "EMA/156789/2026 Page 2/15" through "Page 15/15" (page 15 is a back cover with address and copyright). |
| 0 | HOST | The report is published by the European Medicines Agency. | SUPPORTED | p.15: "European Medicines Agency / Domenico Scarlattilaan 6 / 1083 HS Amsterdam" ... "© European Medicines Agency, 2026"; document number EMA/156789/2026 in every header. |
| 0 | HOST | The third page of the report says the drafting group has discussed allowing generative AI in GMP applications, critical or not, provided it is supported by a documented control strategy. | PARTIAL | p.3, Introduction: "The drafting group has therefore discussed the scope of Annex 22 to allow dynamic or adaptive models, probabilistic models, and Generative AI/LLMs in GMP applications, provided they comply with the requirements of the annex and are supported by a fully documented, robust, risk-based control strategy." |
| 2 | HOST | The report describes risk scenarios mapped to controls mapped to evidence, an oversight role with a stated competence, vendor agreements giving access to the model's logs, and a documented go/no-go (do-not-use) decision. | SUPPORTED | p.6: "it should describe the risk scenarios, the controls mapped to those scenarios, and the evidence generated through verification and monitoring"; p.8: "must understand the relevant model principles, the process, the output and the decision context sufficiently to challenge the AI"; p.12: "agreements must clarify responsibilities, configuration expectations, change control participation, evidence access"; p.9: "If a third-party model or API cannot provide necessary outputs, rationale, logs or evidence needed for control, the manufacturer cannot measure and justify the system"; p.7: "document the go/no-go conclusion"; p.6: "A valid QRM outcome may still be 'do not use'" |
| 3 | SARAH | EMA has published the report of its expert workshop on Annex 22. | SUPPORTED | p.1 cover: "Multistakeholder workshop on expert contributions to artificial intelligence guidance development (Annex 22) / 30 June 2026"; p.15: "© European Medicines Agency, 2026" |
| 3 | SARAH | The report records what industry told the drafting group and what the drafting group said it is now considering. | SUPPORTED | p.3: "The drafting group has therefore discussed the scope of Annex 22 to allow..."; each topic has "Industry Perspective" and "Potential Impact on Annex 22 Drafting"; p.14: "will be carefully reviewed and considered by the Annex 22 drafting group" |
| 6 | SARAH | The report is titled as EMA's report of a multistakeholder workshop on Annex 22. | SUPPORTED | p.1: "Good manufacturing practice (GMP) / Multistakeholder workshop on expert contributions to artificial intelligence guidance development (Annex 22)" |
| 6 | SARAH | The report's document number is EMA/156789/2026. | SUPPORTED | pp.2-15 headers: "EMA/156789/2026 Page 2/15" |
| 6 | SARAH | The report is fifteen pages. | SUPPORTED | p.15 header: "EMA/156789/2026 Page 15/15" |
| 6 | SARAH | The report's cover carries the workshop date, 30 June 2026. | SUPPORTED | p.1: "...guidance development (Annex 22) / 30 June 2026" |
| 8 | SARAH | The drafting group requested the meeting with AI experts that the report describes. | SUPPORTED | p.3, Introduction: "The drafting group therefore requested a meeting with AI experts to assess whether a risk-based approach can be applied to Generative AI/LLMs" |
| 8 | SARAH | The document is a final report of a workshop; it is not a revised annex, a consultation or guidance. | SUPPORTED | p.14: "The outcomes of these discussions will be used to inform the ongoing revision of Annex 22"; p.3: "The insights obtained were intended to inform the revision of the scope of Annex 22" |
| 8 | SARAH | The report's closing page is headed next steps and says the input will be carefully reviewed and considered by the drafting group. | SUPPORTED | p.14, section 3: "3. Next Steps / The information presented during the workshop, including the consolidated responses provided by the participating industry associations, will be carefully reviewed and considered by the Annex 22 drafting group." |
| 8 | SARAH | The report says what the drafting group has been discussing regarding the scope of Annex 22. | SUPPORTED | p.3, Introduction: "The drafting group has therefore discussed the scope of Annex 22 to allow dynamic or adaptive models, probabilistic models, and Generative AI/LLMs in GMP applications" |
| 10 | SARAH | The report says the drafting group has discussed the scope; it does not say a revised text has been drafted. | SUPPORTED | p.3: "The drafting group has therefore discussed the scope of Annex 22"; p.14: "will be used to inform the ongoing revision of Annex 22" |
| 11 | HOST | One of industry's worked examples at the 30 June workshop was an agentic AI system supporting deviation management. | SUPPORTED | pp.8-9, Topic 2: "The presentation used five use cases: ... an agentic AI system supporting deviation management; ..." |
| 12 | SARAH | The report's introduction quotes the draft annex: the document does not apply to generative AI and large language models, models with a probabilistic output, dynamic models and models which adapt performance, and such models should not be used in critical GMP applications. | SUPPORTED | p.3, Introduction: "The draft version of Annex 22 currently states 'the document does not apply to Generative AI and Large Language Models (LLM), models with a probabilistic output, dynamic models and models which adapt performance, such models should not be used in critical GMP applications.'" |
| 12 | SARAH | The introduction says comments received during the public consultation indicated support for allowing manufacturers to use large language models and generative AI systems in GMP applications, whether critical or non-critical. | SUPPORTED | p.3, Introduction: "Comments received during the public consultation on Annex 22 indicated support for allowing manufacturers to use Large Language Models and generative AI systems in GMP applications, whether critical or non-critical." |
| 14 | SARAH | The introduction says: the drafting group has therefore discussed the scope of Annex 22 to allow dynamic or adaptive models, probabilistic models, and generative AI/LLMs in GMP applications, provided they comply with the requirements of the annex and are supported by a fully documented, robust, risk-based control strategy. | SUPPORTED | p.3, Introduction, verbatim: "The drafting group has therefore discussed the scope of Annex 22 to allow dynamic or adaptive models, probabilistic models, and Generative AI/LLMs in GMP applications, provided they comply with the requirements of the annex and are supported by a fully documented, robust, risk-based control strategy." |
| 14 | SARAH | The paragraph after that sentence says concerns remain about the elements of such a control strategy, and that the drafting group therefore requested a meeting with AI experts to assess whether a risk-based approach can be applied to generative AI. | SUPPORTED | p.3, Introduction: "However, concerns remain about the elements needed for such a control strategy. The drafting group therefore requested a meeting with AI experts to assess whether a risk-based approach can be applied to Generative AI/LLMs" |
| 16 | SARAH | The report frames the workshop as seeking experience-based input on control, governance and mitigation across the AI lifecycle, with emphasis on ICH Q9. | SUPPORTED | p.3, Introduction: "to gather structured, experience-based input from industry on the control, governance, and mitigation measures applied throughout the AI system lifecycle, with particular emphasis on the application of ICH Q9 quality risk management principles" |
| 16 | SARAH | The industry associations registered with EMA responded jointly, as one consolidated view, across six topics. | SUPPORTED | p.3, Introduction: "industry stakeholder associations registered with European Medicines Agency (EMA) collaborated to provide a consolidated response to the List of Questions ... a collective industry perspective across the six predefined topics" |
| 16 | SARAH | For each topic the report gives the drafting group's questions, the industry perspective, a Q&A, key messages, and a paragraph headed potential impact on Annex 22 drafting. | PARTIAL | Topic 1 headings, pp.4-5: "Questions from the Drafting Group", "Industry Perspective", "Q&A session", "Key Messages from the Session", "Potential Impact on Annex 22 Drafting". Topic 2, pp.8-10: "Questions from the Drafting Group" then "The fourth session provided the most technical discussion..." with no 'Industry Perspective' or 'Q&A session' heading. |
| 16 | SARAH | The potential-impact paragraphs are written in conditional language throughout (may be influenced, could require, definitions may need). | SUPPORTED | p.7: "Annex 22 may be influenced towards an explicit performance-based acceptability framework"; p.10: "The annex could require layered controls"; p.5: "Definitions may need to clarify"; p.12: "Definitions may need to include" |
| 16 | SARAH | The report's body takes the topics in the order 1, 5, 3, 2, 4, 6. | SUPPORTED | p.2 Table of contents: "1.1. Topic 1 ... 1.2. Topic 5 ... 1.3. Topic 3 ... 1.4. Topic 2 ... 1.5. Topic 4 ... 1.6. Topic 6"; body sections 1.1-1.6 follow the same order. |
| 17 | SARAH | Topic one is titled regulatory pathways for adaptive and probabilistic models. | SUPPORTED | p.4: "1.1. Topic 1: Regulatory Pathways for Adaptive/probabilistic AI models in GMP" |
| 17 | SARAH | In topic one the drafting group asked how adaptive/probabilistic models could be accommodated, what their validation paradigm would be, and how quality risk management should apply across the lifecycle of a model in high-risk areas where deficiencies have low detectability and high patient impact. | SUPPORTED | p.4, Questions from the Drafting Group: "How could we accommodate adaptive and probabilistic models in Annex 22?" "What would be the validation paradigm for adaptive models..." "...quality risk management approach to the entire lifecycle of such AI system models that may be used in GMP high risk areas where there may be a low detectability of deficiencies but a high impact on the patient" |
| 19 | SARAH | Industry's topic-one position was that adaptive, probabilistic and generative models should not be excluded from GMP applications on the basis of a technology label alone. | SUPPORTED | p.4, Industry Perspective: "adaptive, probabilistic and generative AI models should not be excluded from GMP applications on the basis of a technology label alone" |
| 19 | SARAH | Industry said the criterion should be whether the model can be shown fit for its intended use and whether its risks can be controlled within the existing quality system. | SUPPORTED | p.4, Industry Perspective: "the central criterion should be whether the model can be demonstrated to be fit for its intended use and whether the associated risks can be controlled within an existing pharmaceutical quality system" |
| 19 | SARAH | Industry used a measurement analogy: instruments drift, calibration intervals are set on a risk basis, and lifecycle review confirms a system is still fit for use. | SUPPORTED | p.4, Industry Perspective: "Measurement systems drift, calibration intervals are adjusted on a risk basis, and lifecycle review is already used to confirm that a system remains fit for use." |
| 19 | SARAH | The report adds that the analogy was not meant to equate AI adaptation with measurement drift, only to show that GMP already reassesses suitability when behaviour changes. | SUPPORTED | p.4, Industry Perspective: "This analogy was not intended to equate all forms of AI adaptation with measurement drift, but to illustrate that GMP already contains mechanisms for reassessing continued suitability when system behaviour changes over time." |
| 21 | SARAH | Regulators asked whether hallucination, fabrication, sycophancy and epistemic overconfidence are ordinary statistical uncertainty or a fundamentally different failure mode. | SUPPORTED | p.4, Q&A session: "including hallucination, fabrication, sycophancy or epistemic overconfidence. Regulators questioned whether these phenomena are merely ordinary statistical uncertainty or a fundamentally different failure mode requiring specific attention." |
| 21 | SARAH | Industry answered that such behaviours must be identified and evaluated during development and validation, and that if the manufacturer cannot demonstrate fitness for intended use in view of them, the use cannot be justified. | SUPPORTED | p.4, Q&A session: "The response was that such behaviours must be identified and evaluated during development and validation; if the manufacturer cannot demonstrate fitness for intended use in view of those failure modes, the use cannot be justified." |
| 21 | SARAH | No technical method for estimating hallucination rates was set at the workshop; the question was deferred. | SUPPORTED | p.4, Q&A session: "The discussion did not set a specific technical method for estimating confidence intervals or hallucination rates; rather, it deferred detailed technical controls to later sessions." |
| 23 | SARAH | Topic one adopted ICH Q9(R1) as its risk vocabulary (uncertainty, importance, complexity). | SUPPORTED | p.4: "The concepts of importance, uncertainty, complexity and level of formality should be taken into consideration when a risk assessment is conducted"; p.5: "complexity is explicitly part of the ICH Q9(R1) formality concept"; the question cites "ICH Q9R1 principles" (p.4). |
| 23 | SARAH | Industry at the workshop leaned on the Q9 sentence that formality should be commensurate with uncertainty, importance and complexity. | PARTIAL | p.4: "The concepts of importance, uncertainty, complexity and level of formality should be taken into consideration when a risk assessment is conducted"; p.5: "complexity is explicitly part of the ICH Q9(R1) formality concept and should be included when determining the necessary rigour of controls" |
| 25 | SARAH | Industry's position was that uncertainty should drive knowledge generation, stronger lifecycle controls, more formal validation and more monitoring, rather than an automatic prohibition. | SUPPORTED | p.4, Industry Perspective: "uncertainty should drive increased knowledge generation, stronger lifecycle controls, more formal validation, monitoring and re-verification, rather than an automatic prohibition" |
| 27 | SARAH | The moderator raised that in adaptive or generative AI the inability to detect failure may itself be a central risk factor. | SUPPORTED | p.4, Q&A session: "The moderator noted that in adaptive or generative AI, inability to detect failure may itself be a central risk factor." |
| 27 | SARAH | The reply at the workshop went back to ICH Q9's text: risk analysis is about likelihood and severity, and in some risk management tools the ability to detect the harm (detectability) also factors in the estimation of risk. | PARTIAL | pp.4-5, Q&A session: "The speaker acknowledged that, although the general definition of risk in ICH Q9 refers to severity and probability, methods such as Failure Mode and Effects Analysis (FMEA) may use detectability because it is useful in certain contexts." |
| 27 | SARAH | A speaker said that for AI, detectability may be highly appropriate as a dimension, particularly where errors are difficult to observe. | SUPPORTED | p.5, Q&A session: "For AI, detectability may be highly appropriate as a dimension of the assessment, particularly where errors are difficult to observe." |
| 27 | SARAH | The session supported considering detectability without mandating a scoring method. | SUPPORTED | p.5, Q&A session: "The discussion therefore supported consideration of detectability, without mandating a single risk scoring method." |
| 28 | HOST | The assessment the report proposes uses the dimensions consequence, influence, uncertainty, complexity and detectability. | SUPPORTED | p.7, Topic 5 Potential Impact: "Annex 22 may be influenced towards an explicit performance-based acceptability framework: identify intended use; evaluate decision consequence, model influence, uncertainty, complexity and detectability; define controls; verify them; monitor residual risk" |
| 30 | SARAH | The report uses both terms, model influence and decision consequence, without citing FDA. | SUPPORTED | p.6: "considering decision consequence, model influence, uncertainty, complexity and the ability to reduce residual risk"; p.7: "evaluate decision consequence, model influence, uncertainty, complexity and detectability". The strings 'FDA' and 'Food and Drug' do not occur anywhere in the document. |
| 30 | SARAH | The topic-one potential-impact paragraph says definitions may need to clarify intended use, model influence, complexity, uncertainty, control strategy, detectability and continued fitness for use. | SUPPORTED | p.5, Topic 1 Potential Impact on Annex 22 Drafting: "Definitions may need to clarify intended use, model influence, complexity, uncertainty, control strategy, detectability and continued fitness for use." |
| 31 | SARAH | Topic five is titled strategic risk and compliance limits. | SUPPORTED | p.5 heading (also TOC p.2, topic list p.3): "1.2. Topic 5: Strategic Risk & Compliance Limits" |
| 31 | SARAH | The drafting group's topic-five questions asked where the limits of risk-based mitigation lie: whether there are classes of critical decisions where no combination of guardrails and oversight would be sufficient, and whether there are cases where the guardrail effort becomes so disproportionate that the system should not be used in certain critical functions. | SUPPORTED | p.5, Topic 5 Questions from the Drafting Group, bullets 2-3: "Where do experts believe the limits of risk-based mitigation lie? Are there classes of critical decisions where no combination of guardrails and oversight would be sufficient? ... level of guardrail effort ... becomes disproportionate or operationally impractical" |
| 33 | SARAH | Industry's topic-five position was strongly against categorical prohibitions. | SUPPORTED | p.6, Topic 5 Industry Perspective: "The industry position was strongly against categorical prohibitions." |
| 33 | SARAH | Industry said no class of decisions should be considered unacceptable solely by technology category or criticality label. | SUPPORTED | p.6, Topic 5 Industry Perspective: "there should be no class of decisions considered unacceptable solely by technology category or criticality label" |
| 33 | SARAH | Industry said acceptability should be assessed case by case on decision consequence, model influence, uncertainty, complexity and the ability to reduce residual risk. | SUPPORTED | p.6, Topic 5 Industry Perspective: "acceptability should be assessed case by case by considering decision consequence, model influence, uncertainty, complexity and the ability to reduce residual risk to an acceptable level" |
| 33 | SARAH | Industry said that where the mitigation effort is heavy, impractical or insufficient, the use is not justified at that time and in that context, and that conclusion should be documented. | SUPPORTED | p.6, Topic 5 Industry Perspective: "in some use cases the mitigation effort may be heavy, impractical, or insufficient. In such cases the conclusion would be that the use is not justified at that time and in that context, and the conclusion should be documented." |
| 35 | SARAH | A topic-five key message says a valid quality risk management outcome may still be do not use where uncertainty cannot be controlled. | SUPPORTED | p.6, Topic 5 Key Messages, bullet 2: "A valid QRM outcome may still be 'do not use' where uncertainty cannot be controlled." |
| 35 | SARAH | The report records the question of how a manufacturer demonstrates that layered controls provide reliable output for a critical decision. | SUPPORTED | p.6, Topic 5 Q&A: "how a manufacturer could demonstrate to inspectors that the combined layered controls do provide reliable output for a critical decision" |
| 35 | SARAH | The answer recorded was a risk assessment that describes the risk scenarios, the controls mapped to those scenarios, and the evidence from verification and monitoring; then monitoring history, responsiveness to signals and a documented state of control become the inspection evidence. | SUPPORTED | p.6, Topic 5 Q&A: "the risk assessment as the starting point: it should describe the risk scenarios, the controls mapped to those scenarios, and the evidence generated through verification and monitoring. Monitoring history, responsiveness to signals, and documented state of control would then be inspection evidence." |
| 37 | SARAH | Regulators asked how a responsible person can take responsibility for a model that is not fully transparent. | SUPPORTED | p.6, Topic 5 Q&A: "Regulators asked how a responsible person could take responsibility for a model that is not fully transparent." |
| 37 | SARAH | The reply was that AI does not remove predicate-rule responsibilities, and accountable persons may rely on qualified experts, as they already do for complex technologies. | SUPPORTED | p.6, Topic 5 Q&A: "the use of AI should not remove predicate-rule responsibilities and that accountable persons may rely on qualified experts, as they already do for complex manufacturing technologies" |
| 37 | SARAH | An industry speaker introduced the phrase AI qualification dossier, covering validation evidence, training-data information, guardrails around prompting and output, and evidence that the system works as expected. | SUPPORTED | p.6, Topic 5 Q&A: "The idea of an 'AI qualification dossier' was introduced by another industry speaker as a package containing validation evidence, training data information, guardrails around prompting and output, and evidence that the system works as expected." |
| 38 | HOST | The report does not state that any revised text has been written into the annex. | SUPPORTED | p.14, Next Steps: "will be carefully reviewed and considered by the Annex 22 drafting group ... used to inform the ongoing revision of Annex 22" |
| 39 | SARAH | Topic three is titled human oversight and accountability. | SUPPORTED | p.7 heading (also TOC p.2, topic list p.3): "1.3. Topic 3: Human Oversight & Accountability" |
| 39 | SARAH | The drafting group asked what level and form of human-in-the-loop oversight is still required when guardrails are implemented. | SUPPORTED | p.7, Topic 3 Questions from the Drafting Group: "What level and form of human -in-the-loop oversight is still required when guardrails are implemented, and is this oversight sufficient to ensure accuracy, traceability, and accountability?" |
| 39 | SARAH | Industry defined human oversight as the broad lifecycle governance objective: a qualified, accountable human can understand, challenge, intervene, override, reject or escalate. | SUPPORTED | p.7, Topic 3 Industry Perspective: "'human oversight' as a broad lifecycle governance objective ... Human oversight asks whether a qualified, accountable human can understand, challenge, intervene, override, reject or escalate." |
| 39 | SARAH | Industry defined human-in-the-loop as one specific implementation pattern in which the process waits for a human action before a GMP-relevant workflow proceeds. | SUPPORTED | p.7, Topic 3 Industry Perspective: "'human in the loop' as one specific implementation pattern ... Human-in-the-loop, by contrast, is where the process waits for a human action before a GMP-relevant workflow proceeds." |
| 40 | HOST | The report presents human-in-the-loop as one way to achieve oversight, not the definition of oversight. | SUPPORTED | p.8, Topic 3 Key Messages, bullets 1-2: "Human oversight is broader than human-in-the-loop and should be lifecycle-wide. / Human-in-the-loop is one possible control, not a default requirement for all AI uses." |
| 41 | SARAH | A topic-three key message says human-in-the-loop is one possible control, not a default requirement for all AI uses. | SUPPORTED | p.8, Topic 3 Key Messages, bullet 2 (verbatim): "Human-in-the-loop is one possible control, not a default requirement for all AI uses." |
| 41 | SARAH | A key message says oversight must be proportionate to risk, autonomy, uncertainty, complexity and decision impact. | SUPPORTED | p.8, Topic 3 Key Messages, bullet 3 (verbatim): "Oversight must be proportionate to risk, autonomy, uncertainty, complexity and decision impact." |
| 41 | SARAH | A key message says oversight should be evidenced and periodically reviewed, and cannot compensate for inadequate validation. | SUPPORTED | p.8, Topic 3 Key Messages, bullet 5: "Oversight should be evidenced and periodically reviewed; it cannot compensate for inadequate validation." |
| 41 | SARAH | The session supports annex language requiring meaningful, risk-based and documented oversight rather than default, symbolic or one-size-fits-all human-in-the-loop. | SUPPORTED | p.8, Topic 3 Q&A closing sentence: "the session supports Annex 22 language requiring meaningful, risk-based and documented oversight rather than default, symbolic or one-size-fits-all human-in-the-loop." |
| 44 | SARAH | On page 8 the report says a human reviewer does not need to understand every internal detail of the model, but must understand the relevant model principles, the process, the output and the decision context sufficiently to challenge the AI. | SUPPORTED | p.8, top of page (Topic 3 Q&A), verbatim: "A human reviewer does not need to understand every internal detail of the model, but must understand the relevant model principles, the process, the output and the decision context sufficiently to challenge the AI." |
| 44 | SARAH | Regulators asked how to monitor the performance of the human doing the oversight, how to manage automation bias, and how oversight is evidenced on inspection. | SUPPORTED | p.7, Topic 3 Q&A: "Regulators asked how to monitor the performance of the human performing oversight; how to manage automation bias; ... and how oversight can be evidenced during inspection." |
| 46 | SARAH | The suggested evidence of oversight was training, qualification, knowledge testing, audit trails, recording of AI outputs and human modifications, and monitoring of human-AI team performance. | SUPPORTED | p.7, Topic 3 Q&A: "Responses identified training, qualification, knowledge testing, audit trails, recording of AI outputs and human modifications, and monitoring of human-AI team performance as potential evidence." |
| 46 | SARAH | A speaker said oversight alone does not correct risk; it provides knowledge, detection and triggers for action, so it should be judged by whether it leads to effective follow-up. | SUPPORTED | p.8, Topic 3 Q&A: "oversight alone does not directly correct risk; it provides knowledge, detection and potential triggers for action. Therefore, human oversight should be judged by whether it produces relevant knowledge and supports effective risk reduction through follow-up actions." |
| 47 | SARAH | Topic two is titled technical reliability and incident response. | SUPPORTED | p.8 heading (also TOC p.2, topic list p.3): "1.4. Topic 2: Technical Reliability & Incident Response" |
| 47 | SARAH | The topic-two question was whether available guardrail mechanisms can reliably prevent, detect or contain hallucinations, incorrect recommendations or fabricated data, and what ensures escalation when they fail. | SUPPORTED | p.8, Topic 2 Questions from the Drafting Group: "To what extent can available guardrail mechanisms reliably prevent, detect, or contain hallucinations, incorrect recommendations, or fabricated data ... When guardrails fail or detect uncertainty, what mechanisms are required to ensure timely escalation" |
| 47 | SARAH | Industry presented five use cases: adaptive HVAC control within validated boundaries; an agentic AI system supporting deviation management; an LLM supporting a GMP risk assessment for an electronic batch record system; text-to-SQL generation for annual product review data; and model predictive control for bioreactor feeding. | SUPPORTED | pp.8-9, Topic 2: "adaptive Heating, Ventilation and Air Conditioning (HVAC) control within validated environmental boundaries; an agentic AI system supporting deviation management; an LLM supporting GMP risk assessment for an electronic batch record system; text-to-SQL generation for annual product review data extraction; and model predictive control for bioreactor feeding." |
| 47 | SARAH | Some of the use cases were pilots or theoretical frameworks rather than production systems. | SUPPORTED | p.9, Topic 2: "Some examples were stated to be pilots or theoretical frameworks rather than production systems." |
| 49 | SARAH | The central model presented was layered control in three layers: input/prevention controls, in-model/detection controls, and output/containment controls, with a Swiss cheese image. | SUPPORTED | p.9, Topic 2: "A central model was layered control: input/prevention controls, in-model/detection controls, and output/containment controls. This was linked to the 'Swiss cheese' concept" |
| 49 | SARAH | Input-layer controls listed include context-of-use verification and out-of-distribution detection. | SUPPORTED | p.9, Topic 2: "Examples of input controls included context-of-use verification, out-of-distribution detection, semantic validation and rule-based limits." |
| 49 | SARAH | In-model controls listed include retrieval-augmented generation with source and citation binding, conformal prediction and uncertainty quantification. | SUPPORTED | p.9, Topic 2: "In-model controls included retrieval augmented generation with source/citation binding, conformal prediction, uncertainty quantification, self-consistency scoring and access to model logs or rationale where available." |
| 49 | SARAH | Output-layer controls listed include hard constraints, confidence gates, cumulative-sum and moving-average monitoring, human gates, and fallback to validated states. | SUPPORTED | p.9, Topic 2: "Output controls included hard constraints, confidence gates, cumulative sum monitoring, exponentially weighted moving averages, ensemble disagreement, human gates and fallback to validated states." |
| 51 | SARAH | The presenter cautioned that guardrails are specific controls for specific risks, and unanticipated failure modes cannot be assumed to be covered by controls designed for known ones. | SUPPORTED | p.9, Topic 2: "The presenter repeatedly emphasised that guardrails are specific controls for specific risks and that unanticipated failure modes cannot be assumed to be covered by controls designed for known failure modes." |
| 51 | SARAH | The report says risk assessments are limited by available knowledge and must be updated through lifecycle learning, deviations and defects. | SUPPORTED | p.9, Topic 2 Q&A: "risk assessments are limited by available knowledge and must be updated through lifecycle learning, deviations and defects." |
| 53 | SARAH | Several participants questioned whether the word guardrail should appear in the annex at all. | SUPPORTED | p.9, Topic 2: "Terminology was a major issue. Several participants questioned whether the term 'guardrail' should appear in Annex 22." |
| 53 | SARAH | Industry suggested the term control mechanisms for the main annex text because guardrails are technology-specific and transient. | SUPPORTED | p.9, Topic 2: "Industry responses suggested that 'control mechanisms' may be preferable in the main annex because guardrails can be technology-specific and transient." |
| 53 | SARAH | Industry suggested 'control mechanisms' for the main annex, and the discussion agreed that principles and objectives (prevention, detection, containment) belong in the annex and methods and use cases in appendices, a Q&A or another updateable format. | SUPPORTED | p. 9: 'Industry responses suggested that control mechanisms may be preferable in the main annex'; 'The discussion reflects agreement that the main annex should contain principles and objectives ... while technical methods and use cases may be better placed in appendices, Q&A, or another updateable format.' |
| 53 | SARAH | The topic-two potential-impact paragraph says definitions may need to cover control mechanisms, unacceptable output, failure mode, escalation, fallback, safe state and monitoring threshold. | SUPPORTED | p.10, Topic 2 Potential Impact on Annex 22 Drafting (verbatim): "Definitions may need to cover control mechanisms, unacceptable output, failure mode, escalation, fallback, safe state and monitoring threshold." |
| 55 | SARAH | The presenter said the key is not ownership of the model but access to sufficient evidence to measure how it works; if a third-party model or API cannot provide the outputs, rationale, logs or evidence needed for control, the manufacturer cannot measure and justify the system. | SUPPORTED | p.9, Topic 2 Q&A: "The presenter stated that the key is not ownership of the model but access to sufficient evidence to measure how it works. If a third-party model or Application Programming Interface (API) cannot provide necessary outputs, rationale, logs or evidence needed for control" |
| 55 | SARAH | On fallback, the HVAC and bioreactor examples revert to a pre-validated classical state, and where a fully model-operated process has no such state, sometimes accepting stoppage may be the contingency. | SUPPORTED | pp.9-10, Topic 2 Q&A: "For HVAC and bioreactor examples, failure or out-of-bound behaviour led to reversion to a pre-validated classical control state. Regulators noted that a fully model-operated process might not have such a classical fallback ... sometimes accepting stoppage may be the contingency." |
| 57 | SARAH | Topic four is titled validation and lifecycle management. | SUPPORTED | p.2 TOC and p.10 heading: "1.5. Topic 4: Validation & Lifecycle Management" |
| 57 | SARAH | Industry framed topic four as the integrating topic. | SUPPORTED | p.10, Topic 4 Industry Perspective: "The fifth session addressed validation and lifecycle management as the integrative framework connecting the other topics." |
| 57 | SARAH | Industry's answer to how guardrails stay effective as a model evolves was a lifecycle decision framework rather than a validation event. | SUPPORTED | p.10, Topic 4 Industry Perspective: "The presentation emphasised a lifecycle decision-making framework rather than a single validation event."; question text: "How can guardrails remain effective when the underlying AI model evolves" |
| 57 | SARAH | The report says the depth of validation should be set by uncertainty, importance, complexity, novelty, supplier experience, decision consequence, interpretability, degree of autonomy and adaptiveness. | SUPPORTED | pp.10-11, Topic 4: "Risk factors discussed included: uncertainty, importance, complexity, novelty of the AI approach, supplier experience, model limitations, decision consequence, interpretability/explainability, degree of autonomy and adaptiveness. These factors should determine validation activities" |
| 59 | SARAH | A regulator asked where the independent test set comes from for a continuously retraining model. | SUPPORTED | p.11, Topic 4 Q&A: "During Q&A, regulators asked how independence can be maintained where a model continuously retrains on operational data." |
| 59 | SARAH | The presenters had said test data should stay independent of training data. | SUPPORTED | p.11, Topic 4 Industry Perspective: "The presenters highlighted that data used for validation/testing should remain independent from development and training data." |
| 59 | SARAH | For a continuously retraining model the answer was defined samples, test sets and procedures that preserve a means of independent verification, with change-control triggers deciding when more verification is needed. | SUPPORTED | p.11, Topic 4 Q&A: "The answer was to use lifecycle mechanisms, defined samples, test sets, procedures and controls to preserve a means of independent verification; where the model evolves, monitoring and change-control triggers should determine when additional verification or re-optimisation is required." |
| 59 | SARAH | A topic-four key message says independent verification remains a critical expectation, but implementation for continuously learning systems needs clarification. | SUPPORTED | p.11, Topic 4 Key Messages: "Independent verification evidence remains a critical expectation, but implementation for continuously learning systems needs clarification." |
| 60 | SARAH | Topic six is titled cybersecurity and outsourced activities. | SUPPORTED | p.2 TOC and p.12 heading: "1.6. Topic 6: Cybersecurity and Outsourced activities" |
| 60 | SARAH | The report says AI systems consist of data, model/software and hardware and therefore remain computerised systems. | SUPPORTED | p.12, Topic 6 Industry Perspective: "The presenter began from the premise that AI systems consist of data, model/software and hardware; therefore, from a GMP perspective they remain computerised systems." |
| 60 | SARAH | The report says Annex 11 and existing cybersecurity standards suffice at the level of principle. | SUPPORTED | p.12 Industry Perspective: "AI does not require a separate cybersecurity regime in Annex 22 if Annex 11 and applicable cybersecurity standards are properly applied."; p.13 Key Messages: "Annex 11 Chapter 7 was considered applicable and sufficient at a principle level." |
| 60 | SARAH | The report says the regulated user remains accountable even when a third party provides the model, platform or cloud, and that agreements must cover change-control participation, evidence access and transparency. | SUPPORTED | p.12, Topic 6 Industry Perspective: "the regulated pharmaceutical user remains accountable for the computerised/AI system even when a third party provides the model, platform or cloud infrastructure ... agreements must clarify responsibilities, configuration expectations, change control participation, evidence access, audit/certification arrangements and transparency of controls." |
| 60 | SARAH | The report cites existing Annex 11 principles, particularly Chapter 7 on outsourced activities. | SUPPORTED | p.12, Topic 6 Industry Perspective: "existing Annex 11 principles, particularly Chapter 7 on outsourced activities, already provide an appropriate framework for cybersecurity and supplier management" |
| 61 | SARAH | The conclusion says stakeholders articulated a largely aligned position across the six topics: a risk-based, technology-neutral approach in which acceptability is determined by intended use, the risks presented and the effectiveness of lifecycle controls, rather than by the underlying technology. | SUPPORTED | p.13, section 2 Conclusion: "a broad range of industry stakeholders were able to articulate a largely aligned position across the six discussion topics ... determined by its intended use, the risks it presents, and the effectiveness of the controls applied throughout its lifecycle rather than by the underlying technology itself." |
| 61 | SARAH | The last substantive sentence of the conclusion says that, subject to appropriate controls and a robust risk-based framework, there may be viable pathways for the use of adaptive, probabilistic and generative AI technologies in GMP-regulated applications. | SUPPORTED | p.14, final sentence of section 2 Conclusion (immediately before '3. Next Steps'): "The discussions suggest that subject to appropriate controls and a robust risk-based framework, there may be viable pathways for the use of adaptive, probabilistic and generative AI technologies in GMP-regulated applications while continuing to protect product quality and patient safety." |
| 63 | SARAH | The report says the drafting group has discussed changing the scope, that industry made a largely aligned case for changing it, and that there may be viable pathways. | SUPPORTED | p.3: "The drafting group has therefore discussed the scope of Annex 22 to allow dynamic or adaptive models, probabilistic models, and Generative AI/LLMs in GMP applications"; p.13: "largely aligned position"; p.14: "there may be viable pathways" |
| 64 | SARAH | The report does not say the scope of Annex 22 has changed. | SUPPORTED | p.3: "The draft version of Annex 22 currently states 'the document does not apply to Generative AI...'"; p.14: "inform the ongoing revision of Annex 22" |
| 64 | SARAH | The report does not say whether a broadened text would go back out for comment, and its next-steps section gives no date. | SUPPORTED | p.14, section 3 Next Steps: "Following the workshop, the drafting group will assess the key themes ... Where necessary, additional clarification and internal discussion may be undertaken" |
| 65 | HOST | The report's assessment dimensions are consequence, influence, uncertainty, complexity and detectability, while the July draft sorted by technology. | PARTIAL | p.7 Topic 5 Potential Impact: "evaluate decision consequence, model influence, uncertainty, complexity and detectability"; p.3 Introduction: "The draft version of Annex 22 currently states 'the document does not apply to Generative AI and Large Language Models (LLM), models with a probabilistic output, dynamic models'" |
| 67 | SARAH | The report says adaptiveness and autonomy increase the required depth of validation. | SUPPORTED | p.11, Topic 4 Key Messages: "Validation depth should be driven by intended use, uncertainty, importance, complexity, autonomy and adaptiveness." |
| 67 | SARAH | The report refers to change triggers and re-optimisation or retraining criteria as lifecycle controls for adaptive models. | PARTIAL | p.11 Topic 4 Q&A: "monitoring and change-control triggers should determine when additional verification or re-optimisation is required"; p.12 Potential Impact: "Definitions may need to include ... model drift, re-optimisation, change trigger" |
| 70 | HOST | The agentic deviation-management system is the second of the five use cases listed in the report. | SUPPORTED | p.9, Topic 2, second bullet of five: "• adaptive Heating, Ventilation and Air Conditioning (HVAC) control within validated environmental boundaries; • an agentic AI system supporting deviation management; • an LLM supporting GMP risk assessment" |
| 73 | SARAH | The report indicates the degree of human involvement may be relaxed as knowledge and experience increase. | SUPPORTED | p. 7: 'Over time, as knowledge and experience increase, the risk profile may shift, potentially allowing more autonomy and less human involvement.' |
| 73 | SARAH | The report says that if a vendor API does not expose outputs, rationale and logs, the system cannot be measured and therefore cannot be justified. | SUPPORTED | p.9, Topic 2 Q&A: "If a third-party model or Application Programming Interface (API) cannot provide necessary outputs, rationale, logs or evidence needed for control, the manufacturer cannot measure and justify the system adequately." |
| 73 | SARAH | The report says supplier agreements should give the regulated user visibility of changes when a third-party or foundation model is updated. | PARTIAL | p.12 Topic 6 Industry Perspective: "agreements must clarify responsibilities, configuration expectations, change control participation, evidence access"; p.13 Q&A: "where there is limited visibility of supplier changes, regulated users need to be part of change control and have visibility over control logic"; p.11: 'cloud/provider change visibility' |
| 75 | SARAH | The report repeatedly says controls cover named failure modes and that unnamed failure modes are found through deviations, which feed the next risk assessment. | SUPPORTED | p.9, Topic 2: "The presenter repeatedly emphasised that guardrails are specific controls for specific risks and that unanticipated failure modes cannot be assumed to be covered ... must be updated through lifecycle learning, deviations and defects. New knowledge should feed back into risk assessments"; p.10 Key Message: "Unanticipated failures require lifecycle learning and risk assessment updates." |
| 76 | HOST | The report names symbolic human-in-the-loop as what not to do. | SUPPORTED | p.8, Topic 3 Q&A: "the session supports Annex 22 language requiring meaningful, risk-based and documented oversight rather than default, symbolic or one-size-fits-all human-in-the-loop." |
| 77 | SARAH | The report's own words give the structure: a risk assessment describing scenarios, controls mapped to them, and evidence from verification and monitoring; and an AI qualification dossier of validation evidence, training-data information, prompting and output controls, and evidence the system works. | SUPPORTED | p.6, Topic 5 Q&A: "it should describe the risk scenarios, the controls mapped to those scenarios, and the evidence generated through verification and monitoring ... 'AI qualification dossier' ... containing validation evidence, training data information, guardrails around prompting and output, and evidence that the system works as expected." |
| 78 | HOST | The report says control mechanism, detectability and safe state may need defining. | SUPPORTED | p.10 Topic 2 Potential Impact: "Definitions may need to cover control mechanisms, unacceptable output, failure mode, escalation, fallback, safe state and monitoring threshold"; p.5 Topic 1 Potential Impact: "Definitions may need to clarify intended use, model influence, complexity, uncertainty, control strategy, detectability" |
| 81 | SARAH | The report is EMA's record of its 30 June 2026 expert workshop. | SUPPORTED | p.1 title page: "Multistakeholder workshop on expert contributions to artificial intelligence guidance development (Annex 22) 30 June 2026"; p.2 footer: "EMA/156789/2026"; p.3: "The drafting group therefore requested a meeting with AI experts" |
| 81 | SARAH | In the report the drafting group says it has discussed opening Annex 22's scope to adaptive, probabilistic and generative models under a fully documented, risk-based control strategy. | SUPPORTED | p.3, Introduction: "The drafting group has therefore discussed the scope of Annex 22 to allow dynamic or adaptive models, probabilistic models, and Generative AI/LLMs in GMP applications, provided they comply with the requirements of the annex and are supported by a fully documented, robust, risk-based control strategy." |
| 82 | SARAH | Industry's position in the report: acceptability by intended use, consequence, influence, uncertainty, complexity and detectability in Q9 language; no categorical prohibition but a documented do-not-use; oversight as a lifecycle objective with HITL as one pattern and a competence named for the reviewer; layered controls that prevent, detect and contain; the regulated user accountable for any vendor model, which must expose enough evidence to be measured. | SUPPORTED | p.6: "acceptability should be assessed case by case by considering decision consequence, model influence, uncertainty, complexity"; p.7: "'human oversight' as a broad lifecycle governance objective and 'human in the loop' as one specific implementation pattern"; p.9: "layered control: input/prevention ... in-model/detection ... output/containment"; pp.4-5 Topic 1 (ICH Q9(R1)); p.6 'do not use' documented; p.8 reviewer competence; p.12 Topic 6 accountability |
| 84 | SARAH | The report is fifteen pages and the drafting-group 'has therefore discussed the scope' sentence is on page 3. | SUPPORTED | p.3 footer and Introduction: "EMA/156789/2026 Page 3/15 ... The drafting group has therefore discussed the scope of Annex 22"; final footer reads "Page 15/15" |