AI in Biopharma Manufacturing: Are We There Yet?
This is the script the AI voices read, so it matches the audio word for word; times are from the render. Researched, scripted and voiced by AI systems under Jack Prior's direction. Sam and Sarah are AI characters; nothing in the episode is Jack speaking, and none of it is a statement of his views or his employer's. Generative AI can be confidently wrong — check the sources. Corrections: chat@jackprior.ai.
00:00 Narrator This is Are We There Yet — a podcast on the evolution of AI in biopharma manufacturing, directed by Jack Prior. A word about how it's made. This episode was researched, scripted and voiced by AI systems. Jack sets the questions and the frames; the AI reads the documents and writes the conversation you're about to hear, between two AI characters: Sam, who plays a manufacturing-science practitioner, and Sarah, who has read the documents. Nothing you hear is Jack speaking, and none of it is a statement of his views or his employer's. Like any generative AI output, it can be wrong — confidently wrong, or missing a nuance — which are exactly the risks this industry is working to mitigate, and exactly what this season is about. Check the sources before you rely on anything. Corrections are welcome at jackprior dot A I. Now, the episode.
00:53 Sam A validation lead, first week of October. For a year her position paper has had one sentence doing all the work: under draft Annex Twenty-two, the deviation assistant is non-critical, it drafts and a person decides, so the gate doesn't apply. Then a colleague forwards a fifteen-page PDF from the European Medicines Agency, and the third page says the drafting group has discussed allowing generative AI in GMP applications, critical or not, under a documented control strategy.
01:23 Sarah And her first reaction is relief.
01:26 Sam For about a minute. Then she reads the next twelve pages and realises the sentence that protected her has been replaced by a list: risk scenarios mapped to controls mapped to evidence, an oversight role with a stated competence, a vendor agreement that gives her the model's logs, a documented go or no-go. The gate was cheap to stand behind. The alternative is a dossier she hasn't started.
01:50 Sarah That's tonight's frame, Sam. Annex Twenty-two, revisited: is the model-type gate coming down? EMA has published the report of its expert workshop. What did industry tell the drafting group, what did the drafting group say it is now considering, and what does that change for our four systems?
02:07 Sam Season one is supposed to be over. This is the episode the rulebook made us add.
02:11 Sam Sarah, the question in one sentence, and the documents.
02:15 Sarah Has the only text in the corpus that sorts AI by model type started to move, and if so, toward what. One anchor: EMA's report of its multistakeholder workshop on Annex Twenty-two, document number one five six seven eight nine of twenty twenty-six, fifteen pages. The cover carries the workshop date, the thirtieth of June twenty twenty-six; EMA's event page lists the report as first published on the second of October. The voices: the July twenty twenty-five Annex Twenty-two draft, EMA's February twenty twenty-six notes of its meeting with industry, ICH Q9 revision one, FDA's January twenty twenty-five draft, and the Annex Eleven draft, for one cross-reference.
03:01 Sam Provenance and maturity.
03:03 Sarah Regulator layer: the Annex Twenty-two drafting group, which EMA's event page places in its GMP and GDP inspectors' working group, reporting on a meeting it requested. Maturity is the thing to be careful about. It is a final report of a workshop: not a revised annex, not a consultation, not guidance. Its closing page, headed next steps, commits the drafting group to one verb: the input will be carefully reviewed and considered. What makes it worth an episode is that, on the way to that modest promise, it says what the drafting group has been discussing.
03:39 Sam How this connects. Episode four read the draft's scope section and reported the gate: it applies to static and deterministic models; dynamic, probabilistic and generative models should not be used in critical GMP applications. Episode nine put adaptive models and generative agents in critical GMP at the top of the open-questions list. This report is the first official text in our corpus that moves either answer, and I want to be precise about how far.
04:10 Sarah Discussed, not drafted. That's the distance, and I'll keep coming back to it.
04:14 Sam And why now is why the season exists: the agent arrived in the plant between the draft and the workshop, and one of industry's own worked examples on the thirtieth of June was an agentic system supporting deviation management. Our fourth system, by name, in an EMA document.
04:30 Sarah Let me read the introduction, because everything else hangs off it. It first quotes the draft back at itself: the document does not apply to generative AI and large language models, models with a probabilistic output, dynamic models and models which adapt performance, and such models should not be used in critical GMP applications. Then: comments received during the public consultation indicated support for allowing manufacturers to use large language models and generative AI systems in GMP applications, whether critical or non-critical.
05:08 Sam Whether critical or non-critical. That's the clause that reorganises the position paper.
05:13 Sarah And then the sentence. The drafting group has therefore discussed the scope of Annex Twenty-two to allow dynamic or adaptive models, probabilistic models, and generative AI and LLMs in GMP applications, provided they comply with the requirements of the annex and are supported by a fully documented, robust, risk-based control strategy. The next paragraph says concerns remain about the elements of such a control strategy, and that is why they asked for a meeting with AI experts: to assess whether a risk-based approach can be applied to generative AI.
05:50 Sam So the structure of the day was a test. We're thinking of opening the door; show us the door can be controlled.
05:56 Sarah That is the report's own framing: experience-based input on control, governance and mitigation across the AI lifecycle, with emphasis on ICH Q9. The industry associations registered with EMA answered jointly, as one consolidated view, across six topics. For each, the report gives the drafting group's questions, the industry perspective, the Q and A, key messages, and a paragraph headed potential impact on Annex Twenty-two drafting, written in the conditional throughout: may be influenced, could require, definitions may need. The report takes the topics in the order one, five, three, two, four, six, and so will I.
06:40 Sarah Topic one, regulatory pathways for adaptive and probabilistic models. The drafting group asked how such models could be accommodated, what their validation paradigm would be, and how quality risk management, the ICH Q9 discipline of assessing, controlling, communicating and reviewing risk to quality, should apply across the lifecycle of a model in high-risk areas where deficiencies have low detectability and high patient impact.
07:08 Sam And industry's answer.
07:10 Sarah That adaptive, probabilistic and generative models should not be excluded from GMP applications on the basis of a technology label alone. The criterion should be whether the model can be shown fit for its intended use and whether its risks can be controlled within the existing quality system. The analogy was measurement: instruments drift, calibration intervals are set on a risk basis, lifecycle review confirms a system is still fit for use. The report adds that the analogy was not meant to equate AI adaptation with measurement drift, only to show GMP already reassesses suitability when behaviour changes.
07:49 Sam Here's where I'd expect a regulator to push back. A thermocouple drifts in one direction, slowly, and you can see it. A language model fails in ways that look like success.
08:00 Sarah The report records exactly that push. Regulators asked whether hallucination, fabrication, sycophancy and epistemic overconfidence are ordinary statistical uncertainty or a fundamentally different failure mode. Industry's answer: such behaviours must be identified and evaluated during development and validation, and if the manufacturer cannot demonstrate fitness for intended use in view of them, the use cannot be justified. No technical method for estimating hallucination rates was set; that was deferred.
08:32 Sam Deferred is honest. Nobody in that room had a validated hallucination rate for a deviation assistant.
08:38 Sarah The second thing topic one did was pick the vocabulary, and it picked ICH Q9 revision one. Q9 defines risk as the combination of the probability of occurrence of harm and the severity of that harm. Revision one, from twenty twenty-three, has a section on formality, and the sentence industry leaned on is this: the degree of rigor and formality of quality risk management should reflect available knowledge and be commensurate with the level of uncertainty, importance and complexity of the issue to be addressed.
09:10 Sam Formality, for someone who hasn't read Q9 since the first version.
09:13 Sarah How structured and how documented your risk work has to be; not a binary but a scale. More uncertainty, more importance, more complexity, more formality. Industry's position was that uncertainty should drive knowledge generation, stronger lifecycle controls, more formal validation and more monitoring, rather than an automatic prohibition.
09:37 Sam And detectability. That's the word I'd underline in the whole report.
09:41 Sarah The moderator raised it: in adaptive or generative AI, the inability to detect failure may itself be a central risk factor. The reply went back to Q9's text: risk analysis is about likelihood and severity, and then one sentence: in some risk management tools, the ability to detect the harm, detectability, also factors in the estimation of risk. For AI, the speaker said, detectability may be highly appropriate as a dimension, particularly where errors are difficult to observe. The session supported considering it without mandating a scoring method.
10:18 Sam So the test being proposed has five words in it: consequence, influence, uncertainty, complexity, detectability. Episode three's grading was consequence times influence. This adds three more axes, and the third is the one that catches the agent.
10:34 Sam Model influence. That's FDA's word, from the January twenty twenty-five draft, in an EMA inspectors' report.
10:41 Sarah It is. FDA's draft pairs model influence with decision consequence, and the report uses both terms in that sense without citing FDA. The potential-impact paragraph says definitions may need to clarify intended use, model influence, complexity, uncertainty, control strategy, detectability and continued fitness for use.
11:06 Sarah Topic five, strategic risk and compliance limits, and the drafting group's hardest questions. Where do the limits of risk-based mitigation lie: are there classes of critical decisions where no combination of guardrails and oversight would be sufficient? And are there cases where the guardrail effort becomes so disproportionate that the system simply should not be used in certain critical functions?
11:29 Sam That's the drafting group asking industry to draw the line for them.
11:33 Sarah And industry declined to draw it by category. The position was strongly against categorical prohibitions: no class of decisions should be considered unacceptable solely by technology category or criticality label. Acceptability should be assessed case by case on decision consequence, model influence, uncertainty, complexity and the ability to reduce residual risk. But the concession is in the same breath: where the mitigation effort is heavy, impractical or insufficient, the use is not justified at that time and in that context, and that conclusion should be documented.
12:11 Sam A valid outcome is do not use.
12:14 Sarah That is a key message, in almost those words: a valid quality risk management outcome may still be do not use where uncertainty cannot be controlled. The report also records the question every inspector will ask: how does a manufacturer demonstrate that layered controls do provide reliable output for a critical decision? The answer: a risk assessment that describes the risk scenarios, the controls mapped to those scenarios, and the evidence from verification and monitoring; then monitoring history, responsiveness to signals and a documented state of control become the inspection evidence.
12:54 Sam Scenarios to controls to evidence. That's a document structure, and I've not seen it in a GMP text before.
13:01 Sarah Two more things. Regulators asked how a responsible person can take responsibility for a model that is not fully transparent; the reply was that AI does not remove predicate-rule responsibilities, and accountable persons may rely on qualified experts, as they already do for complex technologies. And an industry speaker introduced the phrase AI qualification dossier: validation evidence, training-data information, guardrails around prompting and output, and evidence that the system works as expected.
13:35 Sam A marker, a third of the way through. The drafting group says it has discussed opening the scope. Industry's price is a five-axis assessment in Q9 language, no categorical bans, a documented do-not-use when the controls can't carry it, and an inspectable dossier that maps scenarios to controls to evidence. Nothing has been written into the annex yet.
13:58 Sarah Topic three, human oversight and accountability. The drafting group asked what level and form of human-in-the-loop oversight is still required when guardrails are implemented. Industry's answer started by splitting the question's own term in two, and this is the distinction I think will outlast the workshop. Human oversight is the broad lifecycle governance objective: can a qualified, accountable human understand, challenge, intervene, override, reject or escalate. Human-in-the-loop is one specific implementation pattern: the process waits for a human action before a GMP-relevant workflow proceeds.
14:40 Sam Episode seven spent twenty minutes on the loop and never had that sentence. The draft's clause three point three defines human-in-the-loop as a model giving input to a decision made by a human operator. This says that's one way to get oversight, not the definition of it.
14:56 Sarah And the key messages make the consequence explicit. Human-in-the-loop is one possible control, not a default requirement for all AI uses. Oversight must be proportionate to risk, autonomy, uncertainty, complexity and decision impact. And the one with teeth: oversight should be evidenced and periodically reviewed, and it cannot compensate for inadequate validation. The session supports annex language requiring meaningful, risk-based and documented oversight rather than default, symbolic or one-size-fits-all human-in-the-loop.
15:35 Sam Symbolic human-in-the-loop is exactly what the non-critical defence had become. A person clicks approve on a draft they didn't have time to read. The report is saying that doesn't count, and I agree with it.
15:46 Sam Now the question I parked in episode seven. Which competence does the human supply? A bounds check is a rule the system should run. What's left for the person that couldn't be coded?
15:56 Sarah The report gives the first stated answer I've seen in any of these documents, on page eight. A human reviewer does not need to understand every internal detail of the model, but must understand the relevant model principles, the process, the output and the decision context sufficiently to challenge the AI. And the regulators' questions in return are yours from episode seven: how do you monitor the performance of the human doing the oversight; how do you manage automation bias; how is oversight evidenced on inspection.
16:31 Sam So the competence is now named as a combination, process plus enough model to argue with it, and the regulator's follow-up is: prove the person is still doing it six months in.
16:43 Sarah The suggested evidence was training, qualification, knowledge testing, audit trails, recording of AI outputs and human modifications, and monitoring of human-AI team performance. And one speaker's point worth keeping: oversight alone does not correct risk; it provides knowledge, detection and triggers for action, so it should be judged by whether it leads to effective follow-up.
17:08 Sarah Topic two, technical reliability and incident response, the most technical discussion of the day: can available guardrail mechanisms reliably prevent, detect or contain hallucinations, incorrect recommendations or fabricated data, and what ensures escalation when they fail. Industry answered with five use cases: adaptive heating, ventilation and air conditioning control within validated boundaries; an agentic AI system supporting deviation management; a large language model supporting a GMP risk assessment for an electronic batch record system; text-to-SQL generation for annual product review data; and model predictive control for bioreactor feeding, a controller that uses a process model to plan the next moves rather than reacting to the last error. Some were pilots or theoretical frameworks rather than production systems.
18:03 Sam Two of our four, near enough. The feeding controller is a soft sensor with a steering wheel, and the deviation agent is our agent.
18:10 Sarah The central model was layered control, in three layers: input or prevention controls, in-model or detection controls, and output or containment controls, the Swiss cheese image, where each slice has holes but the holes don't line up. At the input: context-of-use verification and out-of-distribution detection, which checks whether the input looks like anything the model was trained on. In the model: retrieval-augmented generation with source and citation binding, so the model answers from retrieved documents and shows which; conformal prediction and uncertainty quantification, which attach a calibrated confidence to each output. At the output: hard constraints, confidence gates, cumulative-sum and moving-average monitoring, human gates, and fallback to validated states.
19:04 Sam Half of that output list is statistical process control from the nineteen-fifties: control charts on a model's outputs is the MVDA batch model watching the agent. I like that more than I expected to.
19:16 Sarah The presenter's caveat is the sentence to carry out of this session: guardrails are specific controls for specific risks, and unanticipated failure modes cannot be assumed to be covered by controls designed for known ones. Risk assessments are limited by available knowledge and must be updated through lifecycle learning, deviations and defects.
19:39 Sam And the word guardrail itself.
19:42 Sarah Several participants questioned whether it should appear in the annex at all. Industry suggested control mechanisms for the main text, because guardrails are technology-specific and transient, and the discussion agreed on the split: principles and objectives, prevention, detection, containment, in the annex, and methods and use cases in appendices, a Q and A, or another updateable format. Definitions, the report says, may need to cover control mechanisms, unacceptable output, failure mode, escalation, fallback, safe state and monitoring threshold.
20:20 Sam Safe state. That's a controls-engineering word in a GMP annex. And the model you can't buy the logs for?
20:27 Sarah The presenter said the key is not ownership of the model but access to sufficient evidence to measure how it works: if a third-party model or API cannot provide the outputs, rationale, logs or evidence needed for control, the manufacturer cannot measure and justify the system. On fallback: the HVAC and bioreactor examples revert to a pre-validated classical state, and where a fully model-operated process has none, sometimes accepting stoppage may be the contingency.
21:00 Sam Second marker. Oversight is the objective and human-in-the-loop is one pattern for it, with a competence finally named. Controls come in three layers, each tied to a named failure mode. A vendor model you can't see inside isn't a model you can control. Still no text.
21:18 Sarah Topic four, validation and lifecycle management, which industry framed as the integrating topic. The answer to how guardrails stay effective as a model evolves was a lifecycle decision framework rather than a validation event, with the depth of validation set by uncertainty, importance, complexity, novelty, supplier experience, decision consequence, interpretability, degree of autonomy and adaptiveness.
21:48 Sam And the question I'd have asked: where does the independent test set come from when the model retrains on last night's batches?
21:54 Sarah A regulator asked it. The presenters had said test data should stay independent of training data; for a continuously retraining model the answer was defined samples, test sets and procedures that preserve a means of independent verification, with change-control triggers deciding when more verification is needed. The key message admits it: independent verification remains a critical expectation, but implementation for continuously learning systems needs clarification.
22:27 Sarah Topic six, cybersecurity and outsourced activities. AI systems are data, software and hardware, so they remain computerised systems; Annex Eleven and the existing cybersecurity standards suffice at principle level; and the regulated user remains accountable even when a third party provides the model, platform or cloud, so agreements must cover change-control participation, evidence access and transparency. One cross-reference to check: the report cites Annex Eleven, particularly Chapter Seven on outsourced activities. In the July twenty twenty-five Annex Eleven draft, chapter seven is Supplier and Service Management; Outsourced Activities is Chapter Seven of Part One of the GMP guide, which isn't in our corpus, so take that as my reading. A site quoting the report should check which it means.
23:20 Sarah The conclusion. Stakeholders articulated a largely aligned position across the six topics: a risk-based, technology-neutral approach in which acceptability is determined by intended use, the risks presented and the effectiveness of lifecycle controls, rather than by the underlying technology. And the last substantive sentence: subject to appropriate controls and a robust risk-based framework, there may be viable pathways for the use of adaptive, probabilistic and generative AI technologies in GMP-regulated applications.
23:55 Sam So the gate is coming down.
23:57 Sarah No, Sam. Say what the record says. The draft's lines twelve to twenty-five still read as they did in July twenty twenty-five: dynamic, probabilistic and generative models should not be used in critical GMP applications, and generative models used in non-critical applications need a human-in-the-loop to ensure the outputs are suitable. The report says the drafting group has discussed changing that scope, that industry made a largely aligned case for changing it, and that there may be viable pathways.
24:30 Sarah It does not say the scope has changed. No revised text has been published. The February minutes expected the final by the end of this year, and the inspectors' work plan targets a final text to the Commission in the fourth quarter; whether a broadened text goes back out for comment first, the report doesn't say, and its next-steps page gives no date.
24:51 Sam Fair. Then my opinion, labelled. The direction has changed, and direction is what a site plans against. The July draft sorted by technology; everything in this report sorts by consequence, influence, uncertainty, complexity and detectability. Consequence and influence are the vocabulary of the FDA draft, of ICH M15 and of the BioPhorum guidance; the other three are ICH Q9's; none of it is the vocabulary of the draft's scope section. And the report reads to me like a drafting group that knows what it wants to write and is checking whether industry can live with the evidence burden.
25:27 Sam Four systems, then, through the test the report describes rather than the gate the draft draws. Sarah, the soft sensor first, the one that retrains itself nightly. In episode four it should not be used in critical use, full stop.
25:40 Sarah Under the report's logic it becomes a candidate, and a demanding one: adaptiveness and autonomy raise validation depth; it needs change triggers, re-optimisation criteria, and the thing the report admits needs clarification, an independent test set it cannot take from its own retraining stream. Its fallback is last night's frozen model or the classical controller. If a site can't preserve independent verification, the report's own answer applies: do not use, documented.
26:13 Sam The MVDA batch model and the fill-volume vision system.
26:17 Sarah Both static, deterministic, in scope yesterday and tomorrow. What changes is the language expected around them: detectability is now a named consideration, which is where a multivariate monitor is strong, and the three control layers map onto what a well-built inspection system already does, rule-based limits at the input, a confidence score in the model, hard constraints and a human gate at the output, with release sampling as the orthogonal control from episode two. The vocabulary is new; the architecture isn't.
26:51 Sam And the agent. Our fourth, and the report's second use case.
26:54 Sarah Take it layer by layer. Prevention: the records it reads are verified as within its context of use, and anything outside the defined deviation types is refused before the model sees it. Detection: it drafts only from retrieved, cited records, so every statement points at a document, and its uncertainty is quantified and logged. Containment: no root cause asserted without a cited record; a confidence gate below which the draft is not shown; monitoring of edit and rejection rates over time; and a human gate, which the report would now ask you to design rather than assume.
27:35 Sam Design, not assume. Say what that means for the person.
27:38 Sarah An investigator who understands the process, the deviation system, what the draft is and what decision it feeds, enough to challenge it; training and knowledge testing recorded; every human modification audit-trailed; the human-AI team's performance monitored. High involvement at first, relaxed only as knowledge and experience justify it. Then the vendor question: does the API expose outputs, rationale and logs, and does the agreement give change visibility when the foundation model is updated? If not, the system cannot be measured, and therefore cannot be justified. And then the documented go or no-go, which may be no.
28:17 Sam So the agent's non-critical defence is replaced by something harder and better: not it doesn't count, but here is how we know it's right, and here is who catches it when it isn't.
28:28 Sarah That's the report's direction, with the caveat it keeps repeating: controls cover the failure modes you named, and the ones you didn't name are found through deviations, which feed the next risk assessment.
28:41 Sam So what for a site. Stop treating the non-critical line as the whole strategy. This report makes it weak in both directions: if the gate stays, symbolic human-in-the-loop is named as the thing not to do; if the gate goes, the line stops mattering and the dossier matters instead. The dossier is the same package either way, so build it.
29:01 Sarah The report gives the structure. The risk assessment that describes the scenarios, the controls mapped to them, and the evidence from verification and monitoring. The AI qualification dossier: validation evidence, training-data information, prompting and output controls, evidence the system works. Those are the report's words, not mine, and they are what an inspector who has read it would expect to be shown.
29:27 Sam Three things this quarter. One, pick one AI use, probably the agent, and write the scenario-to-control-to-evidence map on a single page; if it can't be written, that's the finding. Two, put a control-mechanism inventory into the vendor agreement: what outputs, logs and rationale the supplier exposes, and how we hear about model updates. Three, write the oversight role down with a competence and a way to monitor it. And the habit from episode nine: the report says control mechanism, detectability and safe state may need defining. Whoever shows up with worked examples shapes the definitions.
30:06 Sam My second opinion, labelled: this is the industry submission that will be remembered as the one that moved the annex, if it moves, and the price quoted for it is an inspectable evidence package most sites haven't started. The basement we keep talking about is in that dossier under the heading training-data information, and the agent is what exposes whether it's dry.
30:26 Sam Three things. Sarah.
30:28 Sarah First, what the report is. EMA's record of its thirtieth of June expert workshop, first published on the second of October per EMA's event page, in which the drafting group says it has discussed opening Annex Twenty-two's scope to adaptive, probabilistic and generative models under a fully documented, risk-based control strategy. Discussed, not drafted; no revised draft exists.
30:54 Sarah Second, what industry put on the table: acceptability by intended use, consequence, influence, uncertainty, complexity and detectability, in Q9 language, with no categorical prohibition but a documented do-not-use; oversight as a lifecycle objective with human-in-the-loop as one pattern and a competence named for the reviewer; layered controls that prevent, detect and contain; and the regulated user accountable for any vendor model, which must expose enough to be measured.
31:25 Sam Third, mine. The gate hasn't come down, but the ground under it has shifted from technology to control strategy, and the dossier that strategy needs is the same either way. Build it for the agent first. Watch for a revised Annex Twenty-two, for whether it goes back to consultation, and for the definitions.
31:45 Sarah And check the report yourself. It is fifteen pages, and the sentence that matters is on page three.
31:51 Sam That's the update. Season one asked nine questions of a rulebook being rewritten; this is the first page of the rewrite showing through. Thank you for walking with us. Check the sources. Next season still asks the question under this one: is the data there yet.