AI and GxP: keeping AI-assisted development controllable

“May we use AI in a GxP environment?” cannot be answered as posed – rather like “may we use software?”. What matters is where the model acts: while building the system, or inside the running system. The two cases have almost nothing in common.

1. Two cases, two answers

Conflating these two cases is why discussions about “AI and GxP” so often end in debates of principle. Taken separately, both are manageable – the first one with quite modest effort.

2. Case 1: AI as a development tool

A coding agent produces code or configuration; what ships and is validated is the result. From a regulatory viewpoint that is unremarkable at first: the frameworks require controlled processes, defined responsibility and solid evidence – not a particular way of producing code.

The comparison helps: a compiler, a code generator or a library is not validated individually either. What gets assessed is what they produce, and whether the process around them is capable of detecting defects.

3. What has to be governed

Four points are enough in most organisations and can be phrased as an addition to existing SOPs:

1. Approved tools and operating model. Which services may be used, and in which form of operation?
2. Data flows. What content may go in? GxP-relevant data, patient data and production data are the critical category here – and are rarely needed in development anyway.
3. Review duties. Identical to hand-written code, plus the points that come up more often with generated code: authorisation logic, error paths, new dependencies.
4. Documentation. That AI was used, who reviewed, and against what the review was performed.

A more detailed template is described in the deep dive AI coding governance.

4. Review duties and evidence

The instruments stay the same – review, test, traceability, change control. What is worth doing is a slight shift of attention:

• Derive test cases from the requirement, not from the generated code. Otherwise the test confirms the behaviour instead of the requirement.
• Check authorisation and error paths explicitly; they are the most common gap.
• Check new dependencies against a curated source.
• Limit the size of change sets so that review can actually happen.
• Labelling of substantial AI use on the change set, so the depth of review can be assessed later.

5. Case 2: AI as a function in the system

As soon as a model produces results at runtime on which GxP-relevant decisions rest, a fundamentally different question arises. Among the things to settle:

• The intended use of the model – what exactly is it used for, and what is explicitly excluded?
• Provenance and representativeness of the training or reference data,
• Model versioning – which version ran when, and how is a change assessed?
• Performance limits and behaviour outside the intended input space,
• Human oversight – who reviews results, and how is that documented?
• Ongoing monitoring in operation, not just a check at introduction.

On top of that come possible obligations under the AI Act, whose applicability depends on the specific use case and has to be assessed individually. This case deserves a separate assessment and is not the subject of this article.

6. Determinism and model versions

A common misconception: “AI is not deterministic, so it cannot be validated.” That is a serious challenge for case 2 – and no issue at all for case 1. In the tool case it is not the generation that has to be reproducible but the artefact. The same source state yields the same build, regardless of how the source came about.

In practice: reproducible builds, pinned dependencies and versioned configuration remain the foundation in the AI era – and they are good practice regardless.

7. Grey areas and open questions

• AI in test creation. Permissible, but: when test and implementation come from the same context, the evidential value drops. For high-risk functions, test cases should be created independently.
• AI in documentation. Harmless for drafts, critical for evidence documents – there a person carries responsibility for the content.
• AI in data analysis. As soon as results feed into GxP decisions you are in case 2, even if the tool “only” assists.
• Assistant features in third-party systems. Tools sometimes enable AI features without being asked; that belongs in supplier assessment and periodic review.

Common mistakes

• Merging both cases into one policy – the result is either too strict or too lax.
• A blanket ban that removes the use only from the documentation.
• A special process for AI code that replaces the regular checks instead of using them.
• Feeding production data as test data into external tools.
• Adopting vendor compliance claims without relating them to your own intended use.

Checklist: AI in a GxP environment

Conclusion

AI in a GxP environment is not a special case as long as it stays a development tool. Then a brief addition to the existing rules and a slight shift of review focus are enough. The effort is a few pages of policy, not a new quality system.

It only becomes demanding once a model influences decisions in the running system. Drawing that line cleanly – and having it in writing – is the single most important measure.