A regulatory perspective of Artificial Intelligence in Medical Devices

Regulatory frameworks in the US and EU are in essence not designed to handle these so-called adaptive AI algorithms out in the field. FDA recognized this issue and proposed a regulatory framework for adaptive AI based Software as a medical device (Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) - Discussion Paper and Request for Feedback [1]). In September 2020 COCIR published an analysis on AI in Medical Device (COCIR, Artificial Intelligence in EU medical device legislation September 2020 [2]).

In this blog we will try to clarify our understanding of what is meant by Artificial Intelligence (AI)  by limiting the definition in this context to self-learning software. So, the scope of this blog is software which can adapt itself based on training datasets with classical machine learning or more advanced deep learning algorithms and is then locked and released on the market.

This contrasts with the COCIR and FDA papers in which also simple look up tables were included in the scope of their papers as being AI algorithms.

In this blog we use the following definitions:

• Discrete improvement: adaptive algorithm is retrained by the manufacturer with more data and tweaks the software where needed to improve or be fitted to a new application (intended use!) based on new data and after V&V it is released.
• Continuous learning: adaptive algorithm learns and improves by itself. The algorithm changes based on new inputs it gets during use.

How to handle change?

From a regulatory perspective dealing with change, either in performance or even of the intended use is quintessential for AI devices. The essence Interesting to see that both papers recommend to use of a so-called Algorithm Change Protocol (ACP) to tackle change. In the COCIR paper the content of the ACP is not discussed. The FDA paper includes the following description:

ACP: specific methods that a manufacturer has in place to achieve and appropriately control the risks of the anticipated types of modifications delineated specifications. The ACP is a step-by-step delineation of the data and procedures to be followed so that the modification achieves its goals, and the device remains safe and effective after the modification. The ACP contains the elements on "how" the algorithm will learn and change while remaining safe and effective.

The proposal of the FDA with respect to. the ACP is, roughly, that if the change of the AI algorithm is within the boundaries agreed with FDA, no FDA premarket review is needed. Alas, it is a proposal.

In the FDA discussion paper also a SaMD pre-specifications document (SPS) is mentioned to be combined with an ACP. The content of the SPS is defined as the following: (COCIR does not mention this document at all)

SaMD manufacturer’s anticipated modifications to “performance” or “inputs or changes related to the “intended use” of AI/ML-based SaMD. These are the types of changes the manufacturer plans to achieve when the SaMD is in use. The SPS draws a “region of potential changes” around the initial specifications and labeling of the original device. This is "what" the manufacturer intends the algorithm to become as it learns.

With the combination of these two documents, the SPS and the ACP, the FDA hopes to control the potential changes of adaptive AI algorithms to be limited and that the software remains save during the changes it will undergo automatically during its lifetime.

In our opinion these documents should be implemented if it will be approved in the market. However, from a practical point of view, we think continuous learning software might lead to too many situations which were not anticipated in the SPS or ACP and then the benefit of these documents is taken away by the time needed to review the algorithm again. Therefore, for practical reasons, we recommend making use of discrete improvement of the algorithm because then the manufacturer remains 100% in control of your released software. Another advantage of the discrete improvements over the continuous learning mode is that the changes can be released to all devices and that all devices have the same performance. Two devices which improve by continuous learning and are used in two different surroundings will end up have different performances based on inequal datasets.

Also, the findings of recent article published by MIT technology review (“The way we train AI is fundamentally flawed” – Will Douglas Heaven Nov. 18 2020 [3]) support using discrete algorithms over continuous learning. In that article it is discussed that the models, even though they have the same training set, may differ from each other due to different starting values of the different nodes in Neural networks. So, when the algorithm is trained on additional clinical data, it might happen that performance does not increase or even deteriorates. Differences in pictures which are evident for humans might not be so evident for AI models after all. This suggests that it is advisable to have a human check (verification and validation) prior to every release of a software update.

When such regulatory pathways for AI devices become active, and continuous learning would be allowed, there are some aspects which the manufacturer should consider: The “what” the software can become, should be restricted for new functionalities. For instance, the detection of another disease with the approved algorithm should be prohibited as it should be verified that the software can output reliable results for the new disease. Also, when the system is bumped up an “significance of information class”, like defined by the IMDRF guidance document (Software as a Medical Device": Possible Framework for Risk Categorization and Corresponding Considerations[IMDRF/SaMD WG/N12 Final 2014 [4]) from drive clinical management to diagnosing by itself should be regarded as significant change or even a new medical device, which should be part of a V&V check and an official approval by NB/FDA. To set clear limits to the SPS one possibility could be to define that just e.g. accuracy, sensitivity or specificity are automatically adaptable characteristics of an algorithm and part of the SPS.

In the COCIR paper it is recommended that manufacturers of an AI device that will continuously change by learning, add an ACP to the technical documentation of their device for evaluation during conformity assessment. COCIR argues that changes that are in scope of the ACP are not significant changes, and as such a device that has been put into service can adapt its algorithm, i.e. change its output, based on the ACP.

We see a restriction here. As said, the COCIR paper does not specify the content of the ACP, but the paper reads “…As long as the device operates within the related change boundaries, its safety and performance are guaranteed”. From a practical point of view, we think that changes of an AI Algorithm within the change boundaries are only interesting when performance is improved. Performance of a device can only be guaranteed after Verification and Validation (V&V) testing. So, in practice adaption (through learning) of an AI Algorithm can go in discrete steps (included V&V) before it is put into service again.

The COCIR paper does not elaborate on the content of the ACP, but COCIR recommends the following: “COCIR recommends updating IEC 62304 by requiring among others that manufacturer define an Algorithm Change Protocol (ACP) for AI based devices that change through learning during runtime.

IEC 62304 defines the set of processes, activities, and tasks to establish a common framework for medical device software life cycle processes. IEC 62304 defines the following processes.:

• Software development
• Software maintenance
• Software risk management
• Software configuration management
• Software problem resolution

  In our attempt to find pointers where the ACP could be integrated or fitted in these processes (the standard), we have difficulty, because adaption of an AI Algorithm through learning is not related to any of these processes, i.e. (clinical) performance is not an attribute of IEC 62304.

  The Proposed Regulatory Framework of FDA gives some more details of what an ACP can be. The FDA defines the ACP as “how the algorithm will learn and change while remaining safe and effective”. Summarized the ACP should have the following components.

• Data Management
• Re-training
• Performance Evaluation
• Update Procedure

In this FDA definition during the Performance Evaluation the clinicians can be taken into the loop. Software V&V is part of the Update Procedure.

Conclusion

Currently, as a manufacturer, it is possible to bring medical devices with AI capabilities to the market. As far as we have seen these are all algorithm with the AI algorithm locked. We also see that manufactures are unsecure and are asking for a regulatory framework on what is allowed and what not with AI and ML. To bring flexibility into the EU regulations (MDR, IVDR) we think manufactures of adaptive AI algorithms would be served by a guidance paper on AI publish Medical Device Coordination Group (MDCG). Their position on how to deal with adaptive AI Algorithms through continuous learning is of great interest. ISO 14155:2020 (Clinical investigation of medical devices for human subjects — Good clinical practice) refers in its scope to the IMDRF guidance document, Software as a Medical Device (SaMD): Clinical Evaluation [IMDRF/ SaMD WG/ N41 FINAL: 2017 [5]], to demonstrate analytical validity and clinical performance. Also, the FDA Proposed Regulatory Framework uses IMDRF guidance documents as building blocks. MDCG could also adhere to the IMDRF guidance documents in compiling the necessary guidance for NB’s and industry.

Just give us a call or drop a message to see how Qserve can support you in these times.