Clinical Evaluation of AI-assisted Diagnostic Medical Device Software in China – A new NMPA guidance

The new NMPA guidance CMDE 2023 No. 38 outlines the agency’s expectations on the clinical evaluation of AI-assisted diagnostic medical device software in China. It includes recommendations on clinical trial design, study subjects, evaluation metrics, clinical reference, sample size and statistics for AI-assisted software devices. This blog post provides a short English summary of this guidance document.

On November 7, 2023, the Center for Medical Device Evaluation (CMDE) of the Chinese National Medical Product Administration (NMPA) released the guidance document “Guidelines for the Registration Review of the Clinical Evaluation of AI-assisted Diagnostic Medical Devices (Software)” (CMDE 2023 No. 38). The document is aimed at guiding manufacturers of AI-assisted diagnostic medical device software (MDSW), as well as the NMPA reviewers, for the preparation and review of this type of MDSW’s clinical evaluation.

Scope

The guidance focuses on AI-assisted MDSW for clinical decision support. This refers to MDSW, either standalone or built-in, that are based on AI algorithms and may include functions such as pattern recognition and data analysis. These MDSW, through methods such as identification, labeling, highlighting, etc., prompt physicians to focus on potential areas of abnormality/lesions, thereby assisting physicians in making corresponding diagnostic and treatment decisions. These MDSW may also include non-decision support functions such as report generation, before-and-after image comparison, segmentation of normal anatomical structures, dimension measurement, CT value measurement and non-clinical functions. 

 Note, the following types of AI-assisted MDSW are excluded from the scope of this guidance document:

• MDSW that identify malignancy, disease stage, or subtype
• MDSW that predict the probability of disease occurrence 
• MDSW that assist in detecting and distinguishing multiple lesions simultaneously 
• MDSW for triage and referral
• MDSW used in conjunction with IvD products

Nonetheless, manufacturers of these MDSW can use relevant principles outlined in this guidance as a reference. 

Key Takeaways

• Trial design
  •  Clinical trials of these MDSW shall focus on their diagnostic performance. In addition, their usability and safety can also be investigated. 
  •  As the clinical significance of these AI-assisted MDSW lies in improving the detection accuracy of physicians, controlled trials are typically needed. Depending on the product's characteristics and clinical practices, relevant trial designs include randomized parallel control, crossover self-control, or multiple-reader multiple-case (MRMC) trials.  

• Investigational subjects
  •  Imaging data from the intended population is typically used as the investigational subject of a trial. For clinical trials of MDSW for real-time imaging-based detection assistance, it is recommended to collect imaging data prospectively. 
  •  Imaging data should be independent from the data used for the device and its predecessor’s development (i.e. training and test sets used).
  •  Collect data with considerations of disease spectrum distribution, such as subtypes and stages.
  •  Gather comprehensive disease-related information when leveraging existing clinical data.
  •  Due to the variability of physicians’ performance and their interaction with patient variability and the AI, it is generally advisable to include physicians that the MDSW intends to assist as subjects in the trial. 
  •  For non-real-time imaging assistive products, MRMC design is advisable as it requires fewer samples.

• Evaluation metrics
  •  The selection of evaluation metrics should include product design features considerations. Generally, metrics such as sensitivity, specificity, receiver operating characteristic (ROC) curve or its derivatives are less affected by differences in disease prevalence, making them preferable. 
  •  Regardless of metric choice, clinical trials should consider overall effectiveness design, e.g., area-under-curve for ROC, superior sensitivity under non-inferiority specificity, or enhanced detection rates. 

• Clinical reference (ground truth)
  • Manufacturers should provide detailed information on the selection, construction methods, and rationale for clinical reference that serves as the ground truth. Available methods for constructing clinical references include clinical confirmation and expert panel judgment. The guidance provides detailed requirements for the construction of each type of reference. 

• Sample size estimation and statistical analysis
  •  Sample size estimation should consider clinical trial design, primary evaluation metrics, and statistical requirements. Manufacturers should provide information on calculation formulas, relevant parameters, justification, and the statistical software used.
  •  For sample size calculation of parallel controlled trials, the manufacturer shall refer to the NMPA guidance document “Guidelines for the Design of Clinical Trials for Medical Devices” (CMDE 2018 No. 6). 
  •  For MRMC trials, sample size calculation needs to take the planned statistical analysis method (e.g. Obuchowski-Rockette and etc.) into account. The guidance provides detailed explanations on this topic. 
  •  Include all patient and physician data in the statistical analysis. Besides point estimates, calculate 95% confidence intervals for sensitivity, specificity, and AUC. Compare with the control group for superiority/non-inferiority to assess clinical significance.

• Evaluating non-decision support functions
  •  The safety and effectiveness of non-decision support functions can be evaluated based on verification and validation data and/or data from clinical trials. 
  •  Verification and validation data of these functions can be collected through test set testing, stress testing, adversarial testing, or testing on a high-quality database, either individually or in combination. Manufacturers are advised to refer to the guidance document “Clinical Evaluation Technical Guidance Principles for Medical Devices” (CMDE 2021 No. 73) for the preparation and use of verification and validation data in clinical evaluation.
  •  If a clinical trial is used, these functions can be investigated as secondary outcomes based on clinically established reference standards or commonly used academic methods (e.g. dice similarity coefficient, fiducial registration error, and etc.).

• Additional issues to consider in clinical trial design
  •  Providing necessary training for physicians reading imaging data before trial initiation can effectively reduce bias.
  •  Adequate quality control is needed for image reading:
    •  Image readers should interpret trial images blinded to clinical information.
    •  Select image readers representing intended user qualifications and settings.
    •  Blind readers from diagnosis, reference standards, and clinical data of trial samples.
    •  Consider crossover reading design with washout periods between readings. 
    •  Order sample readings differently for each reader. 

• Information on clinical evaluation that shall be contained in IFU
  •  The IFU generally needs to include the following information on clinical evaluation:
    •  Clinical trial summary - basic information, metrics, results.
    •  Intended use - aided detection indications, imaging modalities, major functions, clinical role.
    •  Requirements on data acquisition in clinical use - devices, processes. 

• In addition, the following key details shall also be included:
  •  Clinical trial results and subgroups if applicable
  •  Indicated modalities and indications
  •  Other major functionalities of the device (e.g. image display, processing, measurement, and analysis)
  •  Clinical role of the product (cannot be solely used for clinical diagnosis and treatment decision-making).

Additionally, the guidance also provides a detailed analysis of the clinical evaluation strategy for 2 fictional MDSW examples (CT lung nodule detection and colonoscopic image-assisted polyp detection) to illustrate the outlined principles in practice.

It is important to note that CMDE emphasized that the guidance is formulated under the current framework of regulations and standards as well as the state of the art in MDSW technology. As regulations, standards, and technology continue to evolve, the relevant content in the guidance will be updated accordingly. Manufacturers shall keep a close watch on the future development of this guidance.

Please feel free to contact us if you have any questions about this guidance or questions in general about the regulatory assessment of MDSW for market entry in China. Qserve has a dedicated local team in China to assist you with Chinese regulatory, quality, and clinical matters.