On April 3, 2023, FDA released the draft guidance titled “Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning (AI/ML)-Enabled Device Software Functions” (hereinafter “draft guidance”) proposing an approach to support iterative improvement through modifications to a machine learning device software function (ML-DSF) while continuing to provide reasonable assurance of device safety and effectiveness. FDA is accepting comments on the draft guidance until July 3, 2023.
Background:
Historically, changes to an approved or cleared medical device that could significantly affect the safety or effectiveness of the device or that constitute a major change or modification in the intended use of the device required submission of a new 510(k). This framework is at odds with the use of artificial intelligence and machine learning in medical devices, which enable and encourage continuous improvements and modifications to the device based on information gathered during use. FDA has been engaged in the process of developing a regulatory approach tailored to AI/ML-enabled devices that allows for safe and rapid modifications in response to new data while ensuring safety and effectiveness.
The idea of using a “predetermined change control plan” (PCCP) was first mentioned by FDA in its 2019 discussion paper entitled “Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD)” and has been widely discussed in the industry. In FDA’s Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan published in January 2021, FDA committed to publishing PCCP guidance. In addition, Section 3308 of the Consolidated Appropriations Act, also referred to as the Food and Drug Omnibus Reform Act (FDORA), added section 515C to the Food, Drug and Cosmetic Act, granting FDA authority to approve a PCCP “submitted in an application . . . that describes planned changes that may be made to the device . . . if the device remains safe and effective without any change.” In addition to FDA’s discussion of AI/ML-enabled devices, the White House released a Blueprint for an AI Bill of Rights on October 4, 2022, which discusses principles regarding the design, use, and deployment of automated systems.
The draft guidance is applicable to ML-DSFs that the manufacturer “intends to modify over time” including those “for which modifications to the ML model are implemented automatically . . . as well as for ML-DSFs for which modifications to the ML model are implemented manually.”[1] The term “PCCP,” as used in the guidance, “refers to a plan that includes device modifications that would otherwise require a premarket approval supplement, De Novo submission, or a new premarket notification.”[2] Proactively pre-specifying and seeking market authorization for intended modifications to an ML-DSF in a PCCP removes the need to submit additional marketing submissions for each future modification that is delineated and implemented in accordance with the PCCP. Modifications to an ML-DSF that “could significantly affect, or that could affect, the safety or effectiveness of the device”[3] require premarket authorization, unless those modifications are covered by a PCCP. In addition, the modifications cannot change the intended use of the device or the benefit-risk profile.
Elements of a PCCP:
A PCCP is to be composed of three elements: (i) a Description of Modifications; (ii) a Modification Protocol; and (iii) an Impact Assessment.
1. Description of Modifications
A PCCP must contain a detailed description of the specific, planned device modifications. The description of modifications includes three sub-elements: (i) an enumerated list of individual proposed device modifications; (ii) a specific rationale for the change to each part of the ML-DSF; and (iii) reference to the labeling changes associated with each modification. FDA recommends that a PCCP include specific modifications that can be verified and validated. These modifications should be “presented at a level of detail that permits understanding of the specific modifications that will be made to the ML-DSF.”[4] The Description of Modifications should clearly indicate whether the planned modifications will be implemented automatically or manually. FDA recognizes this is an evolving area and is proposing to consider PCCPs for ML-DSFs where modifications are implemented automatically to the extent FDA can “properly review them for substantial equivalence to the predicate or a reasonable assurance of safety and effectiveness.”[5] FDA reiterates that this subset of ML-DSFs is complex and will consider the benefit-risk assessment when reviewing PCCPs with automatically implemented modifications.
Not all modifications may be appropriate for inclusion within a PCCP. Types of modifications that are acceptable for inclusion in a PCCP include:
- Modifications related to quantitative measures of ML-DSF performance specifications (e.g., improvements to analytical performance resulting from re-training the ML model using new data within the intended use population from the same type and range of input signal);
- Modifications related to device inputs to the ML-DSF (e.g., expanding the algorithm to include new sources of the same type of signal); and
- Limited modifications related to the device’s use and performance (e.g., for use within a specific subpopulation).
FDA expects that modifications included in a PCCP should maintain the device within both the device’s intended use and indications for use.
2. Modification Protocol
The Modification Protocol describes the methodology used to develop, validate, and implement those modifications in a way that ensures the continued safety and effectiveness of the devices across relevant patient populations. FDA states that the draft guidance is consistent with the principles in the Blueprint for an AI Bill of Rights. The Modification Protocol should have four primary content elements, each with their own requirements:
1. Data Management Practices – The Modification Protocol should include an outline of how data to support the proposed modifications will be collected, annotated, stored, retained, controlled, and used. The Modification Protocol should also clarify the relationship between the Modification Protocol data and data that is used to train and test both the initial and following versions of the ML-DSF. This is called the “data sequestration strategy” by FDA to ensure that training data sets and testing data sets are separated. Training data and testing data are sequestered to prevent overfitting and misquotes of test performance.
The Modification Protocol should also include a description of the control methods used to prevent data or performance information leaking into the modification development process. In essence, the Modification Protocol should provide information that describes the protocols put in place to prevent access to performance testing data during the training and tuning process.Examples of the types of information to be included in a Modification Protocol describing data management practices are provided in Appendix A of the draft guidance.
2. Re-training Practices – The Modification Protocol should include information about re-training practices, including the objective of re-training, a description of the ML model, the device components that may be modified, and triggers for re-training.
3. Performance Evaluation – The Modification Protocol should discuss how the performance of the ML-DSF will be evaluated, including the triggers for evaluation, performance metrics to be computed, the statistical analysis plans that will be used to test performance for each modification, and confirmation that failures will be recorded and screened from implementation.
4. Update Procedures – The Modification Protocol must include a description of how the devices will be updated to implement the modifications. This includes confirmation that either the verification and validation plans have not changed or that any changes are justified. Differences in performance and testing methods will be communicated to users. The PCCP should include a description of any labeling changes that will result, and the available labeling must include adequate directions for use and reflect information about the currently available versions of the ML-DSF, including information on site-specific modifications. Labeling should not reflect information on modifications that have not yet been implemented in the current version, as it could cause confusion and would be deemed misbranded. Examples of the types of information that should be submitted describing update procedures are also provided in Appendix A of the draft guidance.
The PCCP should also clearly delineate which parts of the Modification Protocol are applicable to each modification. For a PCCP with multiple modifications, this may be achieved through a traceability table. The draft guidance includes a sample traceability table that provides an example of how a manufacturer can depict the traceability between the Description of Modifications and the Modification Protocol.
3. Impact Assessment
The draft guidance refers to the Impact Assessment as an assessment of the benefits and risks of the planned modifications and risk mitigations. There are five elements of an Impact Assessment:
- Comparison of the modified device to the unmodified version of the device;
- Benefits-risk assessment for each individual modification;
- Continued assurance of the safety and effectiveness of the device;
- Impact of one modification on the implementation of another; and
- Collective impact of implementing all modifications.
Modifications to an authorized PCCP will generally constitute changes to the ML-DSF that require a new marketing submission for the device, including a modified PCCP. This is because modifications described in the PCCP include device changes that would typically require a PMA supplement, De Novo submission, or 510(k) premarket notification. FDA expects that the modified PCCP will need to be reviewed as part of the premarket review of the modified device because the modification will generally significantly affect the safety or effectiveness of the device. Where modifications to the PCCP are the only significant modifications since the prior device and PCCP authorization, FDA intends to focus review on the proposed PCCP.
The draft guidance includes a decision tree[6] for determining whether a modification is within the scope of an authorized PCCP or if a new submission is required:
Referencing a PCCP in a Submission:
FDA considers the PCCP to be part of the technological characteristics of the device. For devices subject to 510(k) requirements, the determination of substantial equivalence where the predicate device was authorized with a PCCP will be determined based on a comparison to the version of the predicate device cleared or approved prior to changes made under the PCCP.
Example Scenarios of ML-DSFs That Employ PCCPs:
Appendix B to the draft guidance provides examples that “illustrate different ML-DSF scenarios where a PCCP could be employed.”[7] For example, a particular ML-DSF analyzes images of skin lesions by identifying and characterizing features to aid in diagnosis. This ML-DSF was validated with a specific camera and is intended for use by a primary healthcare provider. The device was authorized with a PCCP. The pre-specified modification was for extension for use on additional general-purpose computing platforms, like smartphones and tablets. The general-purpose computing platform must include a 2D-camera that meets the minimum specifications in the PCCP, and the updated device must achieve a minimum performance defined in the Modification Protocol. In a scenario where the manufacturer implements a modification that was already specified in the PCCP and in conformance with the PCCP, the modification would not require a new marketing submission. On the other hand, if the manufacturer wanted to deploy a modified ML model that uses images captured by a thermographic camera, which was not specified in the PCCP, then a new marketing submission would be required because the modification could significantly affect the safety or effectiveness of the device and was not included in the PCCP.
Appendix B also provides examples for patient monitoring software, ventilator settings software, an image acquisition assistance device, and feeding tube placement radiograph analysis software, with various hypothetical modification scenarios for each.
Takeaways:
This draft guidance is a significant step forward in FDA’s ability to effectively regulate AI/ML-enabled medical devices. The PCCP approach has the potential to save manufacturers from continuous submissions and to reduce the uncertainty around what support is required before making changes to devices with AI/ML components. However, as the draft guidance details, there are extensive requirements that manufacturers will need to satisfy in order to utilize the PCCP process. Depending on feedback from stakeholders, the final version of the guidance may be different from the current draft version.
Device manufacturers developing ML-DSFs or incorporating them into their medical devices should consider submitting comments to the draft guidance. FDA recommends that the public submit comments to the draft guidance (Docket No. FDA-2022-D-2628) within 90 days of publication in the Federal Register, by July 3, 2023. FDA hosted a public webinar for medical device manufacturers and other stakeholders to discuss the draft guidance on April 13, 2023. Webinar materials, including the presentation slides and transcript, are available here.
Morrison Foerster continues to closely monitor developments regarding AI/ML and FDA’s regulation of medical devices. For questions or assistance, please contact the authors.
[1] Draft guidance at 5.
[2]Id. at 6.
[3]Id. (citations omitted).
[4]Id. at 16.
[5]Id.
[6]Id. at 15.
[7]Id. at 8.