B. Comprehensive Documentation for Marketing Submission:
Purpose:
To provide the FDA with a thorough understanding of your device’s design, development, validation, and performance.
Details:
Organize your submission with clear and detailed documentation in the following areas:
Quality System Documentation:
Consider if the information required in the marketing submission can be sourced from your Quality System documentation. This can include documentation related to design validation, design changes, and handling of nonconforming products.
Device Description:
- State that AI is used in the device.
- Clearly define the intended use of the AI-enabled device including AI model features, user qualification, and compatible input devices and acquisition protocols.
- Describe how the AI component contributes to achieving the intended use.
- Detail the device architecture, including hardware and software components.
- Describe device inputs (manual or automatic) and outputs, including compatible input devices and acquisition protocols.
- Include a use environment and clinical workflow description.
User Interface and Medical Device Labeling:
- Describe the user interface and how users will interact with the AI-enabled features.
- Provide clear and concise labeling that includes information on how the AI helps achieve the device’s intended use, model architecture, model inputs and outputs, performance metrics and their monitoring, known limitations, and the potential for the model to change over time.
- Include information on installation and maintenance instructions, especially regarding integration with data systems and ensuring input data compatibility. Define terms explicitly (e.g., “sex”).
- Describe any customizable features, how to configure them, when it’s appropriate, and how users can discern the current configuration.
- Explain any additional metrics or visualizations used to add context to the model output.
- For devices intended for patients or caregivers, provide labeling material at an appropriate reading level, describing instructions for use, indications, intended use, risks, and limitations. Explain how patients/caregivers will understand device use and output if specific materials aren’t provided.
Risk Assessment:
- Provide a “Risk Management File” including a risk management plan, a risk assessment and a risk management report.
- Consider risks across the Total Product Life Cycle (TPLC) in both normal and fault conditions, including those resulting from user error.
- Specifically consider risks related to understanding information necessary to use or interpret the device, including risks related to lack of information or unclear information.
- Explain risk controls, including user interface elements (e.g., labeling).
Data Management:
- Provide a clear explanation of data management practices (i.e., collection, processing, annotation, storage, control, and use).
- Characterize the data used in development and validation, which is critical for understanding how the device was developed and validated.
- Describe data sources, characteristics, and preprocessing steps.
- Explain how data quality and representativeness are ensured, including addressing potential biases.
- Document data annotation methodologies and expertise involved.
- Describe data governance, security, and privacy measures.
- Explain data provenance and audit trails.
Model Description and Development:
- Provide a comprehensive explanation of the AI model(s) used, including the algorithms, architecture, and training methodology.
- Document the rationale for model selection and any design choices made.
- Describe the model inputs and outputs, including a description of any pre- or post- processing, and/or data augmentation or synthesis, as applicable.
- Explain the model architecture, including layers and connectivity.
- Detail the training process, including data splitting, augmentation, and optimization.
- Describe the performance metrics used during development and their target values.
- Explain the handling of uncertainty in model outputs.
- Describe the use of ensemble methods, if applicable.
- Explain how thresholds (operating points) were determined.
- Explain any calibration of the model output.
Validation:
- Provide comprehensive validation data demonstrating the safety and effectiveness of the AI-enabled device for its intended use.
- Include details on the validation datasets, methodologies, and acceptance criteria.
- Address the performance of the device across relevant demographic groups to mitigate bias.
- Include a software version history, detailing model versions and differences between tested and released versions.
Performance Validation:
Provide objective evidence that the device performs predictably and reliably in the target population.
- Describe the validation study design, including the rationale for the chosen approach.
- Characterize the validation dataset, ensuring it is appropriate for the intended use population and accounts for potential data drift.
- Define performance metrics and acceptance criteria, justifying their relevance.
- Present results for the overall population and relevant subgroups (stratification).
- Include uncertainty and variability analyses.
- Discuss clinical significance of the findings.
Human Factors Validation (or Usability Evaluation):
Demonstrate users’ ability to interact with and understand the device safely and effectively.
- Describe the methodology used for human factors validation or usability testing.
- Characterize the user groups involved in the testing.
- Present the results of the testing, focusing on the effectiveness of risk controls related to information understanding.
Medical Device Monitoring:
- Describe data collection and analysis methods for identifying and assessing post-market changes in model performance and their impact on safety and effectiveness.
- If employing proactive performance monitoring as a risk control, include information regarding your performance monitoring plans.
- Explain how you will monitor potential causes of undesirable performance changes (e.g., changes in patient demographics, data drift, user behavior).
- Describe robust software lifecycle processes for monitoring in the deployment environment.
- Outline your plan for deploying updates, mitigations, and corrective actions.
- Describe procedures for communicating performance monitoring results and mitigations to users.
Cybersecurity:
- Include elements in the cybersecurity risk management report, threat modeling, cybersecurity risk assessment, labeling, and other deliverables specified in the “Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions” guidance.
- Explain how cybersecurity testing addresses risks associated with the model, including malformed input (fuzz) testing and penetration testing.
- Provide a Security Use Case View covering AI-enabled considerations.
- Describe controls for data vulnerability and preventing data leakage (access controls, encryption, anonymization or de-identification of sensitive data).
- Consider controls for data poisoning, overfitting, model bias, and performance drift attacks (data validation, anomaly detection, adversarial training, differential privacy, secure multi-party computation, watermarking, continuous monitoring).
Public Submission Summary:
Prepare a summary document outlining the device’s intended use, key technological aspects, and performance data.