MEASURE-2.4—The functionality and behavior of the AI system and its components – as identified in the MAP function – are monitored when in production.

>Control Description

The functionality and behavior of the AI system and its components – as identified in the MAP function – are monitored when in production.

>About

AI systems may encounter new issues and risks while in production as the environment evolves over time. This effect, often referred to as “drift”, means AI systems no longer meet the assumptions and limitations of the original design. Regular monitoring allows AI Actors to monitor the functionality and behavior of the AI system and its components – as identified in the MAP function - and enhance the speed and efficacy of necessary system interventions.

>Suggested Actions

Monitor and document how metrics and performance indicators observed in production differ from the same metrics collected during pre-deployment testing. When differences are observed, consider error propagation and feedback loop risks.
Utilize hypothesis testing or human domain expertise to measure monitored distribution differences in new input or output data relative to test environments
Monitor for anomalies using approaches such as control limits, confidence intervals, integrity constraints and ML algorithms. When anomalies are observed, consider error propagation and feedback loop risks.
Verify alerts are in place for when distributions in new input data or generated predictions observed in production differ from pre-deployment test outcomes, or when anomalies are detected.
Assess the accuracy and quality of generated outputs against new collected ground-truth information as it becomes available.
Utilize human review to track processing of unexpected data and reliability of generated outputs; warn system users when outputs may be unreliable. Verify that human overseers responsible for these processes have clearly defined responsibilities and training for specified tasks.
Collect uses cases from the operational environment for system testing and monitoring activities in accordance with organizational policies and regulatory or disciplinary requirements (e.g. informed consent, institutional review board approval, human research protections),

>Documentation Guidance

Organizations can document the following

To what extent is the output of each component appropriate for the operational context?
What justifications, if any, has the entity provided for the assumptions, boundaries, and limitations of the AI system?
How will the appropriate performance metrics, such as accuracy, of the AI be monitored after the AI is deployed?
As time passes and conditions change, is the training data still representative of the operational environment?

AI Transparency Resources

GAO-21-519SP - Artificial Intelligence: An Accountability Framework for Federal Agencies & Other Entities.
Artificial Intelligence Ethics Framework For The Intelligence Community.

>References

Luca Piano, Fabio Garcea, Valentina Gatteschi, Fabrizio Lamberti, and Lia Morra. “Detecting Drift in Deep Learning: A Methodology Primer.” IT Professional 24, no. 5 (2022): 53–60.

Larysa Visengeriyeva, et al. “Awesome MLOps.“ GitHub.