AI Governance – Step 5: Audit and reporting

Are you ready for an AI audit tomorrow?

Auditing and reporting for AI Governance

A transparent audit and reporting process is critical for ensuring accountability in AI Governance.⁷ Audits offer the opportunity to assess compliance with governance frameworks and identify areas for improvement, while regular reporting on AI usage, risk management, and compliance ensures that all stakeholders, including regulators, stay informed about the organization’s AI practices.¹³

Tracking issues and control breaches

The GRACE AI Platform supports this by tracking issues that arise throughout all workflows, such as those identified during AI Impact Assessments, model validation, and post-deployment monitoring (e.g., AI system and model alerts). It also provides robust capabilities to track and report on control breaches.

GRACE enables issue tracking at various levels, including model, business domain/function, and enterprise-wide, ensuring comprehensive oversight. It also captures resolution action and makes error traces available for any bugs or errors discovered during use. This logged information will allow a thorough review in an audit scenario.

Gaining insights into AI performance

With its extensive logging and reporting features, GRACE covers AI inventory, risks, AI system status, monitoring and control metrics, and issue/incident statistics. These statistics include key metrics such as outstanding issues by severity, time to closure, and trends over time. Granular and aggregate reporting is available across model/system, business domain/function, and enterprise levels, providing flexibility in how insights are presented.

Metrics can be collected from a wide range of development environments and AI systems, with the results and control data extractable to third-party risk management systems or BI dashboards. This fosters instant reporting or ad hoc analysis in an audit.

Key actions:

• Establish audit protocols: Define audit processes that evaluate AI Governance practices, risk management efforts, and compliance with relevant regulations.
• Prepare regulatory reports: Develop clear and comprehensive reports that demonstrate adherence to AI Governance frameworks, addressing key concerns like bias mitigation, explainability, and accountability.
• Engage external auditors: Periodically involve third-party auditors to assess AI models, particularly high-risk ones. External validation helps enhance credibility and ensures compliance with both internal policies and external regulations.

Practice

After utilizing the LLMs in their portfolio for some time, it becomes possible to review the logs and compliance statuses of various systems over time. The immutable database provides insights into which controls failed and when, allowing for comparisons of performance across different systems. Additionally, it highlights any violations of controls and outlines the remedies implemented to address them. This comprehensive overview enables the client to maintain a usable audit trail for effective AI governance.

References

• ⁷Wirtz, Bernd W., et al. “Governance of Artificial Intelligence: A Risk and Guideline-Based Integrative Framework.”
• ¹³ Perry, Brandon, and Risto Uuk. “AI Governance and the Policymaking Process: Key Considerations for Reducing AI Risk.” Big Data and Cognitive Computing, vol. 3, no. 2, May 2019, pp. 26–26, doi:10.3390/bdcc3020026.