AI Insights, December 2024

AI Governance – Step 5: Audit and reporting

Björn Preuß

Chief Data Scientist, 2021.AI

Step 1

Step 2

Step 3

Step 4

Step 5

Are you ready for an AI audit tomorrow?

Auditing and reporting for AI Governance

A transparent audit and reporting process is critical for ensuring accountability in AI Governance.7 Audits offer the opportunity to assess compliance with governance frameworks and identify areas for improvement, while regular reporting on AI usage, risk management, and compliance ensures that all stakeholders, including regulators, stay informed about the organization’s AI practices.13

Tracking issues and control breaches

The GRACE AI Platform supports this by tracking issues that arise throughout all workflows, such as those identified during AI Impact Assessments, model validation, and post-deployment monitoring (e.g., AI system and model alerts). It also provides robust capabilities to track and report on control breaches.

GRACE enables issue tracking at various levels, including model, business domain/function, and enterprise-wide, ensuring comprehensive oversight. It also captures resolution action and makes error traces available for any bugs or errors discovered during use. This logged information will allow a thorough review in an audit scenario.

Gaining insights into AI performance

With its extensive logging and reporting features, GRACE covers AI inventory, risks, AI system status, monitoring and control metrics, and issue/incident statistics. These statistics include key metrics such as outstanding issues by severity, time to closure, and trends over time. Granular and aggregate reporting is available across model/system, business domain/function, and enterprise levels, providing flexibility in how insights are presented.

Metrics can be collected from a wide range of development environments and AI systems, with the results and control data extractable to third-party risk management systems or BI dashboards. This fosters instant reporting or ad hoc analysis in an audit.

Key actions:

  • Establish audit protocols: Define audit processes that evaluate AI Governance practices, risk management efforts, and compliance with relevant regulations.
  • Prepare regulatory reports: Develop clear and comprehensive reports that demonstrate adherence to AI Governance frameworks, addressing key concerns like bias mitigation, explainability, and accountability.
  • Engage external auditors: Periodically involve third-party auditors to assess AI models, particularly high-risk ones. External validation helps enhance credibility and ensures compliance with both internal policies and external regulations.

Practice

After utilizing the LLMs in their portfolio for some time, it becomes possible to review the logs and compliance statuses of various systems over time. The immutable database provides insights into which controls failed and when, allowing for comparisons of performance across different systems. Additionally, it highlights any violations of controls and outlines the remedies implemented to address them. This comprehensive overview enables the client to maintain a usable audit trail for effective AI governance.

References

  • 7Wirtz, Bernd W., et al. “Governance of Artificial Intelligence: A Risk and Guideline-Based Integrative Framework.”
  • 13 Perry, Brandon, and Risto Uuk. “AI Governance and the Policymaking Process: Key Considerations for Reducing AI Risk.” Big Data and Cognitive Computing, vol. 3, no. 2, May 2019, pp. 26–26, doi:10.3390/bdcc3020026.

AI Governance guide:

Step 1: Defining your AI compliance landscape

Step 2: Register your AI systems and models

Step 3: Establish and manage AI risks and controls

Step 4: Monitor and validate AI models continuously

Step 5: Audit and reporting

Björn Preuß

Björn Preuß

CHIEF DATA SCIENTIST, 2021.AI

Björn is a Chief Data Scientist at 2021.AI. He is responsible for the Data Science function and aligns the business, product and data science needs in 2021.AI.

You might also like…

You might also like - The rise of AI

Series 1/3. The rise of Shadow AI and what it means for your organization

Shadow AI refers to the unauthorized use or implementation of AI systems and tools within an organization without the explicit approval…

READ MORE

You might like - Shadow AI: how to manage and control AI in your organization

Series 2/3. Shadow AI: how to manage and control AI in your organization

Today we are implementing both traditional AI models and Large Language Models (LLMs) across our organizations to assist us in solving…

READ MORE