Advai: Regulatory Aligned AI Deployment Framework 

Case study from Advai.

From:
Centre for Data Ethics and Innovation and Department for Science, Innovation and Technology
Published
12 December 2023
Use case:
Natural language processing and generation, Image recognition and video processing, Machine learning, Deep learning
Sector:
Manufacturing (SIC Code Section C), Energy & Utilities (SIC Code Sections D & E), Financial and Insurance (SIC Code Section K), Professional, Scientific & Professional Activities (SIC Code Section M), Public Administration & Defence (SIC Code Section O), Other Services (SIC Code Section S)
Principle:
Safety, security and robustness, Appropriate transparency and explainability, Fairness, Accountability and governance, Contestability and redress
Key function:
R&D, Risk management, Strategy and corporate finance
AI Assurance Technique:
Data assurance, Risk Assessment, Impact Assessment, Impact Evaluation
Assurance Technique Approach:
Technical, Procedural, Educational

Background & Description

The adoption of AI systems in high-risk domains demands adherence to stringent regulatory frameworks to ensure safety, transparency, and accountability. This use case focuses on deploying AI in a manner that not only meets performance metrics but also aligns with regulatory risks, robustness, and societal impact considerations.

System interfaces are built around task APIs that can be used to generate benchmarks, run evaluations and create metrics. The metrics are selected and compiled to deliberately align with regulatory principles relevant to the use case. Practically speaking, the service is deployed as a microservice architecture in the cloud or on-premise dependent on the user requirement. For data scientists, a lab interface enables users to easily interact with technical assessments and fine-tune required metrics. For business users, a dashboard presents the data to the user at an appropriate granularity.

By integrating MLOps practices with a regulatory framework, we aim to address the pain points of various stakeholders such as data scientists, risk/compliance officers, C-suite executives, and the public, ensuring responsible innovation and deployment of AI systems.

How this technique applies to the AI White Paper Regulatory Principles

More information on the AI White Paper Regulatory Principles.

Safety, Security & Robustness

We ensure the end-to-end safety and security of the AI system by incorporating MLOps practices that focus on the full development cycle. For example, this includes checking any changes under real-world conditions, using metrics for drift detection and endpoint monitoring.

Appropriate Transparency & Explainability

We enable stakeholders to understand the regulatory implications of different parts of the AI’s processes using Data Cards and Model Cards at appropriate points on the MLOps development cycle.

Fairness

Regular scans for bias and the implementation of mitigation strategies ensure that the AI system is fair and does not discriminate against any individual or group.

Accountability & Governance

We ensure that there is clear accountability and governance structures in place by tracking KPIs and metrics throughout the AI lifecycle and maintaining extensive documentation.

Contestability and Redress

Mechanisms for feedback and redress are integrated into the framework, allowing for contestability of decisions surrounding system development and ensuring that individuals can seek remedies when necessary.

Why we took this approach

This approach is taken to overcome the disconnect between technical AI development and the broader needs of stakeholders. We are future-proofing AI deployment against emerging legal and ethical standards by aligning MLOps with underlying regulatory principles and current regulatory requirements, enabling innovation without compromising on accountability, compliance or public trust.

Benefits to the organisation using the technique

  • Ensures greater compliance with international regulatory standards, reducing legal risks. It does this by filtering, interpreting and matching compliance requirements appropriate to the industry and use case.

  • Fosters trust among end-users and the public by demonstrating a commitment to ethical AI deployment.

  • Enhances the resilience of AI systems against adversarial attacks and real-world uncertainties.

  • Connects the broader agenda of regulatory compliance with the more granular details along the machine learning ideation-development-deployment lifecycle.

Limitations of the approach

  • Interpretation of Regulatory Requirements: The principles set forth by regulations can be subject to interpretation. Different stakeholders might have varying interpretations of how to meet these standards, leading to potential misalignment and inconsistencies in implementation.

  • Dynamic Regulatory Environments: Different regions have different regulatory requirements which can change rapidly. An approach effective in one region needs adjusting to the relevant regulatory environment.

  • The efficacy of the feedback mechanisms designed to fine-tune the approach depends on user willingness and ability to use them.

  • It’s a substantial task to extract the relevant regulatory requirements for a given use case and industry.

  • This is still fundamentally a subjective approach with no metric or evidence that the correct regulations are selected by the organisation, although we advise based on prior comparable experience.

Aligning MLOps with Regulatory Principles

Further AI Assurance Information

Published 12 December 2023
Contents