BABL AI: Conducting third-party audits for automated employment decision tools

Case study from BABL AI.

From:
Centre for Data Ethics and Innovation and Department for Science, Innovation and Technology
Published
6 June 2023
Use case:
Natural language processing and generation, Machine learning, Deep learning
Sector:
Digital & Comms (SIC Code Section J), Professional, Scientific & Professional Activities (SIC Code Section M), Administrative & Support Services (SIC Code Section N)
Principle:
Fairness, Accountability and governance, Contestability and redress
Key function:
Human Resources
AI Assurance Technique:
Compliance audit, Certification, Risk Assessment, Bias Audit

Background & Description

This case study is focused on auditing AI systems used for making employment decisions in the human resources (HR) sector, with a primary focus on hiring and promotion decisions. The majority of these systems provide simplified scores that can be used to rank candidates for hiring or promotion and are used to inform human decisions.

These audits were developed specifically for New York City’s Local Law No. 144, which requires yearly bias audits. The results of these must be made publicly available.

BABL AI conducts independent, third-party, criteria-based audits and certifications for automated employment decision tools (AEDT), with a particular focus on disparate impact (bias), governance, and risk assessment. In the language of the CDEI Assurance Guidance these are simultaneously ‘Compliance Audits’, ‘Certifications’, and ‘Bias Audits.’

Before an audit takes place we walk the prospective auditee through a series of questions to determine if sufficient testing of tools has been conducted to qualify for an audit. In cases where more preparation is required, we provide guidance for the prospective auditee. Once these criteria are met, then the auditee is on-boarded onto a client portal for progression to the actual audit phase.

During the core phase of the audit the auditee submits documentary evidence to the client portal, which our trained and certified auditors review to satisfy audit criteria.

During the review process, BABL AI auditors might ask for more supporting documentation or interact with the auditee’s internal and external stakeholders, such as employees or other third-parties, to verify the truth of statements made in the submitted documentation. Within the scope of our engagement, we perform amongst others the following procedures:

  • Inspection of submitted documents and external documentation

  • Interviewing company employees to gain an understanding of the process for determining the disparate impact and risk assessment results

  • Observation of selected analytical procedures used in company’s bias testing

  • Inspection of the select samples of the bias testing data and results

  • Inquiry of personnel responsible for governance and oversight of the bias testing and risk assessment

At the end of the audit phase, the auditors reach an overall audit opinion that determines the result of the audit. This opinion can be a ‘Pass,’ ‘Minor Remediations,’ or ‘Fail’ result. In this final phase, BABL AI drafts a public report for each AEDT, if mandated by the regulatory body, and presents the final deliverable, including the audit opinion, to the auditee. The audit results are meant to be shared publicly (in the case of recent laws proposed/passed in the US), or to be shared with those procuring the AI system.

The criteria have been constructed to require ethical risk assessments, governance mechanisms, and bias assessments (done by the company), and all of our auditors have themselves conducted these assessments for other clients as they need to understand what best practice looks like.

How this technique applies to the AI White Paper Regulatory Principles

More information on the AI White Paper Regulatory Principles.

Fairness

The primary goal of the audits are to assess the potential for these systems to discriminate based on protected characteristics such as race/ethnicity and gender.

Accountability and Governance

The audits have several requirements for the organisation developing or deploying these automated tools to have accountability and governance in place. This includes a requirement for a party (either a group or an individual) that is accountable for managing risks related to bias, with corresponding duties, responsibilities, and power to affect change in the organisation.

Contestability and Redress

Summary results of these audits are made publicly available, which facilitates the ability of applicants to contest the results, seek redress, or seek alternative selection procedures if desired.

Why we took this approach

This technique has a long history of success in financial and cybersecurity auditing and assurance, it removes ambiguity in the results, facilitates transparency in the process (the criteria are public and published with the certification), reduces liability for the auditors, and is more cost and time efficient as compared to requiring auditors to manually conduct the risk and bias assessments.

Benefits to the organisation

We hold ourselves to the highest standards of integrity. Our auditors are ForHumanity Certified Auditors under NYC AEDT Bias Audit.

  • Our auditors follow ForHumanity’s Code of Ethics, PCAOB AS 1105 for audit evidence, and ISAE 3000 for assurance engagements (where applicable).

  • To improve transparency our audit criteria are publicly available and published as part of the required public summary of audit results. This shows what we test for and why.

  • The criteria-based process audit does not require integration into a company’s technical workflow. Our method simply asks that the organisation keep detailed and verifiable documentation of the development, testing, and/or use of their algorithms so that it may be verified and evaluated by our auditors.

Limitations of the approach

Despite our efforts to provide reasonable assurance, companies can still lie, hide problems, or modify testing results in ways we can’t detect. Our procedures are designed to avoid this, and are risk-based, but this problem still exists.

Further AI Assurance Information

Published 6 June 2023
Contents