Holistic AI: Open Source Library
Case study from Holistic AI.
Background & Description
The Holistic AI Open Source Library is a tool used to assess and improve the trustworthiness of AI systems. The Library is provided as a user-friendly Python module that can be downloaded and installed and is supported by documentation that details each method contained in the Jupyter notebooks which guide the user through example implementations. Holistic AI has also provided a number of tutorials for using the Library on our blog.
It serves as a resource for identifying and mitigating risks associated with AI systems, reducing preventable harms and supporting AI assurance efforts.
Currently, the Library has functions for data visualisation, bias metrics, and bias mitigation for both classification and regression models. Specifically, the library includes metrics for assessing bias for binary and multiclass classification, regression, clustering, and recommender systems, and provides pre-processing, in-processing, and post-processing approaches to bias mitigation through built-in functions.
The Library will continue to evolve, with new techniques to measure and mitigate bias added, as well as capabilities to address additional technical risk verticals such as explainability and robustness.
How this technique applies to the AI White Paper Regulatory Principles
More information on the AI White Paper Regulatory Principles.
Fairness
The Holistic AI Open Source Library supports a number of bias metrics and mitigation approaches derived from research in the field. These can be used to assess models for bias and mitigate them, with performance metrics providing a way to compare models and mitigations to find the fairest model while maximising accuracy.
Why we took this approach
Although there are other bias metric libraries already available, they are often not user-friendly and can be complicated. The Holistic AI Open Source Library is easy to use and is supplemented by several easy-to-follow tutorials using real-life datasets. It is important that bias mitigation tools are as widely available and accessible as possible, and the Holistic AI library is there to fulfil this need.
Benefits to the organisation using the technique
Bias can easily be assessed and mitigated in one easy-to-use dashboard. AI is increasingly being targeted by global regulation and is being used in critical contexts every day. Identifying and mitigating bias early can prevent harm before it occurs, reducing legal liability, reputational risk, and financial damage
Limitations of the approach
The library currently focuses on bias, but will soon be expanded to address other technical risk verticals.
Further Links (including relevant standards)
- Open source library webpage
- Holistic AI Open Source Library Documentation
- Holistic AI Open Source Library GitHub Repository
- Build Your Own Bias Measuring and Mitigation Dashboard in 5 Steps
- Using Python to Mitigate Bias and Discrimination in Machine Learning Models
- Bias Mitigation Strategies and Techniques for Classification Tasks
- Holistic AI Library Tutorial: Fairness Analysis for Binary Classification with Python
- Holistic AI Library Tutorial: Bias Measuring and Mitigation in Regression Tasks
- Using Python to Mitigate Bias and Discrimination in Machine Learning Models
- How to Create Interactive Visualisations in Colab with Holistic AI and Plotly
- Debiasing Artificial Neural Networks with Holistic AI and PyTorch
- Visualising Bias Metrics: Insights from Holistic AI’s Open-Source Library