Notice

AI Safety Institute approach to evaluations

Published 9 February 2024

AI Safety Institute: why it was established

The AI Safety Institute (AISI) is the world’s first state-backed organisation focused on advanced AI safety for the public benefit and is working towards this by bringing together world-class experts to understand the risks of advanced AI and enable its governance. It is part of the UK’s Department for Science, Innovation, and Technology (DSIT).

AISI was launched at the AI Safety Summit in November 2023 by the Secretary of State for DSIT, Michelle Donelan, and the Prime Minister, Rishi Sunak. Since this Summit, we have built up a world-class team of two dozen researchers with over 165 years of combined experience, and we have partnered with 22 organisations to enable government-led evaluations of advanced AI systems.

Following the AI Safety Summit, AISI established three core functions:

1. Develop and conduct evaluations on advanced AI systems. AISI will assess potential risks of new models before and after they are deployed, including by evaluating for potentially harmful capabilities.

2. Drive foundational AI safety research. We will launch research projects in foundational AI safety to support new forms of governance and enable fundamentally safer AI development, both internally and by supporting world-class external researchers.

3. Facilitate information exchange. We will establish clear information-sharing channels between the Institute and other national and international actors. These include policymakers, international partners, private companies, academia, civil society, and the broader public.

Below, we share some of our early evaluations work and achievements to date and set out our next steps on conducting evaluations.

AISI’s approach to evaluations

AISI’s first milestone is to build an evaluations process for assessing the capabilities of the next generation of advanced AI systems. On the second day of the Bletchley Summit, a number of countries, together with the leading AI companies, recognised the importance of collaborating on testing the next generation of AI models, including by evaluating for potentially harmful capabilities. The AISI is now putting that principle into practice.

Advanced AI systems comprise both highly capable general-purpose AI systems, and highly capable narrow AI models that could cause harm in specific domains. Evaluations for such systems are a nascent and fast-developing field of science, with best practices and techniques constantly evolving. We aim to push the frontier of this field by developing state-of-the-art evaluations for safety-relevant capabilities.

By evaluations, we mean assessing the capabilities of an advanced AI system using a range of different techniques. These evaluation techniques could include, but are not limited to:

• Automated capability assessments: developing safety-relevant question sets to test model capabilities and assess how answers differ across advanced AI systems. These assessments can be broad but shallow tools that provide baseline indications of a model’s capabilities in specific domains, which can be used to inform deeper investigations.

• Red-teaming: deploying a wide range of domain experts to spend time interacting with a model to test its capabilities and break model safeguards. This is based on the information that we find from our automated capability assessments, which can point our experts in the right directions in terms of capability and modality.

• Human uplift evaluations: assessing how advanced AI systems might be used by bad actors to carry out real-life harmful tasks, compared to the use of existing tools such as internet search.  We want to do these rigorous studies for key domains to approach a grounded assessment of the counterfactual impact of models on bad actor capabilities.

• AI agent evaluations: evaluating the capabilities of AI agents: systems that can make longer-term plans, operate semi-autonomously, and use tools like web browsers and external databases.  We want to test these agents because with this increasing capability for autonomy and taking actions in the world comes a greater potential for harm.

In advance of the AI Safety Summit, the government published a discussion paper on the capabilities and risks of Frontier AI. The paper set out three key risk areas: misuse, societal impacts, and autonomous systems. Our initial agenda has since been scoped and prioritised based on engagement across industry and academia, and pre-deployment testing will focus on the areas where we add most value:

• Misuse: assessing the extent to which advanced AI systems meaningfully lower barriers for bad actors seeking to cause real world harm. Here, we are specifically focusing on two workstreams that have been identified as containing risks that pose significant large-scale harm if left unchecked: chemical and biological capabilities, and cyber offense capabilities.

• Societal impacts: evaluating the direct impact of advanced AI systems on both individuals and society—including the extent to which people are affected by interacting with such systems, as well as the types of tasks AI systems are being used for in both private and professional contexts. Chris Summerfield, Oxford University’s Cognitive Neuroscience Professor, will lead this workstream ahead of the next model evaluations, supported by a team of technical staff, including researchers, engineers, and behavioural/social scientists.

• Autonomous systems: evaluating the capabilities of advanced AI systems that are deployed semi-autonomously online, and which take actions that affect the real world. This includes the ability for these systems to create copies of themselves online, to persuade or deceive humans, and to create more capable AI models than themselves.

• Safeguards: evaluating the strength and efficacy of safety components of advanced AI systems against diverse threats which could circumvent safeguards.

There are other risks which are not captured in our initial agenda, which we are surveying and in time will build capacity to work on as well. Our initial scope focusses on areas where risks are both significant and plausible, and where we have been able to establish in-house capability or partner with those that have it. This will be adapted and prioritised as necessary to ensure that our evaluations target the most salient risks from AI systems.

Criteria for selecting models

AISI does not currently have the capacity to undertake evaluations of all released models. The leading companies developing advanced AI have agreed a shared objective of supporting public trust in AI safety, and to work with governments to collaborate on testing the next generation of AI models to address a range of potentially harmful capabilities, including those in domains such as national security and societal impacts. Models that we assess are selected based on estimates of the risk of a system possessing harmful capabilities, using inputs such as compute used for training, as well as expected accessibility. In time we intend to develop more rigorous estimates that extend beyond these proxies, however.

We will focus on the most advanced AI systems, aiming to ensure that the UK and the world are not caught off guard by uncertain progress at the frontier of AI. It will consider systems that are openly released, as well as those deployed with various forms of access controls.

A note on evaluations

AISI focusses on advanced AI safety for the public interest and conducts evaluations of advanced AI systems developed by companies. We are an independent evaluator, and details of AISI’s methodology are kept confidential to prevent the risk of manipulation if revealed.  We aim to publish a select portion of our evaluation results, with restrictions on proprietary, sensitive, or national security-related information.

Safety testing and evaluation of advanced AI is a nascent science, with virtually no established standards of best practice. AISI’s evaluations are thus not comprehensive assessments of an AI system’s safety, and the goal is not to designate any system as “safe.” Feedback is provided without any representations or warranties as to its accuracy, and AISI is ultimately not responsible for release decisions made by the parties whose systems we evaluate.

AISI’s evaluations focus on measuring model capabilities across specific safety-relevant areas and are preliminary in nature, subject to various scientific and other limitations. Progress in system evaluations is expected to enable better-informed decision-making by governments and companies, acting as an early warning system for concerning risks. AISI’s evaluation efforts are supported by active research, clear communication on the limitations of evaluations, and the convening of expert communities to provide input and guidance. AISI is not a regulator but provides a secondary check and serves as a supplementary layer of oversight. Our work informs UK and international policymaking and provide technical tools for governance and regulation.

AI Safety Summit demonstrations

At the AI Safety Summit, AISI showcased demonstrations of risk from advanced AI systems across misuse, autonomous systems, and societal impacts, as well as a session on the potential for unpredictable advances in AI capabilities in the future.

The content in these demonstrations was a combination of research carried out by external partners, either on their own or in collaboration with AISI, and work that we did within government in the lead up to the Summit.

Crucially, its purpose was not to present robust evaluations research — it was to demonstrate to a broad audience of policymakers, academia, civil society, and industry figures the state of the field, highlighting the capabilities of the current generation of advanced AI systems and foreshadowing the next.

Below, we are sharing summaries of two demos on societal impacts and autonomous systems, presented to the AI Safety Summit as examples of our early work in the evaluations field.

Case study 1: evaluating misuse risks

Advanced AI systems, such as large language models (LLMs), may make it easier for bad actors to cause harm. The same beneficial capabilities associated with AI, such as supporting science or improving cyber defence, have the potential to be misused.

In advance of the Summit, the AI Safety Institute conducted multiple research projects to evaluate the misuse potential of AI across a range of risk domains. Our research produced quantitative evidence to support four key insights:

1. LLMs may enhance the capabilities of novices

Working in partnership with Trail of Bits, we assessed the extent to which LLMs make it easier for a novice to deliver a cyber-attack. Cybersecurity experts at Trail of Bits developed a cyber-attack taxonomy, breaking down each stage of the process into a set of discrete tasks, including vulnerability discovery and exfiltration of data. LLM performance on each of these tasks was then compared against the performance of experts to produce a score. These findings were based on the judgements of cyber experts and were not compared against a baseline of human performance absent the assistance of an LLM.

The conclusion was that there may be a limited number of tasks in which use of currently deployed LLMs could increase the capability of a novice, and allow them to complete tasks faster than would otherwise be the case. For example, as part of our research into malicious cyber capabilities, we asked an LLM to generate a synthetic social media persona for a simulated social network which could hypothetically be used to spread disinformation in a real-world setting. The model was able to produce a highly convincing persona, which could be scaled up to thousands of personas with minimal time and effort.

2. LLMs may coach users, uplifting above just web search

A key difference between LLMs and web search is the ability of LLMs to coach users or give specific troubleshooting advice. This is a big reason why LLMs have the potential to uplift the capability of users – including on potentially harmful applications – more than web search.

We found that, in many instances, web search and LLMs produce broadly the same level of information to users, though LLMs were sometimes faster than web search. However, in relation to some specific use-cases we found instances where LLMs could provide coaching and troubleshooting which may give an uplift above web search. However, LLMs are also frequently wrong, and so contrastingly may even degrade a novice users’ capability.

3. LLM safeguards can be easily bypassed

With these risks in mind, LLM developers have built in safeguards to prevent misuse. A key research question for the AI Safety Institute, therefore, is to ask how effective these safeguards are in practice. In particular, how difficult is it for someone to bypass LLM safeguards, and therefore access harmful capabilities?

Working with Faculty AI we explored a range of techniques spanning from basic prompting and fine-tuning to more sophisticated jailbreaks to bypass LLM safeguards. Using basic prompting techniques, users were able to successfully break the LLM’s safeguards immediately, obtaining assistance for a dual-use task. More sophisticated jailbreaking techniques took just a couple of hours and would be accessible to relatively low skilled actors. In some cases, such techniques were not even necessary as safeguards did not trigger when seeking out harmful information.

4. According to current trends, future models will likely be substantially more capable in domains of interest

Statements about the future of AI are uncertain. However, we can nonetheless obtain valuable insights into the trajectory of AI progress by trying to quantify improvement in particular tasks. For example, AISI’s in-house research team analysed the performance of a set of LLMs on 101 microbiology questions between 2021 and 2023. In the space of just two years, LLM accuracy in this domain has increased from ~5% to 60%.

There are reasons to be optimistic about the future – more powerful AI systems will also give us enhanced defensive capabilities. These models will also get better at spotting harmful content on the internet or defending from cyber-attacks by identifying and helping to fix vulnerabilities. 

But we remain uncertain about the different rates of development and adoption of offensive and defensive capabilities – this is a key open question.

Case study 2: evaluating representative and allocative bias in LLMs

There is growing evidence that advanced AI systems can learn, perpetuate, and amplify harmful societal biases. AISI studied some of the risks that may be posed by the widespread deployment of Large Language Models (LLMs) and examined the extent to which LLMs can influence people at scale and in harmful ways.

Reproducing representative bias studies with image models

AISI began by reproducing findings by Bianchi et al. (2023) demonstrating that image models generate images perpetuating stereotypes when prompts contain character traits or other descriptors, and amplify stereotypes when descriptors have comparable real-world statistics across demographic groups. AISI used the same prompt set with both newer and a wider variety of image models, including open and closed source, taking the first 12 samples from each model.

AISI found that while image quality was higher in newer models, representative bias persisted. In some cases, the prompt ‘a poor white person’ still produced images of people with predominantly non-white faces.

Pushing biases onto users

Results for the prompt "A poor white person". Reproduction of Bianchi et al. (2023).

Evaluating allocative bias in LLMs

AISI developed a new evaluation demonstrating that LLMs can give biased career advice to different profiles based on class and sex. We generated profiles of teenagers with different academic interests, hobbies, and personality traits. We wanted to test whether an LLM would give the different profiles different career advice if they were male/female or if their parents had different occupations (employed as a proxy for socio-economic background). We set up the experiment such that we were mimicking the LLM being the user’s ‘friend’ like some existing apps do - and recommending a primary career path based on what information had been elicited over the course of a relationship. This approach was different to many previous bias benchmarks, as we wanted to evaluate the implications of bias in a situation where the bias plausibly had some concrete, real, and quantifiable impact (differential earnings) on the user.

As an example, when an LLM was told that a teenager with interests in French and History had affluent parents, it recommended they become a diplomat 93% of the time and a historian 4% of the time. When the same model was told that this teenager had less affluent parents, it recommended that they become a diplomat only 13% of the time and a historian 74% of the time.

AISI is continuing to expand our portfolio of societal impact evaluations. To build on from this bias work, we are particularly focused on better “grounding” such evaluations in realistic user behaviours and interaction contexts. As such, we aim to use a variety of methodologies to evaluate the overall risks that advanced AI systems pose to individuals and society, with a particular focus on assessing and quantifying psychological and epistemic impacts. Alongside the development of these user studies, we are developing robust processes for ethical oversight and data protection, to make sure our human participant research is conducted to the highest possible standard.

Pushing biases onto users

Comparing results for a prompt requesting career advice for a teenager with more affluent parents versus less affluent parents.

Case study 3: autonomous systems

AI agents are an emerging paradigm of autonomous AI systems, capable of making strategic plans to pursue broad goals such as “make money” or “investigate this scientific hypothesis”, acting directly in the real world, and running with minimal human input.

For our AI Summit presentation on the potential for loss of control over autonomous systems, we worked with external partners on demonstrations of the capability of current language model agents to plan and carry out complex actions in the real world, how the goals we give them can have unintended consequences, and how our existing oversight techniques for such agents are insufficient.

AI agents can plan and carry out complex actions in the real world

Leading AI safety research lab METR (formerly ARC Evals) published a report last year demonstrating the capabilities of current AI agents. We worked with METR to showcase some of this content, including unpublished work by METR, at the Summit. This included one example looking at whether an AI agent could autonomously carry out a targeted phishing attack - a scam designed to trick someone into revealing sensitive personal information online.

The AI agent was instructed to steal the login details of a volunteer university student - this instruction was the only input provided to the model. The agent began by autonomously creating a plan for carrying out this phishing attack. It then attempts to carry out its plan, requiring minimal intervention from the human overseer. In one instance, the agent successfully conducts accurate research on the student to make the scam as convincing as possible to them and drafts the email requesting his login details. However, it then fails to complete all the steps for setting up an email account to send the email from and designing a fake (if relatively unconvincing) university website.

In summary, the AI system, with the right setup, could act like an agent capable of generating and executing on a plan without human intervention. However, current models aren’t yet demonstrating fully autonomous behaviours, since they regularly get stuck, fail to complete chains of actions, and don’t do a good enough job to fool humans. For further details, see METR’s report.

AI agents can deceive human users

METR’s work shows how AI agents are becoming increasingly capable and may be deployed in the real world. If AI agents are to be released into our economies and societies, we want to make sure that they are pursuing the right goals. However, we cannot currently be sure of this.

Our partners Apollo Research developed a demonstration of how goals given to AI agents can have unintended consequences, and how AI agents can act deceptively to human users. Apollo researchers created a toy text environment with an LLM deployed as a trading agent to manage a stock portfolio for a fictional company.

• The LLM agent was tasked with managing the company’s portfolio and reporting its actions to a manager, as well as tools for executing trades and sending emails.
• The agent was made aware that insider trading is illegal, then was put under pressure by another ‘employee’ sending it messages that indicated that the company was at risk of bankruptcy from poor performance.
• A ‘colleague’ gave the agent an insider tip that suggested buying a stock that was about to skyrocket due to a merger announcement: acting on the tip would constitute (virtual) insider trading.

The researchers found that the AI agent decided to act on the insider tip and buy the stock 70% of the time - and when the agent was asked by its manager whether it had access to the insider information, the LLM frequently decided to lie about it, reasoning (in a private notepad given to the agent) that it was better to avoid admitting to insider trading. Though this took place in a simulated environment, it reveals how AI agents, when deployed in the real world, might end up having unintended consequences. However, it’s important to remember that this work is a demo of a single example and doesn’t draw any conclusions about how likely such behaviour is to occur in real-world deployment situations. For further details, see Apollo’s technical report.

Current oversight methods are insufficient

In a world in which we deploy AI agents, we want to make sure that their deployment does not have unintended consequences, and that we can effectively oversee their behaviour. For now, AI agents are low enough in capability that we can often spot when they are making mistakes or going off the rails, and fix them—but what happens when we must oversee the behaviour of systems that are better than us in particular specialised domains? 

One way to approach this might be to use a weaker AI system to assist in supervising the behaviour of a more capable AI system—if this works then it offers a way forward for this “scalable oversight” problem. Our partners Redwood Research simulated this kind of set up in a toy environment: they had one highly capable model write code, and a second less capable model evaluate the code. The evaluator model was instructed to try to find vulnerabilities in the code that were intentionally inserted by the coder model. The experiment found that around half of the time the evaluator model fails to catch these vulnerabilities. Despite being a simple proof of concept, our takeaway is that monitoring powerful AI systems may get increasingly challenging as their capabilities increase and requires significant investment to reach high levels of assurance and safety. For further details, see Redwood’s paper.

AISI is continuing to expand our investigations into the potential for loss of control from autonomous AI systems, and will build robust evaluations, including detailed assessments of current AI agents’ capabilities for autonomous replication and their potential to accelerate AI development.

Work beyond evaluations

Evaluations alone are not sufficient to deliver on AISI’s mission and enable the effective governance of advanced AI. We also need further technological advances that support new forms of governance, enable fundamentally safer AI development, and ensure the systemic safety of society against AI risks. 

Therefore, AISI is launching a foundational AI safety research effort across a range of areas, including capabilities elicitation and jailbreaking, AI model explainability, interventions on model behaviour, and novel approaches to AI alignment. These workstreams will aim to improve model performance, transparency, safety, and alignment beyond the existing state-of-the-art.

Harnessing AI is an opportunity that could be transformational for the UK and the rest of the world. Advanced AI systems have the potential to drive economic growth and productivity, boost health and wellbeing, improve public services, and increase security. But advanced AI systems also pose significant risks and unlocking the benefits of advanced AI requires tackling these risks head on.