© Crown copyright 2021
This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: firstname.lastname@example.org.
Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.
This publication is available at https://www.gov.uk/government/publications/assessment-and-procurement-of-coronavirus-covid-19-tests/protocol-for-evaluation-of-rapid-diagnostic-assays-for-specific-sars-cov-2-antigens-lateral-flow-devices
On 15 August 2020, DHSC Ministers commissioned PHE Porton Down to establish a time-limited SARS-CoV-2 test development and evaluation programme. A scientific team has been established to deliver this work in collaboration with the University of Oxford, PHE Porton Down’s external scientific advisors.
A delivery group, chaired by the Director of PHE Porton Down, will meet daily to direct and co-ordinate the programme. Oversight will be provided by Sir John Bell to ensure that it the work is carried out to appropriate quality and timeliness standards, and to whom progress will be reported at least weekly through an Oversight Group that he chairs. Membership shall comprise the external scientific advisors, Director of Laboratories, Supplies and Innovation at the Department of Health and Social Care (DHSC), Director of PHE Porton Down and the dedicated DHSC leads.
There are an increasingly large number of commercial lateral flow antigen devices available. It is not feasible to conduct large scale evaluations on all of them; current sample resources only allow a limited number of devices for full evaluation. There is therefore a need to shortlist this limited number from many candidates as quickly as possible.
The Oversight Group has agreed the 3-phase process set out below for the conduct of the evaluations to undertake a phased short-listing approach. It is important to note that a failure to be short-listed does not indicate that the product is otherwise not usable. Rather it simply means that other products, on the basis of test results, are considered to have a better chance of meeting the required standards to proceed to larger scale field-testing.
DHSC will identify a pipeline of products – direct viral antigen detection lateral flow devices and distributed amplification tests – that could enable saturation testing for SARS-CoV-2 through comprehensive and repeated whole population home-testing.
DHSC will undertake a desk-top review, including of manufacturers’ claimed performance and instructions for use, to identify tests which, prima facie, may perform with sufficient sensitivity and very high specificity to enable them to be used to detect and direct responses to emerging outbreaks.
Only products deemed by DHSC to have potential, and agreed as such by the Oversight Group, will be referred to PHE Porton Down for phase 2 evaluation, with the necessary transport and logistic arrangements handled by DHSC in liaison with the companies concerned. A minimum of 500 kits will be shipped for phase 2 testing.
On receipt at PHE Porton Down, the scientific team dedicated to this work will carry out a ‘futility test’ on each product, the objectives of which are to:
- identify the kit failure rate (indicator of robustness)
- identify whether known negative samples give a negative result (indicator of specificity)
- identify whether known positive samples give a positive result (indicator of sensitivity)
- provide an initial view on usability of each test, including the time to result, and ease of use
The Director of PHE Porton Down will report the findings for each product to the Oversight Group, who, based on the criteria below, will recommend which tests should proceed to phase 3 evaluation. If results do not meet the required standard at the futility testing stage, no further testing will take place and manufacturers will be advised accordingly by DHSC, who will share the results with them.
Phase 2 methodology
Saliva will be collected from healthy adult volunteers at PHE Porton Down and screened for SARS-CoV-2 RNA via PCR after inactivation with AVL using the Roche cobas® 6800 system and their proprietary SARS-CoV-2 assay. Any PCR positive samples will be discarded. Saliva sample sets may be different between evaluations, i.e. will be collected on an ongoing basis to ensure sufficient supply.
Positive samples will be processed in a Class III microbiological safety cabinet.
Saliva samples from 15 different individuals will be spiked with SARS-CoV-2 virus stock (7.8x106 plaque forming units(pfu)/mL VIC/1/2020) and serially diluted in the same saliva sample to give dilutions of 105, 104, 103 and 102 pfu/mL. Samples may be frozen at this stage.
Each diluted sample (n=60) will be tested on the lateral flow device (LFD) according to the manufacturer’s instructions. Where LFDs require swab samples to be tested, 200uL of sample will be substituted. If the manufacturer’s instructions indicate a lower volume, the amount specified will be used.
Negative samples will be processed in a Class I microbiological safety cabinet. Saliva samples from 71 different individuals, including those that were used to form the positive sample set, will be tested on the LFD according to the manufacturer’s instructions. Where LFDs require swab samples to be tested, 200uL of sample will be substituted. If the manufacturer’s instructions indicate a lower volume, the amount specified will be used.
Phase 2 results
LFDs will be interpreted according to the manufacturer’s instructions. Details of incubation times and interpretation of control and test bands will be recorded on data results sheets. A kit shall be deemed to have failed if the control band on the test does not appear.
Data will be collated as follows and will be available to the manufacturer.
|Sample||Sample dilution||Sample volume added (uL)||Samples tested||Kit failures||Negative||Positive|
|SARS-CoV-2 viral stock in saliva||X105||15|
|SARS-CoV-2 viral stock in saliva||X104||15|
|SARS-CoV-2 viral stock in saliva||X103||15|
|SARS-CoV-2 viral stock in saliva||X102||15|
Phase 2 evaluation criteria
Results of the futility test will be interpreted as follows and reported to the Oversight Group accordingly:
|Number tested||Number of failures for threshold||Percentage (95% CI) wrong result at threshold||Definition of failure|
|Kit failure rate||131||13||13/131 9.9% (5.4–16.4)||13 or more kit failures|
|Specificity||71||3||3/71 4.2% (0.88–11.9)||3 or more false positives|
|Sensitivity (102 pfu/ml)||15||8||8/15 53.3% (26.6–78.7)||8 or more false negatives|
For those tests that pass phase 2, an assessment using 3 seasonal coronaviruses will be carried out to identify any cross-reactivity.
Only if the Oversight Group agrees that a product has passed phase 2 and 2A will it proceed to phase 3. DHSC will advise the relevant manufacturer accordingly and request them to ship at least 4,000 kits directly to PHE Porton Down.
Phase 3 will be carried out against a larger sample set, namely 1,000 true negatives and 200 true positives, which will be sourced by Oxford University Hospitals. The negatives will be fresh samples, transported within 48 hours, the positives will be frozen. This follows recent work at PHE Porton Down to support the development of this protocol which found there to be no discernible difference in performance when using frozen samples.
Additionally, samples from people from PCR-confirmed positive cases identified by the Lighthouse Laboratories will be tested both ‘in the field’ and subsequently by PHE Porton Down. Wherever possible, at least 2 products will be evaluated side-by-side in phase 3 to ensure most efficient use of time and resources.
Phase 3 findings will be reported to the Oversight Group, with DHSC and ministers using this information and any recommendations to inform potential purchasing decisions.