A guide to non-standard files outside the standard Foreign and Commonwealth Office corporate file plan.
The ‘non-standard’ files – overview
The term ‘non-standard’ denotes a file which is not a standard FCO annual departmental file. Many of the files were not created by the modern FCO, they were created by the FCO’s predecessors or other bodies eg Foreign Compensation Commission. However there is also FCO material amongst the non-standard collections, which is just not in the standard departmental format. Non-standard material could be loose papers in boxes or material in ring binders. We no longer refer to this collection as a ‘special collection’ because it is not an accurate description of these files and records. This is a term used mainly in academic libraries to describe their collection of rare books and manuscripts. We now refer to them as ‘non-standard files’.
It does not follow that the ‘non-standard’ files are necessarily of high value for historians. The historical value of the files is determined through an appraisal and selection process under the guidance and supervision of The National Archives. FCO selection decisions are submitted to TNA Records Decision Panel and their decisions are published in meeting summaries on the TNA website.
Under the Public Records Act, government departments are required to transfer files selected for permanent preservation to TNA in line with TNA’s collection policy and the statutory timetable. However the FCO’s transfer of non-standard files has been delayed because in the past we have given priority to the transfer to TNA of standard departmental files and the colonial administration files. This means we continue to hold a large collection of non-standard files. We are committed to the release of our historical records so that they can be freely consulted by the academic community and the wider public. We are not retaining records simply because they are embarrassing or because they shed a particular light on the past.
In 2013, a detailed records audit of the non-standard material was carried out and since then we have continued to update our records inventory. All of these files are now legally retained under a Retention Instrument.
The role of the Independent Reviewer
On 12 December 2013 the then Minister for Europe announced in a written statement to Parliament the appointment of Professor Tony Badger, Professor of History, Northumbria University, as Independent Reviewer of the non-standard files. Professor Badger provides external oversight of the FCO’s review of the non-standard collection.
In July 2017 Professor Badger made the following statement:
I welcome the release on 20 July 2017 at The National Archives of further records from the non-standard collections. This reflects the progress made by the FCO in transferring high priority records from the non-standard collection to The National Archives (TNA). Significant releases include files relating to Guy Burgess and Donald Maclean in October 2015, followed by over 4,000 compensation claim files relating to Nazi persecution, over 2,000 Registry Day Books, as well as the vast majority of Confidential Print volumes, including 36 volumes which held documents regarding the Slave Trade, in 2016. This year we have seen the release of a number of other high priority non-standard file series such as Information Research Department records, Allied Control Commission (regarding Spandau prison) and material relating to the Palestinian mandate. Later this year, the FCO expects to transfer almost 2,000 files relating to the Allied Control Commission for Germany.
The successful effort to release the files in the Migrated Archive from 2011 to 2013 inevitably slowed the transfer of the standard annual FCO files to TNA. While my remit is to provide independent oversight of the release of files in the non-standard collections, I have been keen from the start to ensure that the work on release of these files does not hinder progress in releasing the standard annual files. I agree with the practice of the FCO now to devote twice as much resource to the annual departmental files as to the non-standard ones. The 20-year rule for the transfer of departmental files is a demanding target. The transfer of high priority annual departmental files is currently running some 2-3 years behind schedule in the FCO compared with the government’s timetable for the transition to release at 20 years. This is a consequence of difficulties in recruiting sensitivity reviewers. I am pleased to hear from the FCO that a recent recruitment campaign has succeeded in filling nearly all of the vacant reviewer positions. The scale of the FCO’s commitment should not be underestimated. In 2010, the FCO had fewer than 20 sensitivity reviewers, it now has over 40.
Difficulties in recruiting sensitivity reviewers have also affected the FCO’s programme to release files from the non-standard collection. 60% of the files originally planned to be ready for transfer to TNA by 2019 will be ready by that date but 40% will need to be reviewed in the following years. I am content with the FCO’s decision to give highest priority to the following file series: Allied Control Commission; Permanent Under Secretary’s Department and other intelligence-related material; Information Research Department and Private Office papers.
The non-standard collections contain a large number files from the former government of Hong Kong. The British Academy has rightly pointed out that the release of these files is increasingly important as it becomes harder to access records in Hong Kong itself. The majority of these files are on microform and will have to be digitised before they can be reviewed. On my first visit to the FCO archive this year, FCO records managers briefed me on a pilot project to provide a proof of concept for the digitisation of the Hong Kong files. The project will ensure compliance with TNA archival standards for all Hong Kong files selected for permanent preservation. I am pleased to report that equipment has now been purchased to trial the scanning of the files although there is much still to be done to digitise these important records. I am more optimistic now about progress towards the preparation for release of the Hong Kong files.
On my last visit to the FCO archive, FCO records managers briefed me on progress in reviewing Information Research Department files, a record series which I know to be of considerable interest to the academic community and wider public. The FCO is making good progress in this area.
As part of my independent scrutiny role, I will be continuing to monitor progress on both the annual transfer and the release of the non-standard collections as well as the broader picture of the FCO’s release programme.
It is important to remember that the FCO is not the only government department facing the challenge of the 20-year rule and of the backlog of legacy files. I think it is important to acknowledge the commitment the FCO has made to meet these challenges. In terms of its review and release of legacy files, the FCO is setting an example which I would encourage other government departments to follow if they have backlogs of legacy files awaiting review under the Public Records Act.
The non-standard files – how many?
In 2013, an external company created an inventory of the FCO’s archive which included the non-standard material (see link below) and initially estimated we held 1.2 million non-standard files. However they subsequently revised this estimate to 600,000 following a reassessment of the number of files photographed onto microform.
The exact number of files will only be known when the files are prepared for transfer to TNA, at which stage each file is individually listed.
View the Foreign Office’s Archive Inventory.
Historical value of the non-standard records
Non-standard material that we have assessed to be of high historical value (and a high priority for FCO to review for release) includes:
- colonial records, which complement TNA Colonial Office (CO) and Dominions Office (DO) series
- records relating to the two World Wars (such as claims files relating to World War II and the Control Commission for Germany series.)
- intelligence-related records, such as those about Burgess and Maclean
Material assessed to be lower priority in terms of the timetable for review includes a substantial collection of Foreign Compensation Commission claims files. As with any of the lower priority non-standard files, the FCO will ensure that material of historical interest is preserved.
The medium priority records are those we have assessed to be of some public and historical interest but which are likely to have a more specialised audience than the high priority records. Examples include claims files relating to war damage and historical records covering topics such as shipping and diplomatic protocol.
There are large volumes of registers and indexes amongst the non-standard material (over 20,000 volumes). The value of this material varies. However, we treat any registers or indexes which are valuable as research aids for non-standard series as an equal priority to the record series to which they relate.
There is some duplication between records currently held by the FCO in certain non-standard file series and records already held at TNA. We work with TNA to decide whether such records should be selected for permanent preservation. In some cases, the reason for the transfer of these records to TNA is the unique way in which records are organised and presented rather their unique content.
Release of the non-standard records
The then Minister of State for Europe, David Lidington, informed Parliament on 12 December 2013 that the FCO was developing plans for the review and release of its legacy records. The then Foreign Secretary gave a further update on to Parliament on 27 February 2014 confirming that work on the release programme was already underway. In a Written Ministerial Statement on 21 January 2015, the then Minister of State for Europe restated the FCO’s commitment to compliance with the Public Records Act and to full transparency with respect to our record holdings.
Officials from the FCO met the then Lord Chancellor’s Advisory Council on National Records and Archives (now the Advisory Council on National Records and Archives) in November 2013 to discuss the release plan and we provided the Council with an update on progress in May 2014 and again in November 2015. In 2016, due to continuing difficulties in recruiting sensitivity reviewers, we had to revise our plans for the review and release of both high priority annual departmental and non-standard files. In the case of the non-standard files, we now anticipate completing the review of 60% of the non-standard files previously identified as high priority by 2019 and the remainder after that date.
We have divided the non-standard material into four categories as a first step in order to plan the release of these records: High, Medium and Low priority categories and a separate category for the Hong Kong records. For the time being, we have created this separate category for Hong Kong government records because we need to do further assessment of the content of these records, partly because of their volume and partly because so many are held on microform.
Our prioritisation has taken into account feedback from a wide variety of sources and interested parties with whom we continue to work closely as we develop our release plans.
We have assessed 10% of the non-standard records to be high priority. These are being reviewed for release first (see table below).
We have not yet estimated how long it would take to prepare the Hong Kong records for release. There are large volumes of microform (microfilm and microfiche) in this record series which will need to be digitised. We have now procured a scanner to digitise the Hong Kong records on microfiche and a proof of concept project is under way.
The first high priority non-standard record series have now been released to the public at TNA. The table below shows the records available at TNA, those transferred awaiting release and those in the final stages of preparation for transfer.
|Class description||TNA Class||Number of pieces||Readiness for transfer to TNA|
|Colonial reports||CO 1071||445||Available at TNA|
|Registry Day Books tranche 1||FO 1103||16||Available at TNA|
|Burgess & Mclean||FCO 158||254||Available at TNA|
|Colonial Administration intake lists||FCO 141||6||Available at TNA|
|Confidential Print tranche 1||Various||2||Available at TNA|
|Registry Day Books tranche 2||FO 1103||1033||Available at TNA|
|Nazi Persecution claims files||FO 950||1001||Available at TNA|
|East Africa Colonial Office||CO 822||7||Available at TNA|
|Colonial Intelligence files||CO 1035||41||Available at TNA|
|Confidential Print: Slave Trade||FO 541||38||Available at TNA|
|Confidential Print tranche 2||Various||746||Available at TNA|
|Registry Day Books tranche 3||FO 1103||1021||Available at TNA|
|Nazi Persecution claims files||FO 950||1100||Available at TNA|
|Nazi Persecution claims files||FO 950||1200||Available at TNA|
|Palestine Mandate Records||CO 733||160||Available at TNA|
|Confidential pamphlets||CO 1073||28||Available at TNA|
|Nazi Persecution claims files||FO 950||1060||Available at TNA|
|Spandau Prison Records||FCO 161||71||Available at TNA|
|Crown Agents Loan agreement||FCO 165||21 files||Available at TNA|
|Colonial photographic collection||CO 1069||22 files||Available at TNA|
|Information Research Dept pamphlets pre 1967||FO 1110||36 files||Available at TNA|
|Information Research Dept pamphlets post-1967||FCO 95||30 files||Available at TNA|
|Berlin Legal Files||FP 1060||1,548 files||2017|
|Confidential Print tranche 3||Various||902 files||2017|
|Private Office papers tranche 1||FCO 73||91 files||2017|
|Trucial States (Dubai) Political agency||FCO 164||97 files||2017|
|Rajah of Sarawak||CO 1040||9 files||2017|
|IRD ‘Interpreter’ series||FO 1059||121 files||2017|
|Allied Control Commission for Germany (British Element)||FO 1005||408 files||2017|
|Consulate Seville||FO 332||1 file||2017|
|Private Office papers tranche 2||FCO 73||163 files||2017|
|Victoria Falls Conference||FCO 163||19 files||2017|
|PUSD unregistered papers||FO 1093||107 files||2018|
|Diplomatic Documents||FCO 160||05 files||2018|
|Sir Ian Gilmour’s papers||FCO 167||43 files||2018|
|Commonwealth Relations Office||DO 231||124 files||2018|
|FCO Maps||FCO 18||1,436 files||2018|
FCO Records Day 2015 - proceedings
On 13 May 2015 the FCO welcomed delegates to the third FCO Records Day, held at Hanslope Park. The purpose of the day was to update delegates on the FCO’s release plans; for the Independent Reviewer to share his views on the current review programme; and provide an overview of current challenges relating to digital transfer from representatives of the FCO and The National Archives. There was also a tour of the FCO’s archive. Ais now available.
is also available.
FCO’s prioritisation of the non-standard records
In 2013, we developed a release plan for non-standard files which was based on the review of 608,529 records.
In February 2015 we reported the findings of the FCO’s 2014 record audit to the Lord Chancellor’s Advisory Council on National Archives and Records and we informed the Council that we would be incorporating a further 18,778 records into this non-standard release plan. The additional records principally comprise file registers and volumes of Confidential Print. Records currently incorporated into our non-standard release plan are shown below.
The following table shows the estimated number of records in the review programme for non-standard files on the basis of file counts carried out in 2013 and 2014:
|Priority||Records for review identified in 2013||Records for review identified in 2014||Total|
|Medium and low priority||273961||16508||290469|
|Records from the former British administration of Hong Kong||267956||awaiting further appraisal|
|Status as public records to be determined (e.g. some of this material may have been commercially published and may not fall within the scope of the Public Records Act)||8220||awaiting further appraisal|
|Requires further assessment (e.g. legacy formats which require special equipment to access such as legacy microform and reel-to-reel audio/film)||27||awaiting further appraisal|
We continue to appraise all identified legacy records in the FCO inventory and we will publish updates to these figures when they become available.
Further Explanatory Notes
1. Prioritisation of the non-standard records for release
We have divided the non-standard material into 4 categories in order to plan the release of these records: High, Medium and Low priority categories and a separate category for the Hong Kong records. This prioritisation has taken into account feedback from academics at the first FCO Records Day in May 2013, Freedom of Information requests received by the FCO for non-standard records and our own assessment that certain topics are of obvious public interest and have a wide audience (such as the two World Wars and colonial history). We have also treated records which supplement an existing records series at TNA (such as the Control Commission for Germany) or which complement TNA record series thematically as high priority because they add to the historical narrative provided by the records already at TNA. We have placed the Hong Kong records in a separate category because there are specific challenges relating to this set of records, including the large volume and the format (chiefly microform).
Our release plans and the specific order of release of non-standard records will continue to be informed by close liaison with a range of stakeholders, including the FCO’s team of historians, the Independent Reviewer for non-standard material (Professor Badger), subject-matter experts, the wider academic community and The National Archives. The FCO also submits details to TNA’s Records Decision Panel of any non-standard record series which will constitute a new record series at TNA. For those records selected for permanent preservation we will propose transfer to a permanent archive (usually TNA itself). TNA Records Decision Panel will review FCO’s proposals and publish its decisions on the TNA website.
2. Estimating the number of non-standard files
Until the 1990s, the FCO calculated the size of its record holdings based on linear footage (the physical length of a record series on a shelf). A regular audit of archive record holdings was carried out on this basis.
The practice of carrying out record audits lapsed for some years and was reinstated in 2012 in line with the recommendations of an internal review into the management and release of the colonial administration files (the Cary report). The 2012 audit was carried out by FCO records managers. We used linear meterage to arrive at an estimate of the number of files. On this basis, we estimated the non-standard material contained approximately 250,000 files and we published a high-level archive record inventory on gov.uk.
In 2013, we asked a specialist contractor to carry out a more detailed audit of our records. This audit showed that linear meterage could not be used reliably to estimate the number of records in the non-standard collection because of the substantial quantities of microform, chiefly microfiche and microfilm. Microform enables large volumes of records to be stored in a reduced format on flat film or reels of film. The number of records per linear metre for microform is very high. We asked the specialist contractor to make an assumption that every microfiche (sheet of film) or microfilm (spool) contained a full set of images and that 100 images should be treated as the equivalent of a standard FCO departmental file (since an image is a photograph of a page from an original file).
The specialist contractor initially overestimated the number of files in the non-standard collections due to errors in applying the formula for calculating equivalent files for microform records. The FCO was at this point provided with a total figure for the non-standard material of 1.2 million files. We reported this estimate to The National Archives for their Record Transfer Report and to The Lord Chancellor’s Advisory Council on National Records and Archives. We also declared this estimate on the gov.uk website since it reflected our understanding at the time. Following quality control work by the specialist contractor, the total number of non-standard files was revised down to 600,000.
Following further audits of material managed by the FCO’s Archive Management Team, in 2015 we incorporated a further 18,778 files into the non-standard release programme. These additional files consist almost exclusively of file registers and Confidential Print volumes..
The large number of different formats in the non-standard material such as standard paper files, microform, computer media, index cards, reel-to-reel film and audio tapes makes a fully accurate count impractical. An exact file count for each records series will only be known when the records are reviewed for release. Counting all of the records individually now would delay higher priority work on reviewing the records for release.
As outlined above, the term ‘non-standard’ refers to files outside the FCO filing plan which are held by the Archive Management Team in the central FCO archive. Our estimate of 600,000 non-standard files does not include any non-standard files amongst the estimated 170,000 files in legacy record series held outside the FCO main archive. These additional records were identified during a records audit carried out in Autumn 2014.
3. Retention by the FCO of non-standard files
All of the FCO’s legacy files, meaning those overdue for transfer to The National Archives, are legally retained by the FCO in compliance with the Public Records Act. The Lord Chancellor has authorised, at FCO’s request, a legal instrument granting administrative retention of the records for five years until 2019. The specific reason for retention of the non-standard records is described in The National Archives publication Access to Public Records p.24:
records which will be transferred, in part at least, but the selection and sensitivity reviewing process has not been completed (backlogs).
The 5-year retention period covers the first phase of the non-standard release plan. Although we plan to prepare further non-standard files for release in subsequent years (the medium and low priority and Hong Kong files), the FCO will to apply to the Advisory Council on National Records and Archives in 2019 for any necessary extension of the legal retention. Administrative retention is different from retention on grounds of sensitivity.
4. Past prioritisation of annual transfer over non-standard files
The transfer of non-standard records to The National Archives (TNA) has been delayed in the past primarily because the FCO has given priority to the review and release of annual departmental files which we know are highly valued by researchers. Between 2011 and 2013 we also transferred the colonial administration files from former British territories to TNA. These two factors have led to an accumulation of non-standard files.
In the majority of cases, the FCO has not in recent years made any formal assessment of the non-standard files for sensitivity. Some files are likely to be sensitive, others we already know are very unlikely to be sensitive. Even for the non-sensitive files the amount of resource needed to transfer a file to the TNA is not insignificant since the files must be appraised, selected, catalogued and prepared for transfer. In the past this resource has not been available because of the concentration of effort on annual departmental and migrated archive files.
5. Dates assigned to non-standard and annual departmental files
The specialist contractor compiling the detailed inventory on the FCO’s behalf was asked to determine the date range of record series in the FCO archive. A record ‘series’ is a collection of related records. An example amongst the non-standard records is the Allied Control Commission series.
Some of the record series in the archive consist of large numbers of files, in which case we asked the inventory team to sample the date of creation of the files as accurately as possible to determine the date range.
View the Foreign Office’s Archive Inventory.