FCO non-standard files
A guide to non-standard files outside the standard FCO corporate file plan.
The ‘non-standard’ files – overview
Any files which are outside the FCO corporate file plan have in the past been called the “special collections”. This term only denotes a file which is not a standard FCO annual departmental file. It does not mean that these files are of special value: whilst some are of considerable historical value, others are ephemeral. Many of the files were not created by the modern FCO, they were created by the FCO’s predecessors or other bodies e.g. Foreign Compensation Commission. However there is also FCO material amongst the non-standard collections, which is just not in the standard departmental format. Non-standard material could be loose papers in boxes or material in ring binders. We no longer refer to this collection as a “special collection” because it is not an accurate description of these files and records. This is a term used mainly in academic libraries to describe their collection of rare books and manuscripts. We now refer to them as “non-standard files”.
It does not follow that the “non-standard” files are necessarily of high value for historians. The historical value of the files is determined through an appraisal and selection process under the guidance and supervision of The National Archives. FCO selection decisions are submitted to TNA Records Decision Panel and their decisions are published in meeting summaries on the TNA website.
Under the Public Records Act, government departments are required to transfer files selected for permanent preservation to TNA in line with TNA’s collection policy and the statutory timetable. However the FCO’s transfer of non-standard files has been delayed because in the past we have given priority to the transfer to TNA of standard departmental files and the colonial administration files. This means we continue to hold a large collection of non-standard files. We are committed to the release of our historical records so that they can be freely consulted by the academic community and the wider public. We are not retaining records simply because they are “embarrassing” or because they shed a particular light on the past.
In 2013, a detailed records audit of the non-standard material was carried out. All of these files are now legally retained under a Lord Chancellor’s Instrument.
The role of the Independent Reviewer
On 12 December 2013 the Minister for Europe announced in a written statement to Parliament the appointment of Professor Tony Badger, Professor of History, Northumbria University, as Independent Reviewer of the non-standard material.
On 11th February 2015 Professor Badger made the following statement:
I welcome the release in December last year of the first high priority record series from the special collections, consisting of 445 colonial reports. Work is now under way on the preparation for release of further special collection record series including Foreign Office compensation claim files relating to Nazi persecution and Foreign Office files on Guy Burgess and Donald Maclean.
The FCO’s Archive Management Team have briefed me about the records audit which was carried out last year across the FCO and which was the subject of the Minister for Europe’s Written Ministerial Statement on 21 January. This audit was the second of the high level searches that were prompted by the Cary Report into the Migrated Archive. Antony Cary’s ‘first and most obvious recommendation’ was to conduct a full inventory of what the FCO holds’ The first survey revealed the extent of the special collections at Hanslope Park. This second audit documented records held in the FCO itself that had not been transferred to Hanslope Park. It is to be hoped that Anthony Cary’s recommendation has now been fully complied with. It would be disappointing at this stage if more unreported records were discovered in the FCO. Most (but not all) of the legacy records identified during this second audit are either copies of original records or they are working records likely to be of business value to the FCO at the moment, rather than of long-term historical value. However some further appraisal work is required. The FCO will be keeping me up to date on how they are prioritising these files for review under the Public Records Act.
In my last statement, I explained that as Independent Reviewer I would be looking into the origins of the FCO’s “Special Collections” and the reasons why it is has taken so long for these files to be reviewed for release. Resource constraints in the past and the pressure to comply with the routine Annual Transfer of documents under the Thirty Year rule have certainly played a part. But those are not the entire explanation. With the support of the FCO’s Archive Management Team, I am researching the background to the past management of specific special collection record series and I will make available my findings in due course.
The non-standard files – how many?
In 2013, an external company created an inventory of the FCO’s archive which included the non-standard material (see link below) and initially estimated we held 1.2 million non-standard files. However they subsequently revised this estimate to 600,000 following a reassessment of the number of files photographed onto microform.
The exact number of files will only be known when the files are prepared for transfer to TNA, at which stage each file is individually listed.
View the Foreign Office’s Archive Inventory.
Historical value of the non-standard records
Non-standard material that we have assessed to be of high historical value (and a high priority for FCO to review for release) includes colonial records, which complement TNA Colonial Office (CO) and Dominions Office (DO) series, and records relating to the two World Wars (such as claims files relating to World War II and the Control Commission for Germany series.) Material assessed to be low priority includes a substantial collection of Foreign Compensation Commission and claims files. Some of these records may be of historical interest, but there are large volumes of routine case files.
The medium priority records are those we have assessed to be of some public and historical interest but which are likely to have a more specialised audience than the high priority records. Examples include claims files relating to war damage and historical records covering topics such as shipping and diplomatic protocol.
There are large volumes of registers and indexes amongst the non-standard material (over 20,000 volumes). The value of this material varies. However, we treat any registers or indexes which are valuable as research aids for non-standard series as an equal priority to the record series to which they relate.
There is some duplication between records currently held by the FCO in certain non-standard file series and records already held at TNA. We work with TNA to decide whether such records should be selected for permanent preservation. In some cases, the reason for the transfer of these records to TNA is the unique way in which records are organised and presented rather their unique content.
Release of the non-standard records
The Minister of State for Europe, David Lidington, informed Parliament on 12 December 2013 that the FCO was developing plans for the review and release of its legacy records. The Foreign Secretary gave a further update on to Parliament on 27 February 2014 confirming that work on the release programme was already underway. In a Written Ministerial Statement on 21 January 2015, the Minister of State for Europe restated the FCO’s commitment to compliance with the Public Records Act and to full transparency with respect to our record holdings.
Officials from the FCO met the then Lord Chancellor’s Advisory Council on National Records and Archives (now the Advisory Council on National Records and Archives) in November 2013 to discuss the release plan and we provided the Council with an update on progress in May 2014 and again in November 2015.
We have divided the non-standard material into four categories as a first step towards planning the release of these records: High, Medium and Low priority categories and a separate category for the Hong Kong records. For the time being, we have created this separate category for Hong Kong government records because we need to do further assessment of the content of these records, partly because of their volume and partly because so many are held on microform.
Our prioritisation has taken into account feedback from a wide variety of sources and interested parties with whom we continue to work closely as we develop our release plans.
We have assessed 10% of the non-standard records to be high priority. These are being reviewed for release first (see table below). We aim to review all high priority non-standard records for release by 2019.
We have not yet estimated how long it would take to prepare the Hong Kong records for release. There are large volumes of microform (microfilm and microfiche) in this record series which will need to be digitised. We are considering how we process these microform records.
The first high priority non-standard record series have now been released to the public at TNA. The table below shows the records available at TNA, those transferred awaiting release and those in the final stages of preparation for transfer.
|Class description||TNA Class||Number of pieces||Readiness for transfer to TNA|
|Colonial reports||CO 1071||445||Available at TNA|
|Registry Day Books tranche 1||FO 1103||16||Available at TNA|
|Burgess & Mclean||FCO 158||254||Available at TNA|
|Colonial Administration intake lists||FCO 141||6||Available at TNA|
|Confidential Print tranche 1||Various||2||Available at TNA|
|Registry Day Books tranche 2||FO 1103||1033||Available at TNA|
|Nazi Persecution claims files||FO 950||1001||Available at TNA|
|East Africa Colonial Office||CO 822||7||Available at TNA|
|Colonial Intelligence files||CO 1035||41||2016|
|Confidential Print: Slave Trade||FO 541||38||2016|
|Confidential Print tranche 2||Various||746||2016|
|Registry Day Books tranche 3||FO 1103||1021||2016|
|Nazi Persecution claims files||FO 950||1100||2016|
|Nazi Persecution claims files||FO 950||1200||2016|
|Palestine Mandate Records||CO 733||160||2016-2017|
|Confidential pamphlets||CO 1073||28||2016-2017|
|Nazi Persecution claims files||FO 950||1060||2016|
|Spandau Prison Records||FCO 161||71||2016-2017|
|Confidential Print tranche 3||Various||849||2016-2017|
|Confidential Print tranches 4 & 5||Various||1377||2016-2017|
FCO Records Day 2015 - proceedings
On 13 May 2015 the FCO welcomed delegates to the third FCO Records Day, held at Hanslope Park. The purpose of the day was to update delegates on the FCO’s release plans; for the Independent Reviewer to share his views on the current review programme; and provide an overview of current challenges relating to digital transfer from representatives of the FCO and The National Archives. There was also a tour of the FCO’s archive. A record of proceedings (PDF, 832KB, 30 pages) is now available.
A record of proceedings from 2014 (PDF, 717KB, 20 pages) is also available.
FCO’s prioritisation of the non-standard records
In 2013, we developed a release plan for non-standard files which entailed the review of 608529 records.
In February 2015 we reported the findings of the FCO’s 2014 record audit to the Lord Chancellor’s Advisory Council on National Archives and Records and we informed the Council that we would be incorporating a further 18,778 records into this non-standard release plan. The additional records principally comprise file registers and volumes of Confidential Print. Records currently incorporated into our non-standard release plan are shown below.
|Priority||Records for review identified in 2013||Records for review identified in 2014||Total|
|Medium and low priority||273961||16508||290469|
|Records from the former British administration of Hong Kong||267956||awaiting further appraisal|
|Status as public records to be determined (e.g. some of this material may have been commercially published and may not fall within the scope of the Public Records Act)||8220||awaiting further appraisal|
|Requires further assessment (e.g. legacy formats which require special equipment to access such as legacy microform and reel-to-reel audio/film)||27||awaiting further appraisal|
We continue to appraise all identified legacy records in the FCO inventory and we will publish updates to these figures when they become available.
Further Explanatory Notes – Index
1. Prioritisation of the non-standard records for release
We have divided the non-standard material into four categories as a first step towards planning the release of these records: High, Medium and Low priority categories and a separate category for the Hong Kong records. This prioritisation has taken into account feedback from academics at the first FCO Records Day in May 2013, Freedom of Information requests received by the FCO for non-standard records and our own assessment that certain topics are of obvious public interest and have a wide audience (such as the two World Wars and colonial history). We have also treated records which supplement an existing records series at TNA (such as the Control Commission for Germany) or which complement TNA record series thematically as high priority because they add to the historical narrative provided by the records already at TNA. We have placed the Hong Kong records in a separate category because there are specific challenges relating to this set of records, including the large volume and the format (chiefly microform).
Our release plans and the specific order of release of non-standard records will continue to be informed by close liaison with a range of stakeholders, including the FCO’s team of historians, the Independent Reviewer for non-standard material (Professor Badger), subject-matter experts, the wider academic community and The National Archives. The FCO also submits details to TNA’s Records Decision Panel of any non-standard record series which will constitute a new record series at TNA. For those records selected for permanent preservation we will propose transfer to a permanent archive (usually TNA itself). TNA Records Decision Panel will review FCO’s proposals and publish its decisions on the TNA website.
2. Estimating the number of non-standard files
Until the 1990s, the FCO calculated the size of its record holdings based on linear footage (the physical length of a record series on a shelf). A regular audit of archive record holdings was carried out on this basis.
The practice of carrying out record audits lapsed for some years and was reinstated in 2012 in line with the recommendations of an internal review into the management and release of the colonial administration files (the Cary report). The 2012 audit was carried out by FCO records managers. We used linear meterage to arrive at an estimate of the number of files. On this basis, we estimated the non-standard material contained approximately 250,000 files and we published a high-level archive record inventory on gov.uk.
In 2013, we asked a specialist contractor to carry out a more detailed audit of our records. This audit showed that linear meterage could not be used reliably to estimate the number of records in the non-standard files because of the substantial quantities of microform, chiefly microfiche and microfilm. Microform enables large volumes of records to be stored in a reduced format on flat film or reels of film. The number of records per linear metre for microform is very high. We asked the specialist contractor to make an assumption that every microfiche (sheet of film) or microfilm (spool) contained a full set of images and that 100 images should be treated as the equivalent of a standard FCO departmental file (since an image is a photograph of a page from an original file).
The specialist contractor initially overestimated the number of files in the non-standard collections due to errors in applying the formula for calculating equivalent files for microform records. The FCO was at this point provided with a total figure for the non-standard material of 1.2 million files. We reported this estimate to The National Archives for their Record Transfer Report and to The Lord Chancellor’s Advisory Council on National Records and Archives. We also declared this estimate on the gov.uk website since it reflected our understanding at the time. Following quality control work by the specialist contractor, the total number of non-standard files was revised down substantially to 600,000. The reason for this large difference is that technical issues with the generation of equivalent file numbers for microform meant that in some cases the number of images on a microform sheet or film had been counted, in other cases a calculation of 10 images per file had been used. This made a substantial difference to the estimate, for some record series by a factor of 100.
Currently the number of non-standard files in the FCO release programme is 627,307. Following further audits of material managed by the FCO’s Archive Management Team, in 2015 we incorporated a further 18,778 files into the non-standard release programme. These additional files consist almost exclusively of file registers and Confidential Print volumes.
The large number of different formats in the non-standard material such as standard paper files, microform, computer media, index cards, reel-to-reel film and audio tapes makes a fully accurate count impractical. An exact file count for each records series will only be known when the records are reviewed for release. Counting all of the records individually now would delay higher priority work on reviewing the records for release.
The term “non-standard” refers to files outside the FCO filing plan which are held by the Archive Management Team in the central FCO archive. Our estimate of 600,000 non-standard files does not include any non-standard files amongst the estimated 170,000 files in legacy record series held outside the FCO main archive. These additional records were identified during a records audit carried out in Autumn 2014.
3. Retention by the FCO of non-standard files
All of the FCO’s legacy files, meaning those overdue for transfer to The National Archives, are legally retained by the FCO in compliance with the Public Records Act. The Lord Chancellor has authorised, at FCO’s request, a legal instrument granting administrative retention of the records for five years until 2019. The specific reason for retention of the non-standard records is described in The National Archives publication Access to Public Records p.24:
records which will be transferred, in part at least, but the selection and sensitivity reviewing process has not been completed (backlogs).
The five year retention period covers the first phase of the non-standard release plan. Although we plan to prepare further non-standard files for release in subsequent years (the medium and low priority and Hong Kong files), the FCO will to apply to the Advisory Council on National Records and Archives in 2019 for any extension of the legal retention. Administrative retention is different from retention on grounds of sensitivity.
4. Past prioritisation of annual transfer over non-standard files
The transfer of non-standard records to The National Archives (TNA) has been delayed in the past primarily because the FCO has given priority to the review and release of annual departmental files which we know are highly valued by researchers. Between 2011 and 2013 we also transferred the colonial administration files from former British territories to TNA. These two factors have led to an accumulation of non-standard files.
In the majority of cases, the FCO has not in recent years made any formal assessment of the non-standard files for sensitivity. Some files are likely to be sensitive, others we already know are very unlikely to be sensitive. Even for the non-sensitive files the amount of resource needed to transfer a file to the TNA is not insignificant since the files must be appraised, selected, catalogued and prepared for transfer. In the past this resource has not been available because of the concentration of effort on annual departmental and migrated archive files.
5. Annual departmental and non-standard files
The FCO main archive contains an estimated 1.2 million files, split approximately 50/50 between annual departmental and non-standard files. In addition we have identified a further 170,000 files in legacy record series which are held outside the FCO main archive in UK departments and at overseas posts.
An annual departmental file is a file organised in line with the FCO’s normal filing sequence. As described on The National Archives (TNA) website, FCO files from 1967 onwards are arranged in regional geographical or subject groups reflecting their departmental organisation. Each of these annual departmental files is assigned departmental and functional codes as well as a code indicating the subject-matter. The vast majority of the archive annual departmental files held by the FCO (around 500,000) are not yet due for selection for permanent preservation and transfer to TNA.
Non-standard files are those which do not conform to the above definition of an annual departmental file. Not every non-standard file is old (although most are) and not every non-standard file is overdue for transfer to TNA (again, most are). For example, the 1999 records amongst the FCO Board of Management papers in the archive inventory (dated 1999-2000) are not due for selection and transfer to TNA until 2021.
In practice, any file which has not been produced by the modern FCO will class as a non-standard file, including Colonial and Commonwealth Relations Office files.
The distinction between non-standard files and annual departmental files is useful mainly for administrative purposes relating to the processing of the files. The FCO’s Archive Management Team has a well-defined and well-established process for the review and release of annual departmental files. Non-standard files require a greater degree of initial appraisal before work begins to review and transfer the files. For instance, many non-standard files require a greater degree of special subject and/or linguistic expertise to review than annual departmental files and many may also require specialist repair because of their age.
In a few cases, we have categorised storage areas in the FCO’s archive inventory as “departmental” because they are used as temporary storage for records managers who are processing records (whether non-standard files or annual departmental records). For instance, we have an area of 50 metres which is used for material being sensitivity-reviewed and for files being examined as a result of Freedom of Information requests. These temporary storage areas constitute a very small proportion of the total storage capacity of the archive.
6. Dates assigned to non-standard and annual departmental files
The specialist contractor compiling the detailed inventory on the FCO’s behalf was asked to determine the date range of record series in the FCO archive. A record “series” is a collection of related records. An example amongst the non-standard records is the Allied Control Commission series.
Some of the record series in the archive consist of large numbers of files, in which case we asked the inventory team to sample the date of creation of the files as accurately as possible to determine the date range. Such dates have been assigned for descriptive purposes so that FCO records managers understand the content and scope of archive record series we hold. It is not the purpose of the inventory to organise records by reference to Section 10.2 of the Public Records Act in terms of whether such files or records are due for transfer to The National Archives (TNA)
View the Foreign Office’s Archive Inventory