The Foreign and Commonwealth Office's Archive Management Team.
The FCO’s commitment to transparency
The FCO is committed to increasing public knowledge about foreign policy and the work of the FCO. As the Minister for Europe stated to Parliament in December 2013, the FCO remains fully committed both to complying with our public records obligations and to doing so with maximum transparency.
The ‘special collection’ files – overview
The Foreign and Commonwealth Office (FCO), like all government departments, maintains an archive of its departmental files. We hold around 600,000 of these files, most of which are not yet due for transfer to The National Archives (TNA).
The FCO holds a further collection of files, also estimated at 600,000, which are known as ‘special collections’ - those which are outside the normal FCO filing system. These special collections files are generally older than the departmental files and were created mainly by the FCO or its predecessors (such as the Colonial Office).
The term ‘special collection’ is therefore used only to indicate that these files are not standard FCO departmental files. It does not follow that they are necessarily of high value for historians. The historical value of the files will be determined through an appraisal and selection process under the guidance and supervision of The National Archives. FCO selection decisions will be sent to TNA Records Decision Panel and their decisions will be published in meeting summaries on the TNA website.
Under the Public Records Act, government departments are required to transfer files selected for permanent preservation to TNA in line with TNA’s collection policy and the statutory timetable. However the FCO’s transfer of special collection files has been delayed because we have given priority to the transfer to TNA of departmental files and the colonial administration files. This means we continue to hold a large collection of special collection files. We are committed to the release of our historical records so that they can be freely consulted by the academic community and the wider public. We are not retaining records simply because they are “embarrassing” or because they shed a particular light on the past.
It’s important to note that all the files we hold which are overdue for transfer are retained in compliance with the Public Records Act. Our retention of these files has been authorised through a legal instrument (known as a Lord Chancellor’s Instrument) which, at the FCO’s request, has been granted by the Lord Chancellor. This Instrument lasts until the end of 2014 and was granted in order to allow the FCO time to acquire the extra capacity needed to implement the release programme. Work that will lead to the release of these files is already under way, overseen by the Independent Reviewer of the special collections – Professor Tony Badger, Paul Mellon Professor of American history and Master of Clare College at the University of Cambridge.
As highlighted by Professor Badger, the scale of the operation to review and release the files should not be underestimated. The large volume of files means it will take time to appraise and process them.
The role of the Independent Reviewer
On 12 December 2013 the Minister for Europe announced in a written statement to Parliament the appointment of Professor Tony Badger as Independent Reviewer of the special collections. Professor Badger is Paul Mellon Professor of American History and Master of Clare College Cambridge. Professor Badger has made the following statement about his role as Independent Reviewer:
It is difficult to overestimate the scale of the process of releasing over 600,000 files. By way of comparison, the release of the migrated colonial archive, which I have just overseen, involved just under 20,000 files. Last year the FCO transferred approximately 12,000 files (chiefly colonial administration files).
As the Independent Reviewer, I intend:
to come to a more complete understanding of how the special collections came into being, why it has taken so long to secure the release of the papers, and to make public those findings.
to approve the determination of the prioritisation of the materials to be released. While this eclectic collection clearly contains a good deal of low-level administrative material that has little or no historical value, the collection also contains some extraordinarily valuable papers that should be transferred to The National Archives as soon as possible. The prioritisation needs to be decided in consultation with the academic community, in addition to The National Archives, and to take into account areas of current public interest and concern.
to provide assurance that the Archive Management Team has the capability to drive this release forward without further delaying the release of papers under annual transfer arrangements.
to provide assurance that files and documents are not retained or redacted unnecessarily in the process of sensitivity review.
to provide assurance about the selection of files to be transferred to The National Archives. In the case of the migrated colonial archive, it was agreed from the start that every document should be transferred, subject to legal exemptions. Given the size of the special collections and the routine administrative nature of significant parts of the material, it would be both impractical and unnecessary to transfer every document. But it is imperative that there should be public confidence in the process of decision-making in the FCO and The National Archives.
As in the oversight of the transfer of the migrated archive, I intend to consult widely with interested academics and to encourage, and to respond to, queries from journalists and interested members of the public.
The special collection files – how many?
In 2013, an external company created an inventory of the special collections (see link below) and initially estimated we held 1.2 million special collection files. However they subsequently revised this estimate to 600,000 following a reassessment of the number of files photographed onto microform.
The exact number of files will only be known when the files are prepared for transfer to TNA, at which stage each file is individually listed.
View the Foreign Office’s Archive Inventory.
Historical value of the special collection records
We use the term “special collection” only to indicate that these files are not standard FCO departmental files, not to indicate that they are necessarily of high value for historians.
Special collection material we have initially assessed to be of high historical value (and a high priority for FCO to review for release) includes colonial records, which complement TNA Colonial Office (CO) and Dominions Office (DO) series, and records relating to the two World Wars (such as claims files relating to World War II and the Control Commission for Germany series).
Material initially assessed to be low priority includes:
large volumes of registers and indexes (over 20,000 volumes) which FCO records managers have used in the past chiefly as finding aids for FCO archive records. Our current assessment is that we should concentrate initially on original records rather than the registers and indexes which refer to those original records
a substantial collection of Foreign Compensation Commission and claims files which account for over one quarter of the special collections (over 170,000 files). Some of these records will be of historical interest, but there are very large volumes of routine case files
The medium priority records are those we assessed to be of some public and historical interest but which are likely to have a more specialised audience than the high priority records. Examples include claims files relating to war damage and historical records covering topics such as shipping and diplomatic protocol.
There is likely to a significant amount of duplication between some special collection record series and existing holdings at TNA, particularly amongst the Confidential Print (over 7,000 volumes) and publications from the former Colonial Office library. Confidential Print replicates material from original records which may have already transferred to TNA. A number of Confidential Print volumes held by FCO are themselves already held at TNA. The extent of this type of duplication will be better known when we start appraising the records in detail as part of the review and release process.
Release of the special collection records
The Minister of State for Europe, David Lidington, informed Parliament on 12 December 2013 that the FCO was developing plans for the review and release of its legacy records. The Foreign Secretary gave a further update on to Parliament on 27 February 2014 confirming that work on the release programme was already underway.
The FCO met with the Lord Chancellor’s Advisory Council on National Records and Archives in November last year to discuss the release plan and we will provide the Council with an update on progress in May.
We have divided the special collections into four categories as a first step towards planning the release of these records: High, Medium and Low priority categories and a separate category for the Hong Kong records. For the time being, we have created this separate category for Hong Kong records because we need to do further assessment of the content of these records, partly because of their volume and partly because so many are held on microform.
Our initial prioritisation has taken into account feedback from a wide variety of sources and interested parties who we are continuing to work closely with as we develop our release plans.
Our initial assessment is that 10% of the special collection records should be considered high priority and reviewed for release first (see table below). Over a six-year period, starting this year (2014), we will aim to review all high priority special collection records for release.
The first special collection record series we will prepare for transfer to TNA will consist of around 4,000 files relating to British Nationals who suffered Nazi persecution, and their subsequent claims for compensation. These case files contain unique and detailed first person accounts of the suffering that some British people suffered at the hands of the Nazis. The TNA Records Decision Panel has accepted FCO’s proposal that these files should be transferred to TNA.
Our current estimate is that we will be able to prepare the medium and low priority records for release by 2027.
We have not yet estimated how long it would take to prepare the Hong Kong records for release. There are large volumes of microform (microfilm and microfiche) in this record series which will need to be digitised. We will also have to de-duplicate any records held both in hard copy and on microform. These are complex and technical processes, but we are looking at how we can do this as quickly and effectively as possible.
FCO’s initial prioritisation of the special collection records
|Priority||Estimated number of files to be reviewed for permanent preservation|
|Medium and low priority||273961|
|Records from the former British administration of Hong Kong||267956|
|Status as public records to be determined (e.g. some of this material may have been commercially published and may not fall within the scope of the Public Records Act)||8220|
|Requires further assessment (e.g. legacy formats which require special equipment to access such as legacy microform and reel-to-reel audio/film)||27|
The table below provides a broad categorisation by subject of the special collection records.
|Category||Estimated number of items|
|Colonial office records / Colonial period||5853|
|Commercial publications or likely to be in the public domain||801|
|Confidential print (material reprinted from original records. Some of this material may already be at TNA in its original form)||7676|
|Foreign Compensation Commission and claims||170917|
|Industrial and commercial||2867|
|Information Research Department and intelligence-related material||23313|
|Other government departments or international organisation material||15953|
|Other diplomatic material||3597|
|Other material currently being assessed||1382|
|Private Office and senior management papers||4468|
|Registers and indices||21342|
|World War II interest inc Allied Control Commission||10801|
We will provide regular updates via this page on our release plans and on our progress in making the special collection records available publicly.
Further Explanatory Notes – Index
1. Prioritisation of the special collection records for release
We have divided the special collections into four categories as a first step towards planning the release of these records: High, Medium and Low priority categories and a separate category for the Hong Kong records. This initial prioritisation has taken into account feedback from academics at the FCO Records Day in May 2013, Freedom of Information requests received by the FCO for special collection records and our own assessment that certain topics are of obvious public interest and have a wide audience (such as the two World Wars and colonial history). We also consider that records which supplement an existing records series at TNA (such as the Control Commission for Germany) or which complement TNA record series thematically should be considered high priority because they add to the historical narrative provided by the records already at TNA. We have placed the Hong Kong records in a separate category because there are specific challenges relating to this set of records, including the large volume and the format (chiefly microform).
As we develop our release plans and determine the specific order in which special collection records will be released, we will continue to work closely with a range of stakeholders, including the FCO’s team of historians, the Independent Reviewer for the Special Collections (Professor Badger), subject-matter experts, the wider academic community and The National Archives. The FCO will also submit details to TNA’s Records Decision Panel of the scope, content and volume of each special collection record series. For those records selected for permanent preservation we will propose transfer to a permanent archive (usually TNA itself). TNA Records Decision Panel will review FCO’s proposals and publish its decisions on the TNA website.
2. Estimating the number of special collection files
Until the 1990s, the FCO calculated the size of its record holdings based on linear footage (the physical length of a record series on a shelf). A regular audit of archive record holdings was carried out on this basis.
The practice of carrying out record audits lapsed for some years and was reinstated in 2012 in line with the recommendations of an internal review into the management and release of the colonial administration files (the Cary report). The 2012 audit was carried out by FCO records managers. We used linear meterage to arrive at an estimate of the number of files. On this basis, we estimated the special collections contained approximately 250,000 files and we published a high-level archive record inventory on gov.uk.
In 2013, we asked a specialist contractor to carry out a more detailed audit of our records. This audit showed that linear meterage could not be used reliably to estimate the number of records in the special collections because of the substantial quantities of microform, chiefly microfiche and microfilm. Microform enables large volumes of records to be stored in a reduced format on flat film or reels of film. The number of records per linear metre for microform is very high. We asked the specialist contractor to make an assumption that every microfiche (sheet of film) or microfilm (spool) contained a full set of images and that 100 images should be treated as the equivalent of a standard FCO departmental file (since an image is a photograph of a page from an original file).
The specialist contractor initially overestimated the number of files in the special collections due to errors in applying the formula for calculating equivalent files for microform records. The FCO was at this point provided with a total figure for the special collections of 1.2 million files. We reported this estimate to The National Archives for their Record Transfer Report and to The Lord Chancellor’s Advisory Council on National Records and Archives. We also declared this estimate on the gov.uk website since it reflected our understanding at the time. Following quality control work by the specialist contractor, the total number of special collection files was revised down substantially to 600,000. The reason for this large difference is that technical issues with the generation of equivalent file numbers for microform meant that in some cases the number of images on a microform sheet or film had been counted, in other cases a calculation of 10 images per file had been used. This made a substantial difference to the estimate, for some record series by a factor of 100.
The only basis on which the estimate of FCO special collection records has been revised from 1.2 million files down to 600,000 is this calculation of equivalent files for microform records. Nevertheless, our calculations remain an estimate. The large number of different formats in the special collections such as standard paper files, microform, computer media, index cards, reel-to-reel film and audio tapes makes a fully accurate count impractical. An exact file count for each records series will only be known when the records are reviewed for release. Counting all of the records individually now would delay higher priority work on reviewing the records for release.
3. Retention by the FCO of the special collection files
All of the FCO’s legacy files, meaning those overdue for transfer to The National Archives, are legally retained by the FCO in compliance with the Public Records Act. The Lord Chancellor has authorised, at FCO’s request, a legal instrument granting administrative retention of the records for one year. The specific reason for retention of the special collection records is described in The National Archives publication Access to Public Records p.24:
records which will be transferred, in part at least, but the selection and sensitivity reviewing process has not been completed (backlogs).
For the special collection records the one year retention period is to enable the FCO time to acquire the extra capacity to starting reviewing and releasing these files. Administrative retention is different from retention on grounds of sensitivity and is typically of shorter duration.
4. Past prioritisation of annual transfer over special collection files
The transfer of special collection records to The National Archives (TNA) has been delayed in the past primarily because the FCO has given priority to the review and release of annual departmental files which we know are highly valued by researchers. Between 2011 and 2013 we have also transferred the colonial administration files from former British territories to TNA. These two factors have led to an accumulation of special collection files.
In the majority of cases, the FCO has not in recent years made any formal assessment of the special collection files for sensitivity. Some files are likely to be sensitive, others we already know are very unlikely to be sensitive. Even for the non-sensitive files the amount of resource needed to transfer a file to the TNA is not insignificant since the files must be appraised, selected, catalogued and prepared for transfer. In the past this resource has not been available because of the concentration of effort on annual departmental and migrated archive files.
5. Departmental and special collection files
The FCO archive contains an estimated 1.2 million files, split approximately 50/50 between departmental and special collection files.
A departmental file is a file organised in line with the FCO’s normal filing sequence. [As described on The National Archives] (TNA) website](http://www.nationalarchives.gov.uk/records/research-guides/fo-fco-from-1782.htm), FCO files from 1967 onwards are arranged in regional geographical or subject groups reflecting their departmental organisation. Each of these annual departmental files is assigned departmental and functional codes as well as a code indicating the subject-matter. The vast majority of the archive departmental files held by the FCO (around 500,000) are not yet due for selection for permanent preservation and transfer to TNA.
Special collection files are those which do not conform to the above definition of a departmental file. Not every special collection file is old (although most are) and not every special collection file is overdue for transfer to TNA (again, most are). For example, the 1999 records amongst the FCO Board of Management papers in the archive inventory (dated 1999-2000) are not due for selection and transfer to TNA until 2021.
In practice, any file which has not been produced by the modern FCO will class as a special collection file, including Colonial and Commonwealth Relations Office files.
The distinction between special collection files and departmental files is useful mainly for administrative purposes relating to the processing of the files. The FCO’s Archive Management Team has a well-defined and well-established process for the review and release of annual departmental files. Special collection files will require a greater degree of initial appraisal before work begins to review and transfer the files. For instance, many special collection files will require a greater degree of special subject and/or linguistic expertise to review than departmental files and many will also require specialist repair because of their age.
In a few cases, we have categorised storage areas in the FCO’s archive inventory as “departmental” because they are used as temporary storage for records managers who are processing records (whether special collection files or departmental records). For instance, we have an area of 50 metres which is used for material being sensitivity-reviewed and for files being examined as a result of Freedom of Information requests. These temporary storage areas constitute a very small proportion of the total storage capacity of the archive.
6. Dates assigned to special collection and departmental files
The specialist contractor compiling the detailed inventory on the FCO’s behalf was asked to determine the date range of record series in the FCO archive. A record “series” is a collection of related records. An example amongst the special collection records is the Allied Control Commission series.
Some of the record series in the archive consist of large numbers of files, in which case we asked the inventory team to sample the date of creation of the files as accurately as possible to determine the date range. Such dates have been assigned for descriptive purposes so that FCO records managers understand the content and scope of archive record series we hold. It is not the purpose of the inventory to organise records by reference to Section 10.2 of the Public Records Act in terms of whether such files or records are due for transfer to The National Archives (TNA)
The vast majority of the departmental files we hold (around 500,000 of the estimated total of 600,000 departmental files) are not yet due for selection for permanent preservation and transfer to TNA. Most of the special collection records (but not all) are overdue for transfer. However, all files overdue for transfer to TNA are covered by a legal instrument granted by the Lord Chancellor specifically to enable the FCO to draw up a detailed workplan to deal with the special collection files in much the same way as the migrated archives were successfully handled.
View the Foreign Office’s Archive Inventory