The National Archive’s new Transfer Digital Records Service: The Hard Drive forward

Kirsten Arnold is Service Owner for Digital Selection and Transfer at The National Archives in the UK

Hi. I’m the Service Owner for Digital Selection and Transfer (DST), working within Digital Archiving at The National Archives (TNA). DST works closely with records creators to help them understand, provide advice and facilitate the transfer of their selected digital public records to The National Archives. These record creators specifically are public record bodies, and in the main UK government departments.

Part of my portfolio as DST’s Service Owner, is looking at ways in which we can better improve our current processes. This includes creating and delivering a transfer service that is robust, secure and intuitive, enabling government departments to deliver their born digital records to us in accordance with the Public Records Act. So this blog is about what we’ve been working on that aspires to achieve this and the future of how digital records will be transferred to The National Archives. 

So in 2019 four of us completed an Agile Discovery, which for the next stage (Alpha) we grew into a full multi-disciplinary Agile product team (of which I am also temporarily acting as Product Owner) to develop an automated digital service that would make it faster and easier for Government to fulfil their Public Record Act obligations to transfer records to TNA.  The benefit being not only to improve the transfer experience for government but to receive a better quality of record. This product is called Transfer Digital Records (TDR)

Home page of TDR

To understand where we needed to start we wanted to identify the current painpoints and/or blockers are users were experiencing. Following the appraisal, selection and sensitivity review of the selected records, the transferring body will inform us they are ready for transfer. Before the records are sent to us at TNA, the DST team ask them to complete a number of mostly manual steps: firstly to complete a ‘transfer’ form, this provides us mainly with cataloguing and compliance information; install and use tools, such as DROID to check and prepare the records, identify formats and generate a metadata spreadsheet. Following this, we ask them to save their records to hard drives that we send them, using a specific structure for the records and supporting metadata. Once received at TNA the records are sanitised, re-checked and often returned to the department when problems are identified. This gives rise to lengthy feedback loops and duplication of work effort. This manual digital transfer process is slow, laborious and technically complex.

In contrast to the manual process, TDR offers a browser-based preparation area where records are uploaded and prepared for transfer aiming to digitalise some of these extant steps.

Specifically,  once a user logs into TDR, they are asked to confirm agreement to a short number of statements pertaining to compliance and to the current parameters we have in place (For example in this beta private release we are temporarily only accepting open records). They are then prompted to upload their data, by selecting a directory on their local file system containing the folder they wish to upload. This triggers client-side metadata harvesting, achieved via the browser’s file API and SHA256 checksum generation. This also triggers three concurrent processes that are run in parallel across the uploaded records:

• Anti-Virus – anything that fails this step is moved to a ‘Quarantine’ S3 AWS bucket (as opposed to the ‘Clean’ S3 bucket)

• Checksum validation –generates a new SHA256 checksum and compares it against the checksum harvested pre-upload to ensure that data upload has completed without error 

• File Format Identification – achieved through running The National Archives’ own DROID tool across the ‘consignment’ of uploaded files 

In the next iteration we release, the user will next be able to review a list of records uploaded and add custom, descriptive and closure metadata at this stage. However for now, all they can do is confirm that they wish to continue with the transfer. This then triggers the ‘export’ process. The record data (retrieved from the ‘Clean’ bucket) and metadata (retrieved from the database) is packaged up in a BagIt package and then placed in an ‘Export’ S3 bucket. 

In effect TDR replaces the manual process with automation, hard drives with file transfers, automatic validation and processing instead of client side software installation, reduces feedback loops and makes for a far simpler and easier user experience with a better quality archived record at the end of it. Before TDR automated and streamlined the process, it was not unusual for a digital transfer to take over a year, from the point at which the department identified records of long term value to those records being safely archived at TNA. This reflects the difficulty departments experience in following this laborious and manual process, the support required from TNA staff, the level of technical skill required, and the sensitivity and complexity of the records being transferred. It is also a reflection on the service we offered which did not, at that time, put users first. TDR (Transfer Digital Records) aims to address our users’ pain points and reduce the time taken to transfer digital public records into The National Archives. We aim to improve both the speed and ease of digital transfer alongside improving the user experience and delivering a more complete and better structured record to TNA for preservation and access for future generations.

User needs, engagement, testing and feedback have been and continue to be absolutely critical to our product. TDR is user centred; we have been working with real users from the beginning.

We are building TDR using an agile approach and as per Government Digital Service Agile Delivery guidance the team have followed the following stages:  Alpha -> Private Beta -> Public Beta -> Live; currently at the private beta stage, which we released in July of this year. During this stage the service is only available to an invited group of users transferring ‘open’ records which can be released to the public immediately. We’re developing iteratively so this early version is by no means feature-complete, but we are continuing to enhance the service as further user requirements are surfaced. Over time, we will make TDR available to more departments and will continue to develop the service incrementally in consultation with our users, to add support for sensitive records and more complex types of records, such as redacted and retained records. To date TDR has received records from three ‘pilot’ transferring bodies who have been working with us during our private Beta stage. We are aiming to release as a public beta product next year meaning any transferring body will be able to access and use TDR to transfer their digital public records. When this service is live TDR will entirely replace the existing digital transfer process.

We operate under an ethos of openness, all our code is open source, available in Github repositories and we encourage re-use. The Government Service Standard demands that code is published under an open source licence (for TDR we use MIT). Additionally, all the work we do as civil servants is Crown copyright. You can find a list of the TDR open repos   here: https://github.com/nationalarchives/tdr-dev-documentation#tdr-repos

TDR also conforms to GDS Service Design principles, NCSC Security for Cloud principles, AWS Well Architected framework and has been built to meet WCAG 2.1 AA standards for accessibility.