Hugh Campbell works in Digital Preservation & Information Systems for the Public Record Office of Northern Ireland
Doesn’t time fly when you’re waiting for your slot on the DPC blog rota to come around?! I had spent some time thinking about different aspects of digital preservation before deciding on a topic and drafting my blog. Then, a few days before submission, comes an email from Jen Mitcham asking politely if I had considered writing about Electronic Document and Records Management (EDRM) as I am a member of the DPC’s EDRMS Preservation Taskforce. A swift change of course later and here are some of my musings on my EDRM experience.
The Northern Ireland Civil Service’s (NICS) corporate EDRM system, RecordsNI, is expected to be one of the main contributors of records to the Public Record Office of Northern Ireland’s (PRONI) digital repository. ‘Expected’ because Departments have not transferred any material from RecordsNI to PRONI yet. The time for such transfers, however, is getting closer. The NICS selected TRIM as the software platform for the corporate EDRM system following a procurement exercise in the early 2000s. TRIM has subsequently evolved through a number of manifestations (HP TRIM, HPRM, HPCM, Micro Focus Content Manager). The NICS currently uses HPRM 8.3 and plans to migrate to Micro Focus CM9 later this year.
A number of PRONI staff were involved in the central project and PRONI was one of three lead implementers of the system. This proved to be very beneficial as we had a member of staff who was interested in records management and therefore became the prime candidate for the role of system administrator for the PRONI implementation. This afforded us a great opportunity to learn about the product particularly as we had someone with higher privileges than regular users.
We were able to use the out of the box export and very quickly determined that it was not for us because it output:
- Metadata in an xml file, something we didn’t have any skills in at that time;
- Metadata by the truckload – it didn’t appear to be configurable with the result that there was so much of it that it was overwhelming;
- Internal database field names, not the labels visible to users, so it was very difficult to determine what exactly you were looking at.
We worked with the Managed Service Provided and specified what we wanted from a tailored export. We went through the various metadata fields as best as we could, discussed the potential usefulness of them and decided (for better or worse) upon the ones to be exported along with the records. Some fields were very straightforward to choose, e.g. description, container (folder) title, whereas some others required lengthy discussion, e.g. dates. We spent many hours looking at dates and trying to work out what options we had and what they meant. ‘Date created’ has always been a thorny issue for us and one that we haven’t resolved. I say this because we have two different date created values for records exported from the EDRM system, neither of which are what any reasonable person would consider to be the true date ‘created’. We also decided that we would only take the ‘final’ version of a record or any version that was marked for preservation. A list of the fields we export is contained within our Submission Guidance.
The export:
- Copies each container selected out into a Windows folder on the file system, and;
- Creates a metadata csv file within each folder, with one row of metadata for every object within the folder.
As part of our processing, we will supplement this with more metadata generated by DROID and we will populate the ‘PRONI use’ fields.
To date, two Public Inquiries have transferred over 51,000 records from HPRM to us; proof that the process works, albeit with some issues. We somehow managed to get the EDRM metadata file out of sync with our master metadata template and we are still paying the price for that mistake. This will be addressed in the not too distant future hopefully. It does mean, however, that there’s going to be some scripting required in order to process the material that has been exported to us from HPRM.
Another issue is going to be the way in which we generate relative filepaths within the metadata file. The plan was to follow the pattern of Accession Number\data\transfer identifier\ContainerId\object. It has now become apparent that there is a flaw in this brought about by the fact that we want to group together a number of folders for processing purposes. The export assigns a different transfer identifier to each set of data exported by the user. Therefore, in order to prevent our processing from failing, we would have to mirror the grouping selected by the user at the point of transfer. There will be many instances where this would not cause a problem but there is, however, one transfer where it would be a serious problem. For some unknown reason, the user was only able to select one folder at a time to transfer…so 834 transfers later it was all over. That means 834 different transfer identifiers which in turn forms part of the relative pathname. We could process this as 834 separate accessions but somehow I see a bit of scripting to remove the need to do that.
One final area for improvement is the analysis of the content within the EDRM system before it transfers. This hasn’t happened to date and the result has been that some records on our black list transferred across, namely .zip files and password protected files. Such files are problematic and it would be much better if they could be detected and addressed at source. Unfortunately we don’t have the means to carry out such an analysis before transfer so we will look to reinforce it in our transfer guidance.
To find out more about PRONI, please visit our website and follow us on Facebook and Twitter.
To find out more about the DPC's EDRMS preservation task force take a look at these blog posts on the topic {Who wants to work with us on EDRMS Preservation? and How do you preserve records from within an EDRMS?} and watch this space for further posts on the topic, a DPC event in 2021 and the launch of some online outputs and resources.
Read more...