David is a senior digital archivist at The National Archives in the UK.
Original development
With DiAGRAM (the Digital Archiving Graphical Risk Assessment Model) we are trying to break down barriers to digital preservation in two ways: firstly, to help archivists make progress with digital preservation by showing them which interventions will have the greatest impact in reducing the risks to their digital materials; secondly by making sure that the tool itself does not present barriers to being used by anyone who needs it.
The first point was a key point of our original bid to the National Lottery Heritage Fund (NLHF). Most of us know that there are many actions that we could take to help preserve the digital material in our care but none of us have the money, systems, or people, to do all them. For on their digital preservation journeys, this can lead to a form of “analysis paralysis” as it can be hard to determine which of the possible steps available to them will provide the biggest immediate risk reduction.
When we began the project we had done enough work to be fairly sure that the Bayesian Network approach would be useful. However, we weren’t sure what our collaborators from the Applied Statistics & Risk Unit (AS&RU) at the University of Warwick called an Integrated Decision Support System (IDSS) might look like in the context of helping archivists explore the risks around digital materials. Fortunately, AS&RU had two computer science students from Monash University on placement. Using the statistically-focused programming language R, coupled with the Shiny package for developing web applications, Stephen and Sidhant were able to quickly build the initial prototype, based around the Shiny Dashboard package which makes rapid development of statistical visualisations more straightforward. This helped the archivists involved in the project understand how the online tool might help. However, at this stage the model built into the tool was an example, not the digital preservation model we were aiming for. We were working on creating the real model and gathering the data needed to populate it in parallel to the work on the prototype tool.
With discussions on the underlying network of risks completed, the structured expert elicitation also carried out and our data finalised, we could load the actual digital preservation model into the tool and start sharing it more widely to get feedback on this initial prototype.
The original screen for entering information on the level of an archives' technical skills - what did the percentages actually mean?
We did this via a series of online events in summer 2020. It soon became clear that while users could quite quickly see the potential of the tool to help them, there were a few stumbling blocks that they still needed to be helped over. In particular, at this stage to setup the specific model for an individual archive required entering of percentages for input values such as the level of technical skills available to the archive. Users weren’t clear how to do this consistently. Similarly, the users weren’t quite sure whether the scores they were getting at the end for the risks to their archive were “good” or not.
Building on the original prototype
By this time Stephen and Sidhant had long since completed their placement at Warwick and had returned home (fortunately before Australia closed its borders due to pandemic). However, we were actually able to turn the situation created by the pandemic somewhat to our advantage. We had initially anticipated that the various events needed for the project would be face-to-face, so our funding from the National Lottery Heritage Fund (NLHF) included amounts for travel expenses and accommodation for participants, as well as venue hire for dissemination events. Fortunately, NLHF agreed that this money could be repurposed to undertake further development of the online tool.
We were able to commission R and data visualisation specialists Jumping Rivers to undertake this work. It was also fortuitous that a new user researcher, Pauline, joined The National Archives at this point, and was available to support the work. Together we came up with ways of changing the initial model setup to a series of questions based on what should be familiar existing models, such as the NDSA Levels of Preservation and DPC RAM, and converting those into the percentages needed for the underlying model.
How good is your mental arithmetic, can you make sure your percentages add up to 100?
We also wanted to make the experience of using the tool more supportive: for example, the initial prototype would flag it as an error if the values you entered for the split of your digital materials been born digital, digitised and surrogate did not add up to 100% but you had to do the arithmetic yourself to work out what you needed to change. Following the development work it impossible to enter values that do not total 100%, once you’ve set the first two values the third value will be calculated automatically.
This type of work also fed into thinking about the broader accessibility of the tool and how possible it was for people to use it if they used only the keyboard for data entry, or if they used assistive technology such as screenreaders. We should really have thought about this earlier due to The National Archives’ desire to be an Inclusive Archive (and the UK’s rules on accessibility for public sector websites).
At this point we only had budget for six weeks’ work on the tool. Those who saw our presentations on the initial prototype and helped us with user testing will (I think) agree that we were able to use this to make a huge difference to the experience of using it. However, we couldn’t fix all the accessibility issues in that time, especially as some seemed rooted in the initial choice of Shiny Dashboard and some of the elements used to produce graphs.
The revised prototype: it's now impossible to enter percentages that don't add up to 100%
Further development
At this point the funded project was coming to a close and it was not clear where future funding for further development would come from. However, we used the modelling functions of the tool to provide evidence for The National Archives’ own submission to the 2020 UK government spending review. The success of this submission, and the importance of the evidence from DiAGRAM, meant that internal funding has been made available to undertake further work.
I took on the responsibility of being product manager for DiAGRAM in April, and initially reviewed the various ideas for further development and outstanding issues to produce a draft road map for development. We then commissioned TetraLogical to undertake a formal accessibility review of DiAGRAM. TetraLogical use a mix of automated testing tools and testers who are regular users of assistive technologies to test websites for accessibility. They produced a breakdown of the areas where DiAGRAM does not meet accessibility requirements. Following a formal tender process through the UK government’s online digital marketplace we re-engaged Jumping Rivers to undertake further development. This began early last month with a review of the issues reported by Tetralogical in order to determine how difficult they would be to resolve, and to what extent re-architecting of DiAGRAM might be required to allow that.
We expect to undertake some further user testing soon to help inform these developments and ensure that DiAGRAM is as useful as possible to those looking to understand their digital preservation risks. Depending on the difficulty of resolving the accessibility issues we may also be able to work on the user experience of the advanced modelling features which we did not have time to work on in detail during the original project. Wide input from digital preservation professionals will also be of key importance for this so we hope that people will be as generous with their time as they were previously: keep an eye out for the request for testers!
In the meantime, try out DiAGRAM for yourself, read more about the original project Safeguarding the nation’s digital memory, read the journal article describing the development of DiAGRAM in more detail, or read the project evaluation report produced by the DPC
Read more...