Data Integrity Application for the Everyday User: Developing and Implementing Basic Integrity Checks Applied in Abu Dhabi During Times of Regional Conflict
Rebecca Pittam
 |
 |
 |
“Data Integrity Application for the Everyday User” is the story of NYU Abu Dhabi Library’s response to its local digital preservation needs, from a small-scale “storage to storage” transfer need addressed in the Fall of 2025 to a larger-scale “emergency full server backup” response in March and April 2026 during regional threats to US institutions in the UAE. This project involved NYU Abu Dhabi Library’s Manager of Digital Preservation, Rebecca Pittam, developing and writing a basic, custom data integrity tool that can be employed by the whole team with a click of a button. Developed with our small team of two archivists, one digital preservation manager, and two part-time library assistants in mind, the tool was conceptualized to be accessible across expertise levels. It is a tool that can be easily shared and utilized by small digital preservation teams like ours.
How does it work? Think of this tool as a high-tech security escort for digital files. Before assets are moved into the Library’s permanent digital vault (Preservica), the application gives every file a unique ""digital fingerprint"" (known as a fixity value). It then tracks these files as they move between servers. If a single bit of data goes missing or gets corrupted during the journey, the tool alerts the team immediately.
During Fall 2025, the tool was initially utilized to ensure that digitization work was correctly transferred from a working drive to a safe storage location and ultimately to the team’s staging server. The tool ensured that already existing files in the destination directory on the safe storage were not overwritten in a newer copy process, ensuring that files remained intact.
On an initial run to an empty destination directory, the tool creates a manifest of all files and their MD5 fixity values (remember, that digital fingerprint) that are placed in the destination drive. On any subsequent transfers, the tool first validates the MD5 fixity value against the manifest file to see if a file exists; if it does exist, that file is placed in a ""skip file"" process. Any file that does not exist has its fixity value created and placed in a ""pending"" process; once the initial scan of the source directory is completed, the files in the ""pending"" status are copied to the destination drive and the MD5 fixity values are appended to the manifest file.
A final phase of the tool scans the entire folder again in the destination drive and performs a validation of the integrity of the files. It looks for any files that may have undergone a name change while the fixity value remains the same, indicating the file’s integrity remains valid but a change has taken place on the destination drive. The tool also detects changes in fixity values that indicate either a corrupt file or a change in that file. Furthermore, the tool detects if a file is present in the destination drive but does not exist in the manifest, and if a file is in the manifest but is missing from the destination directory. Throughout the process, no files are replaced if an error is found, keeping the source and destination files available for comparison.
The tool then reports all results both in a pop-out window and in a “{DATE_TIME}_validated-md5.txt” report file that is saved in the destination directory alongside the files. Within this report file, the user is alerted to any issues and is asked to send the report to the Manager of Digital Preservation for investigation. By translating complex fixity errors into actionable alerts for non-technical staff, the tool democratizes the preservation process, ensuring that integrity monitoring is not bottlenecked by a single manager. To ensure long-term maintenance, the tool is accompanied by easy-to-read documentation, allowing future staff to deploy it without requiring the original developer’s presence.
All of these steps are automated, occurring with a click of a button, relieving the team of having to conduct manual checks.
Implementation and Impact: In September 2025, the initial tool was tested on a small scale by part-time assistants to transfer preservation assets from one storage device to a second safe storage device. On a larger scale, the tool was essential in transferring around two terabytes of data from on-campus server locations to external drives based off-campus over a VPN connection. During March 2026, the tool ran simultaneously on four machines over roughly a seven-day period. It was then shared and explained to team members to utilize should any file transfers be necessary during remote operations (which were necessary during a physical campus temporary threat closure). Following the remote operations period at the end of April 2026, the integrity reports generated by the tool allowed for a 100% audit of the transferred data, confirming that no assets were lost or corrupted during the emergency migration.
In terms of technical resilience, the tool’s ""skip file"" logic ensured continuity of work without data duplication. In terms of institutional continuity, the tool acted as the bridge between high-security on-campus servers and emergency off-site redundancy, maintaining a chain of custody that would have been impossible via manual drag-and-drop methods under duress.
We believe that this project should be considered for this DPC award because it exemplifies how a localized, accessible application can be implemented to safeguard institutional memory against extraordinary external pressures. This project also proves that you don't need a million-dollar infrastructure to be resilient—sometimes, the best solution is a smart, accessible tool that empowers an entire team to protect their heritage together. While many digital preservation solutions rely on complex, high-cost infrastructures – indeed, we also rely on this model as a Preservica client – this project illustrates that a tool simple enough for part-time staff but robust enough for a 2TB emergency migration is also an important form of resilience. By removing the bottleneck of specialist-only operations and establishing a cloud-independent redundancy, the initiative successfully maintained a 100% data integrity rate during a geopolitical crisis. It serves as a model that can be used by anyone looking for local solutions to maintenance and continuity.