Completed Investigations Based on Open Source Intelligence Sources

   Endangered large

Open source social media and web content that has been used to support the conclusions of crowd-sourced investigation and fact-checking in political or military conflict.

Group: Digital Legal Records

Trend in 2021:

Unanimous Decision

Added to List: 2019

Trend towards greater risk

Previous classification: Endangered

Imminence of Action

Action is recommended within twelve months, detailed assessment is a priority.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Social media sources relating to recent conflicts, such as in Ukraine.

Critically Endangered in the presence of Aggravating Conditions

Encryption; loss of authenticity; lack of preservation agency; limited or no digital preservation capability.

‘Vulnerable’ in the Presence of Good Practice

Offline backup captured by journalist or investigating authority; materials presented and documented in court; court able to deliver preservation; authenticity protected

2021 Review

This entry was added in 2019 and subsequently split into three elements by the Jury, relating to current, recent and historic sources.  This entry relates to materials used in evidence in completed investigations, as well as those presented to courts or other investigatory agencies. 

Social media companies have a policy to take down or suppress content that they consider to be propaganda for terrorist groups.  This has had the unintended consequence of deleting or supressing content that was being used in open source investigation or fact checking for journalistic or judicial purposes, and which may therefore be an impediment to refutation or prosecution.  However, a new generation of cloud based services now allow investigators to copy and stabilise content to private accounts in the process of investigating it: so the ethical requirements of social media companies and the integrity of investigation are both served.

The 2020 Jury notes that such content remains at risk, but the presentation of data to a court or prosecuting authority, or the publication through news media implies the introduction of a long-term preservation function.  The 2021 Jury agrees with this assessment and endangered classification, but has changed the trend towards greater risk in light of recent developments in crowd-sourced investigation and fact-checking

Additional Jury Comments

See: Higgins, E. (2019). Bellingcat and beyond. The future for Bellingcat and online open source investigation, iPres Conference 2019, Amsterdam, https://www.youtube.com/watch?v=kZAb7CVGmXM.

Read More

Born Digital Photographs and Video shared via Social Media or Uploaded to Cloud Services

   Endangered large

Digital images or video with no analogue equivalent and where the only copy is online with a social media platform or cloud image hosting service.

Group: Social Media

Trend in 2021:

Consensus Decision

Added to List: 2018

Trend towards greater risk

Previous classification: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within one year.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Flickr; Vimeo; YouTube; Instagram; Periscope; DropBox; Facebook; Twitter.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of preservation capacity in provider; lack of explicit preservation commitment or incentive from provider to preserve; lack of storage replication by provider; dependence on proprietary products or formats; poor management of data protection; inaccessibility to automated web crawlers; political or commercial interference; lack of offline equivalent; over-abundance; poorly managed intellectual property rights; lossy compression applied in upload scripts.

‘Vulnerable’ in the Presence of Good Practice

Offline backup; lossless compression; good documentation; accessible to web harvesting; clarity of intellectual property rights which enable preservation; credible preservation commitment from service provider; export pathway.

2021 Review

This entry was added in 2018 and was placed within a wider social media group in 2019 with other entries sharing common risk profiles and challenges, including dependency on global service providers whose business model can only be presumed and tied to users via asymmetrical contracts that favour the supplier. In 2019 it became a standalone entry given the distinct preservation challenges of images and videos in cloud services which need to be addressed. The 2020 Jury noted a trend towards increased risk based on cloud services with a low barrier to entry leading to use by agencies or individuals least able to respond to closure or loss.

While there has been development and wider use of pywb tools (e.g. the Webrecorder toolset) and API harvesting tools (twarc, TAGS, etc) and should say 'inaccessibility to automated web crawlers',  the 2021 Jury agreed with this continued trend towards increased risk, adding in light of recent global crises (predominantly the coronavirus pandemic, compounded by vaccine hesitancy, but also the increasing deterioration of the world's democracies) as a result of widespread misinformation, this category is now more significant and has a greater impact

Additional Jury Comments

The vast majority of content may be accessible for as long as the platform where it is hosted is popular (and has a viable business model); however, more insidious content (such as malicious misinformation or hate speech) may be deleted by content creators (potentially backed by hostile governments) to avoid prosecution or tracing. It is unclear to what extent these platform providers are compelled to provide access to servers / deleted content or private content for evidential purposes in the course of legal or criminal investigations. The lack of transparency and standardized international regulation of these platforms make their content vulnerable to exploitation and malicious use by individuals, corporations, and hostile governments.

Museums, Libraries, and Archives have begun to pay attention to this content through projects like Collecting Social Photo (CoSoPho), but no breakthroughs have been made. See: https://www.collectingsocialphoto.org/en/home

Case Studies or Examples:

Read More

3D Digital Engineering Drawings

   Endangered large

3D digital engineering models produced as part of building or engineering design processes. The production of such drawings has progressed from a digital analogue of paper to complex digital environments such as BIM (Building Information Modelling) which combine original drawings, libraries of compound objects, and links to external data sets such as about the performance of materials and maintenance of parts.

Group: Engineering

Trend in 2021:

Consensus Decision

Added to List: 2017

Trend towards greater risk

Previous classification: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within one year.

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Building Information Management, Computer Aided Design, Product Data Management in engineering and architecture,

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of preservation mandate or collecting institution; lack of preservation capability in data owner; irregularities in supply chains; complex or long data supply chains; dependencies on proprietary software or formats; lack of persistent identifiers; poorly managed IPR; temporary joint-venture companies; poor records management; poor regulatory compliance; encryption.

‘Vulnerable’ in the Presence of Good Practice

Well managed data infrastructure; preservation from the point creation; carefully managed IPR; persistent identifiers; well managed records management processes; recognition of preservation requirements at highest levels; strategic investment in digital preservation; host clearly identified; participation in digital preservation community.

2021 Review

This entry was first submitted in 2017 when the Jury lacked the capacity to consider it in detail at that time. In 2019 it was assessed with additional expertise co-opted, with the decision for it to remain a very broad category, including major one-off construction and engineering projects, a long tail of more minor building programmes, as well as large volume but homogenous production processes in engineering. Both the 2020 and 2021 Jury agreed with its endangered status, with the key consideration being that the lifecycle of the products and the data which describes them vastly exceeds the short lifecycles of the infrastructures on which they are designed. This challenge is compounded by supply chains that may involve many different stages of production, as well as the delivery of large projects through transitory joint ventures companies that have no residual mechanism or capacity to preserve the data thereafter. Although there have been advancements in the development of new preservation tools and techniques for these materials, the 2021 Jury noted recent examples of the loss of 3D architectural drawings (see additional jury comments below) which have had huge impact especially at the local level, as well as significant impacts on infrastructure, travel, and how people interact with built environments throughout the world. For these reasons, the trend has moved towards greater risk.

Additional Jury Comments

Data in this category enables the safety and security of critical infrastructure, but the responsibility to maintain data is unclear, nor are retention periods clear. Although examples of good practice exist, the extent to which there are working solutions at large seems doubtful, and it is surprising that there are not more diverse success stories to report.

 

Case Studies or Examples:

  • The Grenfell Tower Inquiry offers a case to consider how the loss of 3D Digital Engineering Drawings can have a huge impact, especially at the local level. For example, if Grenfell had been done using 3D technologies, do we have confidence that those materials would have been adequately preserved? What would have been the local impact? What would have been the impact on the inquiry? See: https://www.grenfelltowerinquiry.org.uk/

Read More

Master Digital Music and Sound Recordings

   Endangered large

Master recordings of music and other performance from which retail products are derived, typically in multiple tracks and uncompressed high-resolution sound quality

Group: Sound and Vision

Trend in 2021:

Unanimous  Decision

Added to List: 2019

No Change

Previous classification: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within one year.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a small effort to address losses in this group, requiring the application of proven preservation tools or techniques.

Examples

Master recordings owned by music industry

‘Critically Endangered’ in the Presence of Aggravating Conditions

Single point of failure; storage on old or degrading media; lack of ongoing investment in changing preservation requirements; lack of capability; poor documentation; dependence on small staff

‘Vulnerable’ in the Presence of Good Practice

High quality storage; meticulous and consistent replication; trusted repository; preservation requirement understood at executive level and funded accordingly; leadership in preservation community; expert staff

2021 Review

In 2019, this entry was created a subset of a previous 2017 entry, ‘Digital Music Production and Sharing,’ which was split to draw attention to the different challenges faced by the different forms. Though it has overlaps with other entries including Digital Archives of music production, it is a separate entry to emphasise the inherent and very great value of master recordings over and above those distributed, and the concomitant need for active preservation.

Additional Jury Comments

The imminence of action will depend on format and age, and the significance of loss may be more largely felt if recordings of a major recording star

This is interesting as the recording houses should be seeing the value of these - so why are they not taking responsibility for looking after them? Do they not feel it is in their financial interests? The archival practices of the studios are typically based on value - the recordings are assumed to be worth keeping. However, this means relatively low-value masters may not be transferred to new media in a timely way and could be lost. There is no comprehensive deposit scheme to address the long tail of music production, and it is often unclear exactly where responsibility lies.

Case Studies or Examples:

Read More

Digital Radio Recordings

   Endangered large

Master recordings of radio broadcasts generated live but often poorly stored thereafter, for example offline recordings on single LTO (Linear Tape Open) Tapes

Group: Sound and Vision

Trend in 2021:

Consensus Decision

Added to List: 2017

No Change

Previous classification: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within one year.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Broadcast archives of UK commercial local radio;

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of archival mandate; lack of capability of archive; lack of policy or capacity within broadcaster; small or unprofitable broadcaster; concern over intellectual property rights; overzealous rights management protection; device or software dependence; dependence on proprietary or obsolete formats; lack or loss of documentation; little use or inaccessibility; storage (typically tapes) older than warranty; lack of media refreshment plan; lack of error or integrity checking process; single copies

‘Vulnerable’ in the Presence of Good Practice

Archival responsibility accepted and acted upon; replication; refreshment of media; good documentation; active in digital preservation community; trusted repository; content re-used

2021 Review

This entry was added in 2017, and added as a separate entry due to concern over recordings on LTO tapes. These provide between 15 and 30 years’ storage which may be less depending on usage and storage conditions. LTO1 and LTO2 which were released in 2000 and 2003 respectively are now entering the final phases of viability. Reader compatibility may be more problematic than media resilience, however. Drives supporting newer releases of the format are typically only compatible within two generations and experience with the recently released LTO8 suggest that it is only backwardly compatible to one generation.  Therefore one major national archive and library has decided to expedite migration away from LTO6 which is becoming obsolete more quickly than anticipated. Through time, the risks to collections that have not been refreshed or replicated from early LTO tapes expand. Thus, the overall trend is towards greater risk when collections are not migrated. Older formats, perhaps as recently as LTO6, extinction events should be anticipated within two to five years. The 2021 Jury agreed with the endangered classification,  noting the importance of a selective approach.

Additional Jury Comments

Depending on the legislative context, digital radio recordings may fall under published works if they are broadcasts. Additionally, if the recordings are broadcast by a taxpayer-funded broadcaster, record-keeping guidelines may already exist to advise how long content be kept, which would then inform a selective approach to caring and looking after them for as long as required.

While broadcasters may keep their own programmes, they are often not comprehensively collected or archived by memory institutions as contributions to cultural heritage.

Read More

Current Portable Solid-State Media

   Endangered large

Materials saved to flash drives or other solid-state media in the last five years where the reader devices are still supported and can be integrated easily into hardware infrastructure

Group: Portable Media

Trend in 2021:

Unanimous Decision

Added to List: 2019

No Change

Previous classification: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within one year.

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a small effort to address losses in this group, requiring the application of proven preservation tools or techniques.

Examples

USB memory sticks; flash cards in cameras; solid state portable hard disks

‘Critically Endangered’ in the Presence of Aggravating Conditions

Poor storage conditions; encryption; digital rights management; lack of replication; lack of documentation; lack of periodic testing; lack of refreshment pathway; lack of access to readers; out of manufacturers’ warranty or no warranty; storage in paper files.

‘Vulnerable’ in the Presence of Good Practice

Regular review and testing; replication; refreshment plan; comprehensive documentation; high quality storage; regular maintenance of readers; multiple readers available;

2021 Review

This entry was added in 2019 to ensure that the range of media storage is properly assessed and presented.

Additional Jury Comments

This entry is highly dependent on who is looking after the portable media but made more difficult over time. The lack of granularity in the definition means that only general advice can be offered, such as to refresh media. In time, it may yet be more useful to split all storage media (maybe 100 items long) with an indication of how long these can be expected to last. In many cases, specialists can recover obsolete media, but the cost of employing them can become an aggravating condition.

Read More

Current Portable Optical Media

  Endangered large 

Materials saved to DVDs, CDs or other optical media in the last five years where the reader devices are still supported and can be integrated easily into hardware infrastructure

Group: Portable Media

Trend in 2021:

Unanimous Decision

Added to List: 2019

No Change

Previous classification: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within one year.

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a small effort to address losses in this group, requiring the application of proven preservation tools or techniques.

Examples

CDs, DVDs produced in the last five years

‘Critically Endangered’ in the Presence of Aggravating Conditions

Poor storage conditions; encryption; digital rights management; lack of replication; lack of documentation; lack of periodic testing; lack of refreshment pathway; lack of access to readers; out of manufacturers’ warranty or no warranty; storage in paper files.

‘Vulnerable’ in the Presence of Good Practice

Regular review and testing; replication; refreshment plan; comprehensive documentation; high quality storage; regular maintenance of readers; multiple readers available;

2021 Review

The judges have introduced this entry to ensure that the range of media storage is properly assessed and presented.

Additional Jury Comments

This entry is highly dependent on who is looking after the portable media but made more difficult over time. The lack of granularity in the definition means that only general advice can be offered, such as to refresh media. In time, it may yet be more useful to split all storage media (maybe 100 items long) with an indication of how long these can be expected to last. In many cases, specialists can recover obsolete media, but the cost of employing them can become an aggravating condition.

Read More

Scroll to top