PDF other than PDF/A

   Endangered large

Documents presented in PDF (Portable Document Format) format (ISO 32000:1 and ISO 32000:2) and other data wrapped inside them, other than PDF/A but including all other variants and versions.

Group: Formats

Trend: No change

Consensus Decision

Added to List: 2017

Last update: 2018

Previous category: Of Concern

Imminence of Action

Action is recommended within five years, detailed assessment within three years.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world..

Effort to Preserve

It would require a small effort to address losses in this group, requiring the application of proven preservation tools or techniques.

Examples

PDF 1.1, 1.2, 1.3, 1.4 (excluding PDF/A as a subset), 1.5, 1.6, 1.7 and 2.0. PDF/X and PDF/E

‘Critically Endangered’ in the Presence of Aggravating Conditions

Loss of context; loss of authenticity or integrity; external dependencies; poor storage; lack of understanding; significant diversity of data; poorly developed digitization specifications; lack of integrity checking; poorly developed migration or normalizations specifications; lack of virus control; poor storage or replication; lack of validation at the point of creation; encryption.

‘Vulnerable’ in the Presence of Good Practice

Well managed data infrastructure; preservation planning; authenticity managed; use of persistent identifiers; reduction of dependencies; application of records management standards; recognition of preservation requirements beyond formats; strategic investment in digital preservation; preservation roadmap; participation in digital preservation community; format validation.

2019 Review

The judges have introduced this entry as a subset of a previous entry for ‘PDF’, emphasizing the different threats faced by different types of PDF. PDF/A explicitly reduces dependencies and thus curtails preservation risks for certain types of content: PDFs of other types do not. PDF and PDF/A have sometimes been misunderstood as a generic solution to digital preservation requirements. In the eyes of the judges it can only offer a preservation solution when embedded within a wider preservation infrastructure.

Additional Jury Comments

See also:

Fanning, B (2017) Preserving with PF/A (Second Edition), DPC Technology Watch Report 17-01 online at http://dx.doi.org/10.7207/twr17-01

Read More

Premium or institutional social media services

   Endangered large

Commercial social media services which are based on a costed subscription and contract.

Group: Social Media

Trend: no change

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Premium versions of Vimeo, Flickr, Yammer, Slack, Mircosoft Teams and others.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Unstable business models and pricing schedules from providers; lack of export functionality; unstable terms and conditions; lack of onsite copy of key media; lack of strategic plan for IT provision; confusion on IPR; asymmetrical contract arrangements; conflating preservation and access.

‘Vulnerable’ in the Presence of Good Practice

Offline back up for key media; fit to preservation and records management plan; strategic roadmap for adoption of social media;

2019 Review

The judges have introduced this entry as a subset of a previous entry, emphasizing the different threats faced by services that are ‘paid-for’ versus ‘free-at-the-point-of-use’. Both depend on the business model of the vendor and the terms and conditions which they impose. For this group, the business model and sustainability are more obvious and contracts may be enforceable more readily. Moreover, because these services have a slightly higher barrier to entry they may be favoured by agencies better able to respond to closure or loss.

Additional Jury Comments

Traditional web archiving can be employed where the user pays for a service but content is ultimately publicly available (such as Flickr) where the user may pay for a full account to be able to host more photos publicly. But much is unclear about how to preserve internal social media / closed networks that web archiving can't get to or existing tools don't cover. The growth in use of these products for communication and social networking leads us to think this is endangered rather than vulnerable. It’s possible that existing tools could be modified to tackle some of these closed networks but that is likely to require some investments, perhaps related to corporate records in some cases (thinking about internal Slacks, for instance), and more education about the importance of preserving this material and not trusting the publishing platforms to host the content forever.

Read More

Pre-Production TV and Movie Materials

   Endangered large

Digital records of the creative and production process for film and television, such as initial designs, screenplay and script, on set still photography, rushes or out-takes that are not included in the final production and therefore not available to on-air broadcast archives or film libraries

Group: Sound and Vision

Trend: New Entry

Consensus Decision

Added to List: 2017

Last update: 2019

Previous category: Of Concern

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

TV and Movie production archives in digital form; outputs of script management software; drafts of screenplay; continuity photography; costume design; set design; lighting and sound design.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of custodial responsibility; confusion over intellectual property rights; lack of appraisal; lack of recognition of preservation at executive level;

‘Vulnerable’ in the Presence of Good Practice

Preservation responsibility understood and acted upon; preservation infrastructure and planning for key items; access and use of collections to inform subsequent productions

2019 Review

This entry was first proposed in 2017 and noted as being ‘of concern’, though the Jury did not have the capacity to assess the entry thoroughly. Additional expertise has been recruited to the Jury this year to help with the assessment.

Additional Jury Comments

These materials are not being collected in any coherent way and will be lost as they are not valued by production companies, the space needed to hold them is costly and the expertise needed to catalogue and collect them is lacking. In short, nobody really cares. This is not primarily a technical problem and advocacy is needed urgently.

Read More

Proceedings and Evidence in Court

   Endangered large

Digital materials presented in court as evidence or documents such as rulings and proceedings generated through legal proceedings

Group: Digital Legal Records

Trend: No change

Consensus Decision

Added to List: 2017

Last update: 2018

Previous category: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Evidence submitted to courts of all kinds including text messages, photography, CCTV, email, 3d and 2d scanning, scientific reports and analyses, documents and websites; digital record of proceedings; digital records of rulings and all manner of quasi-judicial proceedings and tribunals.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Loss of context; loss of integrity; external dependencies; poor storage; lack of understanding; churn of staff; significant or diversity of data; poorly developed specifications; ill-informed records management; poorly developed transfer protocols; poorly developed migration or normalization; longstanding protocols or procedures that apply unsuitable paper processes to digital materials.

‘Vulnerable’ in the Presence of Good Practice

Well managed data infrastructure; preservation enabled at ingest; carefully managed authenticity; use of persistent identifiers; finding aids; well managed records management processes; recognition of preservation requirements at highest levels; strategic investment in digital preservation; preservation roadmap; participation in digital preservation community.

2019 Review

This entry is a subset of an entry made in 2017 for ‘Digital Legal Records and Evidence’ which the Jury has split into four more discrete entries. This category includes evidence that has been presented as evidence in court, as well as court proceedings. It recognizes that courts are not limited in the types of evidence that they can admit but that they have a responsibility to provide robust preservation that ensures the authenticity of their records and evidence.

Additional Jury Comments

Standard Records Management processes within designated agencies should be able to take care of the preservation of materials like this but given that evidence is likely to involve complex types of data, such agencies may not be equipped to deliver preservation effectively. It is surprising that courts are not more obvious in the digital preservation community, where solutions now exist.

More concrete examples would be welcome. It is the evidentiary value of submissions to court that may be lost, and therefore veracity of decision could be questioned. Evidence submitted in digital form is of a greater risk (e.g a video file submitted on a CD in the 90s) than records of the proceedings themselves (e.g. transcripts).

Read More

Recordings of video game play uploaded to online platforms

   Endangered large

Recordings of game playing and e-sports that show how games are experienced and played, especially multi-user online games and tournaments.

Group: Gaming

Trend: Improving

Consensus Decision

Added to List: 2017

Last update: 2018

Previous category: Critically Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within one year.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a major effort to prevent losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Material uploaded to Amazon Twitch, game channels on YouTube and other playback services

‘Critically Endangered’ in the Presence of Aggravating Conditions

Controversies around intellectual property rights; lack of offline backup; changing business model of providers; limited recognition of cultural and historic value of game play; over dependence on goodwill subsidy of ad-hoc community; lack of preservation know-how at service providers; dependency on bespoke hardware or interfaces.

‘Vulnerable’ in the Presence of Good Practice

Offline backup; managed intellectual property rights; players and audiences invested in data

2019 Review

This is a subset of an entry made in 2017 for ‘Gaming’ which the Jury has decided to split into four more discrete entries. It has overlaps with an entry on Consumer Social Media except this category specifically draws attention to gaming and e-sports. It is a subset of both. By including as a separate entry,the Jury encourages greater consideration of the cultural and historic value which such recordings are likely to acquire as well as the technical and economic challenges to preservation. It also notes that this entry is categorised as a lower risk than the entry in 2017: but this is a function of splitting that entry into components.

Additional Jury Comments

Vulnerable is an appropriate classification. The content is not particularly distinctive in technical terms but there are aggravating circumstances, namely an almost complete reliance on commercial third parties (Google/YouTube and Amazon/Twitch) for the infrastructure around video capture and hosting. Copyright claims on video content by publishers such as Nintendo - while less prevalent now than a few years ago - also complicate things. The significance of loss here is high because recordings, including commentary, and onscreen interactions with other players, seem likely to be the best way of preserving the experience of playing certain games at certain times. We are familiar with the challenges of preserving video, but we need to think about how established approaches will work in the context of the aggravating circumstances outlined above. There is a degree of urgency associated with working out how (legally and technically) preserving the materials that they hold may be preserved.

Read More

Records of Quasi Non-Governmental Agencies

   Endangered large

Records from agencies at arms-length to government whether locally, nationally or internationally. They may be required to maintain archives for the purposes of transparency, sometimes for extended periods, and sometimes in diverse and complicated forms. But because they are at arm’s length to government but the ’QuaNGO’ or ‘ALEO’ (Arms-Length Executive Organization) may lack the capacity to meet complex digital preservation requirements that arise, nor be able to deposit in the government archive.

Group: Public Records

Trend: no change

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: 2019

Imminence of Action

Action is recommended within three years, detailed assessment within one year.

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to prevent losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Records of non-executive state or national agencies; museum or leisure trusts; industry or public regulators; public audit services; public-good funding and investment agencies; autonomous and semi-autonomous public agencies; sovereign wealth funds; public/private partnerships; publicly owned companies.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of preservation infrastructure; conflation of backup with preservation; loss of authenticity or integrity; Long-lived business processes; poor storage; churn of staff; significant volumes or diversity of data; poorly developed digitization specifications; ill-informed records management; poorly developed migration or normalizations specifications; longstanding protocols or procedures that apply unsuitable paper processes to digital materials; encryption; political instability; lack of sustained funding.

Vulnerable in the Presence of Good Practice

Well managed data infrastructure; preservation enabled at the point of creation; carefully managed authenticity; use of persistent identifiers; finding aids; well managed records management processes; application of records management standards; recognition of preservation requirements at highest levels; strategic investment in digital preservation; preservation roadmap; participation in digital preservation community.

2019 Review

This is a new entry which the Jury introduced in 2019 based on an earlier larger entry for ‘Records of long duration from Local Government or Other Government Agencies’. The split is intended to allow greater concentration on the challenges that these different types of agency face.

Additional Jury Comments

Although the split draws attention to the different pressures faced by QuaNGO’s it could be further subdivided into legally required public records and additional information that may enrich our digital preservation of society. The classification assumes that the roles and requirement for records management is clearly defined, but if this is not the case or there is inadequate resource to match the requirement, then the risk goes up.

Read More

Published research data appended to journal articles

   Endangered large

Closed research data sets produced and documented in accordance with good practice and simply appended to a journal article or transferred to a repository which does not have sufficient subject-matter expertise or funding commitment to ensure reliable or ongoing preservation for the long term.

Group: Research Outputs

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within five years, detailed assessment within three years.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a small effort to prevent losses in this group, such as the deployment of proven preservation tools or techniques.

Examples

Data sets added to papers in repositories that are designed primarily for papers; electronic journals offering data sets without obvious preservation capacity; institutional repositories servicing highly complex scientific data sets with insufficient subject-matter expertise.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Unstable funding or revenues; poorly designed migration or normalization processes; poorly formed ingest and quality assurance procedures; rapid churn of staff; incoherent patterns of subject matter; lack of domain knowledge; no or very small numbers of users; weak or absent collecting policy; deposit to ensure minimal compliance with funder mandate; limited or dysfunctional data management planning.

‘Vulnerable’ in the Presence of Good Practice

Clear preservation planning; repository development roadmap; ability to transfer collections or share metadata with subject repositories or portals; strong user base; demonstrable re-use of data; clear collecting policy; data management planning early in data lifecycle.

2019 Review

Research data has been an entry in the BitList since 2017 so the Jury this year has tried to spell out the different complexities that arise in the preservation of research outputs more generally. This entry draws attention to services which take upon themselves commitments to preserve research data, but which may not be able to deliver those promises through lack of capability.

Additional Jury Comments

Research data is complex and has specific requirements for documentation which may only be known to subject matter experts. However well intended, it is risky for institutions to attempt to replicate that level of expertise across all the domains within the institution, and it can be hard for smaller publishers to make commitments to sustain data in the long term.

Read More

Semi-Published Research Data

   Endangered large

Data sets produced in the course of research and shared informally between researchers such as by posting to a website or portal but without preservation capability or commitment. Typically the data remains in the hands of the researchers who have the job of maintaining it.

Group: Research Outputs

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: New Entry

Previous category: New Entry

Imminence of Action

Action is recommended within five years, detailed assessment within three years.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a major effort to prevent losses in this group, such as the development of new preservation tools or techniques.

Examples

Departmental webservers; project wikis; GitHub repositories

‘Critically Endangered’ in the Presence of Aggravating Conditions

Originating researcher no longer active or changed research focus; staff on temporary contracts; dependence on single student or staff member; weak or fluid institutional commitment to subject matter; weak institutional commitment to data sharing; complicated or contested intellectual property; encryption; limited or dysfunctional data management planning.

Vulnerable in the Presence of Good Practice

Data in preparation for transfer to specialist repository; robust data management planning; documented and managed professionally;

2019 Review

Research data has been an entry in the BitList since 2017 so the Jury this year has tried to spell out the different complexities that arise in the preservation of research outputs more generally. This entry represents ‘self-help’ data sharing which is to be encouraged as a means to facilitate open science but shouldn’t be confused with long-term preservation.

Additional Jury Comments

Research data is complex and has specific requirements for documentation which may only be known to subject matter experts. However, data creators are not necessarily well placed to sustain data in the long term.

Read More

Video files

   Endangered large

Video files in any format containing moving picture and sound recordings

Group: Sound and Vision

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within five years, detailed assessment within three years.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a small effort to prevent losses in this group, such as the deployment of proven preservation tools or techniques.

Examples

WAV; MP3; MP4

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of replication; encryption; digital rights management; proliferation of file formats; weak or non-existent technical documentation; lack of preservation capability or commitment; poorly managed or digitisation processes or QA.

Vulnerable in the Presence of Good Practice

Effective replication; normalization of file formats; strong technical documentation; preservation pathway; good descriptive cataloguing; trusted repository.

2019 Review

This is a new entry submitted through the open nomination process. There are connections between this entry and others relating to social media but has been included as its own entry because the Jury wanted to emphasize the issues of video preservation that pertain to offline recording, whether from broadcast, film industry, institutional and private collections too.

Additional Jury Comments

There are simply too many formats and too many standards, but the FFMPEG project and its related tools have significantly mitigated the technical risk to most video files. This enables a practitioners to transform the vast majority of file formats to safer preservation formats while retaining significant properties. However, technical risk is only one of the factors. There needs to be institutional engagement with audio-visual data as a priority. The issue then becomes one of identifying the organizations responsible and, constrained by the cost to store video data, making effective selection decisions.

Read More

Scroll to top