Pre-Production TV and Movie Materials

   Endangered large

Digital records of the creative and production process for film and television, such as initial designs, screenplay and script, on set still photography, rushes or out-takes that are not included in the final production and therefore not available to on-air broadcast archives or film libraries

Group: Sound and Vision

Trend: New Entry

Consensus Decision

Added to List: 2017

Last update: 2019

Previous category: Of Concern

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

TV and Movie production archives in digital form; outputs of script management software; drafts of screenplay; continuity photography; costume design; set design; lighting and sound design.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of custodial responsibility; confusion over intellectual property rights; lack of appraisal; lack of recognition of preservation at executive level;

‘Vulnerable’ in the Presence of Good Practice

Preservation responsibility understood and acted upon; preservation infrastructure and planning for key items; access and use of collections to inform subsequent productions

2019 Review

This entry was first proposed in 2017 and noted as being ‘of concern’, though the Jury did not have the capacity to assess the entry thoroughly. Additional expertise has been recruited to the Jury this year to help with the assessment.

Additional Jury Comments

These materials are not being collected in any coherent way and will be lost as they are not valued by production companies, the space needed to hold them is costly and the expertise needed to catalogue and collect them is lacking. In short, nobody really cares. This is not primarily a technical problem and advocacy is needed urgently.

Read More

PDF other than PDF/A

   Endangered large

Documents presented in PDF (Portable Document Format) format (ISO 32000:1 and ISO 32000:2) and other data wrapped inside them, other than PDF/A but including all other variants and versions.

Group: Formats

Trend: No change

Consensus Decision

Added to List: 2017

Last update: 2018

Previous category: Of Concern

Imminence of Action

Action is recommended within five years, detailed assessment within three years.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world..

Effort to Preserve

It would require a small effort to address losses in this group, requiring the application of proven preservation tools or techniques.

Examples

PDF 1.1, 1.2, 1.3, 1.4 (excluding PDF/A as a subset), 1.5, 1.6, 1.7 and 2.0. PDF/X and PDF/E

‘Critically Endangered’ in the Presence of Aggravating Conditions

Loss of context; loss of authenticity or integrity; external dependencies; poor storage; lack of understanding; significant diversity of data; poorly developed digitization specifications; lack of integrity checking; poorly developed migration or normalizations specifications; lack of virus control; poor storage or replication; lack of validation at the point of creation; encryption.

‘Vulnerable’ in the Presence of Good Practice

Well managed data infrastructure; preservation planning; authenticity managed; use of persistent identifiers; reduction of dependencies; application of records management standards; recognition of preservation requirements beyond formats; strategic investment in digital preservation; preservation roadmap; participation in digital preservation community; format validation.

2019 Review

The judges have introduced this entry as a subset of a previous entry for ‘PDF’, emphasizing the different threats faced by different types of PDF. PDF/A explicitly reduces dependencies and thus curtails preservation risks for certain types of content: PDFs of other types do not. PDF and PDF/A have sometimes been misunderstood as a generic solution to digital preservation requirements. In the eyes of the judges it can only offer a preservation solution when embedded within a wider preservation infrastructure.

Additional Jury Comments

See also:

Fanning, B (2017) Preserving with PF/A (Second Edition), DPC Technology Watch Report 17-01 online at http://dx.doi.org/10.7207/twr17-01

Read More

Electronic hospital and medical records

   Endangered large

Personal medical records and records of hospital treatment are increasingly, if not uniformly born digital. By implication those records should be retained trough the life-time of the patient and perhaps longer as required for intergenerational study; and yet there is little evidence of the medical profession participating in the digital preservation community.

Group: Sensitive Data

Trend: New Entry

Consensus Decision

Added to List: 2017

Last update: 2017

Previous category: Of Concern

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Medical scans; records of treatment and care plans; health advice and notifications;

‘Critically Endangered’ in the Presence of Aggravating Conditions

Loss of context; loss of authenticity or integrity; poor storage; lack of understanding; churn of staff; significant volumes of data; significant diversity of data; ill-informed records management; poorly developed transfer and integrity checking; poorly developed migration or normalizations specifications; longstanding protocols or procedures that apply unsuitable paper processes to digital materials; encryption

‘Vulnerable’ in the Presence of Good Practice

Well managed data infrastructure; preservation enabled at the point of creation; carefully managed authenticity; use of persistent identifiers; well managed records management processes; application of records management standards; recognition of preservation requirements at highest levels; strategic investment in digital preservation; preservation roadmap; participation in digital preservation community.

2019 Review

Medical and hospital records were introduced to the BitList in 2017 though at that time there was limited capacity to address the topic and it was simply published as ‘of concern’. It would have been reviewed in 2019 in any case but was independently received as a submission to the open nomination process. This topic remains large and ill defined. It is useful to draw attention to the scale of the digital preservation challenges which arise in hospitals and in the medical profession, but it would be useful in future years to split this entry into more discrete elements which would be more impactful.

Additional Jury Comments

Increasing sensitivity and awareness of data protection requirements could act inadvertently as a barrier to lifecycle data management. It is striking how little evidence there is of the health technology companies participating in the global digital preservation community.

Read More

Digital music and ephemera shared on social media

   Endangered large

Digital materials created by musicians and fans as a by-product of performance or recording, shared on websites and other social media platforms.

Group: Sound and Vision

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Fan sites; private or illicit recordings of concerts; informal music sharing between networks.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Dependence on social media provider; lack of offline equivalent; infringing intellectual property right; unstable or small community of interest; encryption

Vulnerable in the Presence of Good Practice

Offline equivalent; intellectual property rights conducive to preservation; partnership with collecting institution; availability to web archiving.

2019 Review

This entry has been split out of the previous entry for ‘Digital Music Production and Sharing’ though it has overlaps with other entries including those relating to social media as well as those relating to community-generated content. It has been retained as a separate theme to emphasize the context in which music is shared and enjoyed – a context which could be lost if our attention were on products controlled by studios or artists. The recent case of the closure of ‘Yahoo Groups’ serves to underline the fragility of community content hosted by third parties.

Additional Jury Comments

The ephemera are increasingly stored on websites that themselves are fragile and are removed - as we see with MySpace's removal of MP3s, Flickr's decision to delete vast numbers of images held on free accounts and material lost as a result of Tumblr's policy changes on the definition of pornography. Nothing held on these services can be relied on in archival timeframes

Read More

Consumer social media free at the point of use

   Endangered large

Social media services offered free at the point of use with a subscription model based on reselling user behavior and/or advertising.

Group: Social Media

Trend: No change

Consensus Decision

Added to List: 2017

Last update: 2018

Previous category: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Flickr, Vimeo, YouTube, Instagram, Periscope, DropBox, Facebook, Twitter, Pinterest, Yahoo Groups

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of preservation capacity in provider; Lack of preservation commitment or incentive from provider; Lack of storage replication; proprietary products or formats; poor data protection; inaccessibility to web archiving; political or commercial interference; Lack of offline equivalent; super-abundance; poorly managed IPR; Lossy compression in upload scripts.

Vulnerable in the Presence of Good Practice

Offline backup and documentation of media assets; Migration plan; Early warning from vendors; Roadmap from vendors; Accessible to web harvest; Suitable export functionality; Licencing enables preservation; Preservation commitment from vendor; Preservation capability in vendor;

Resilient to hacking; Selection criteria;

2019 Review

The judges have introduced this entry as a subset of a previous entry, emphasizing the different threats faced by online services that are ‘paid-for’ versus ‘free-at-the-point-of-use’. Both depend on the business model of the vendor and the terms which they impose. For this group of digital materials the business model and sustainability can only be guessed and contracts tend to be asymmetrical in favour of the supplier. Moreover, because these services have a low barrier to entry they may be favoured by agencies or individuals least able to respond to closure or loss.

Additional Jury Comments

Social media capture via web harvesting has become increasingly difficult. The platforms continue to put up barriers to automated capture that prevent preservation of even so-called public content. For example, campaign websites or other election-related content that is only published in Facebook or on Twitter, because these services are "Free". This content is of particular concern as it appears on no other website. Web archivists are constantly shifting strategies and approaches and trying out new (but limited) tools to best capture this content. If we cannot successfully preserve these platforms, we are missing out on documenting organizations, campaigns and elections around the globe. Much of this data exists as data sets based on aggregated use rather than individual files.

Read More

Contractual Documents and Related Records

   Endangered large

Documents, correspondence and other records created in the course of contractual dealings between individuals and agencies, especially where the subjects are of long duration and may be subject to legal scrutiny at undefined points in the distant future.

Group: Digital Legal Records

Trend: No change

Consensus Decision

Added to List: 2017

Last update: 2018

Previous category: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a small effort to preserve materials in this group, with the deployment of proven tools or techniques.

Examples

Contracts, receipts, correspondence, license agreements, building consent, warranties, and any other document or record that represents a legally binding transaction or permission. Such records may be useful in the avoidance or resolution of disputes whether in court or prior to proceedings. Includes Online Terms and Conditions for e-commerce or end-user agreements for services.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Loss of context; loss of authenticity or integrity; external dependencies; poor storage; lack of understanding; churn of staff; poorly framed or over-zealous disposal; ill-informed records management; misplaced fears with respect to data protection, encryption.

Vulnerable in the Presence of Good Practice

Well managed data infrastructure; preservation enabled at the point of creation; carefully managed authenticity; use of persistent identifiers; finding aids; well managed records management processes; application of records management standards

2019 Review

This entry is a subset of an entry made in 2017 for ‘Digital Legal Records and Evidence’ which the Jury has decided to split into four more discrete entries this year. It has significant overlap with a separate entry about Pension Mortgage and Insurance Records, except that those are created explicitly with the expectation that they are tied to long-lifecycle products and are held by long lived agencies who provide those products. This entry notes that many digital records have value over the long term which may not be immediately obvious at the point of creation.

Additional Jury Comments

Legal agreements should be dealt with through records management standards and processes with agencies who have experience of handling this.

Read More

Corporate Records of Long Duration on Network Drives, Intranets and EDRMS

   Endangered large

Records on internal corporate network drives, intranets or document management services where access is limited to a distinct group of users, and in which the lifecycle of the record or the business processes they support is greater than the technology on which they are created or retained.

Group: Sensitive Data

Trend: No Change

Consensus Decision

Added to List: 2017

Last update: 2018

Previous category: Endangered

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to preserve materials in this group, with the development of new preservation tools or techniques.

Examples

Born digital records of small and medium sized enterprises; fasting-changing internal manuals, advice or policies shared on intranets or EDRMS; records of long-lived products and services; Historic guidelines and manuals which evidence 'best practice'; Documentation supporting long-lived contractual relations; Online terms and conditions; Corporate Slack channels; Google Drives; EDRMS; Email.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of systematic preservation function; lack of preservation path or plan for data; dependence on proprietary products or formats; poor management of data protection; political or commercial interference; lack of offline equivalent; over-abundance through poor disposal or version control; lack of capacity; lack of commitment; loss or lack of documentation; sector specific software or data types; encryption

‘Vulnerable’ in the Presence of Good Practice

Preservation infrastructure and pathways; replication; appraisal and selection including de-duplication;

2019 Review

This entry draws attention to the pressing need for digital preservation in business, especially in small to medium enterprises.

Additional Jury Comments

Corporate records should form part of organisational records management schemes and so responsibilities should be clear, however this may be much more challenging for smaller organisations without dedicated roles, or with complex data types.

Read More

Current Portable Magnetic Media

   Endangered large

Materials saved to magnetic tape, portable hard disks or other magnetic media in the last five years where the reader devices are still supported and can be integrated easily into hardware infrastructure.

Group: Portable Media

Trend: new entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a small effort to address losses in this group, requiring the application of proven preservation tools or techniques.

Examples

LTO8 tapes; portable hard disks

‘Critically Endangered’ in the Presence of Aggravating Conditions

Poor storage conditions; encryption; digital rights management; lack of replication; lack of documentation; lack of periodic testing; lack of refreshment pathway; lack of access to readers; out of manufacturers’ warranty or no warranty; storage within paper files.

Vulnerable in the Presence of Good Practice

Regular review and testing; replication; refreshment plan; comprehensive documentation; high quality storage; regular maintenance of readers; multiple readers available.

2019 Review

The judges introduced this entry to ensure that the range of media storage is properly assessed and presented. Magnetic media is typically more fragile than optical media because it is susceptible to ‘bitrot’ and magnetic damage in ways that optical media are not.

Additional Jury Comments

This entry is highly dependent on who is looking after the portable media but made more difficult over time. The lack of granularity in the definition means that only general advice can be offered such as to refresh media. In time it may yet be more useful to split all storage media (maybe 100 items long) with an indication of how long these can be expected to last. In many cases obsolete media can be recovered by specialist companies but the cost of employing them can become an aggravating condition.

It’s important to emphasize that the short lifetime of many storage devices is not a problem to be solved with new long-lasting storage technologies (and indeed, many inventions have come and gone). Cheap, commodity storage has been purposely designed to deliver value at a low price for a short time, and therefore management and preservation processes for monitoring and refreshment need to take these characteristics into account.

Read More

Premium or institutional social media services

   Endangered large

Commercial social media services which are based on a costed subscription and contract.

Group: Social Media

Trend: no change

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within three years, detailed assessment within twelve months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to address losses in this group, possibly requiring the development of new preservation tools or techniques.

Examples

Premium versions of Vimeo, Flickr, Yammer, Slack, Mircosoft Teams and others.

‘Critically Endangered’ in the Presence of Aggravating Conditions

Unstable business models and pricing schedules from providers; lack of export functionality; unstable terms and conditions; lack of onsite copy of key media; lack of strategic plan for IT provision; confusion on IPR; asymmetrical contract arrangements; conflating preservation and access.

‘Vulnerable’ in the Presence of Good Practice

Offline back up for key media; fit to preservation and records management plan; strategic roadmap for adoption of social media;

2019 Review

The judges have introduced this entry as a subset of a previous entry, emphasizing the different threats faced by services that are ‘paid-for’ versus ‘free-at-the-point-of-use’. Both depend on the business model of the vendor and the terms and conditions which they impose. For this group, the business model and sustainability are more obvious and contracts may be enforceable more readily. Moreover, because these services have a slightly higher barrier to entry they may be favoured by agencies better able to respond to closure or loss.

Additional Jury Comments

Traditional web archiving can be employed where the user pays for a service but content is ultimately publicly available (such as Flickr) where the user may pay for a full account to be able to host more photos publicly. But much is unclear about how to preserve internal social media / closed networks that web archiving can't get to or existing tools don't cover. The growth in use of these products for communication and social networking leads us to think this is endangered rather than vulnerable. It’s possible that existing tools could be modified to tackle some of these closed networks but that is likely to require some investments, perhaps related to corporate records in some cases (thinking about internal Slacks, for instance), and more education about the importance of preserving this material and not trusting the publishing platforms to host the content forever.

Read More

Video files

   Endangered large

Video files in any format containing moving picture and sound recordings

Group: Sound and Vision

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within five years, detailed assessment within three years.

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a small effort to prevent losses in this group, such as the deployment of proven preservation tools or techniques.

Examples

WAV; MP3; MP4

‘Critically Endangered’ in the Presence of Aggravating Conditions

Lack of replication; encryption; digital rights management; proliferation of file formats; weak or non-existent technical documentation; lack of preservation capability or commitment; poorly managed or digitisation processes or QA.

Vulnerable in the Presence of Good Practice

Effective replication; normalization of file formats; strong technical documentation; preservation pathway; good descriptive cataloguing; trusted repository.

2019 Review

This is a new entry submitted through the open nomination process. There are connections between this entry and others relating to social media but has been included as its own entry because the Jury wanted to emphasize the issues of video preservation that pertain to offline recording, whether from broadcast, film industry, institutional and private collections too.

Additional Jury Comments

There are simply too many formats and too many standards, but the FFMPEG project and its related tools have significantly mitigated the technical risk to most video files. This enables a practitioners to transform the vast majority of file formats to safer preservation formats while retaining significant properties. However, technical risk is only one of the factors. There needs to be institutional engagement with audio-visual data as a priority. The issue then becomes one of identifying the organizations responsible and, constrained by the cost to store video data, making effective selection decisions.

Read More

Scroll to top