Open source intelligence sources of current conflicts

   Critically Endangered small

Open source social media and web content which supports crowd-sourced investigation and fact-checking to verify or refute claims of state agencies and rebel groups in the context of current political or military conflict.

Group: Digital Legal Records

Trend: New Entry

Unanimous Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within twelve months, detailed assessment is now a priority

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

Loss seems likely: by the time tools or techniques have been developed the material will likely have been lost.

Examples

Social media sources relating to current conflicts, such as in Yemen or Syria.

‘Endangered’ in the Presence of Good Practice

Offline backup captured by journalist or investigating authority;

2019 Review

This is a new entry received through open submission in 2019 and subsequently split into three elements by the Jury, relating to current, recent and historic sources.  This entry relates in particular to materials relating to current and ongoing conflicts.  Social media companies have a policy to take down or suppress content that they consider to be propaganda for terrorist groups.  This has had the unintended consequence of deleting or supressing content that was being used in open source investigation or fact checking for journalistic or judicial purposes, and which may therefore be an impediment to refutation or prosecution.  However a new generation of cloud based services, such as Hunchly have emerged in the last few years which allow investigators to copy and stabilise content to private accounts in the process of investigating it: so the ethical requirements of social media companies and the integrity of investigation are both served. The Jury notes that the such content remains at risk, and the process of investigation is slower than algorithmic deletion.  Nonetheless there is a notable difference in the investigation of current conflicts than historic ones where evidence has been lost.

Additional Jury Comments

--

Read More

Non-current Hard Disk Technologies

   Critically Endangered small

Materials saved to storage devices with a variety of underlying magnetic or solid-state technologies that are hardwired into a computer that is no longer under warranty or supported: typically hard disks more than five years old.

Group: Integrated Storage

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within twelve months, detailed assessment is now a priority

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

It would require a small effort to prevent losses in this group, such as the deployment of proven preservation tools or techniques

Examples

Disks installed into computers or servers that are more than fives years old, or out of warranty

‘Practically Extinct’ in the Presence of Aggravating Conditions

Lack of replication; poor storage; non-standard connections or controllers; aggressive compression; encryption

‘Endangered’ in the Presence of Good Practice

Maintenance schedule; renewable extendable warranty; best practice storage and operation; replication

2019 Review

The judges have introduced this entry to ensure that the range of media storage is properly assessed and presented.  The lifecycles of most consumer hard disk technology is stable in comparison to portable devices because they are integrated into systems and therefore inherit the lifecycle and replacement of the entire system.  This is less true at scale however where disks are used in storage arrays and refreshment is more loosely tied to the server architecture.  Storage at scale also means the percentage likelihood of finding a disk failure increases.

Additional Jury Comments

Perversely the greater density of newer disks, as well as encryption and compression mean that they can be more fragile than older disks with less density and less sophisticated read/write technologies.  The age of a disk is not the best or only indicator of its reliability.

Read More

Non-current Portable Magnetic Media

   Critically Endangered small

Materials saved to floppy disks, tape, portable hard disks or other magnetic storage devices where the media is out of warranty and reader devices may no longer be supported or integrated easily into hardware infrastructure: typically more than fives years old.

Group: Portable Media

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within three years, detailed assessment is in 12 months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to prevent losses in this group, including the development of new preservation tools or techniques

Examples

Floppy disks; tape; certain kinds of portable hard disks, zipdrives.

‘Practically Extinct’ in the Presence of Aggravating Conditions

Poor storage; inability to access readers; no replication; encryption; aggressive compression

‘Endangered’ in the Presence of Good Practice

Active management; dependable access to readers; strong documentation; documentation independent from the media

2019 Review

The judges have introduced this entry to ensure that the range of media storage is properly assessed and presented.  Portable magnetic media was ubiquitous but is fragile not just to physical wear and tear but also to magnetic interference and bit-rot.  The substrates of the disks can prove unstable and in some cases proprietary reader technology means that the disk becomes obsolete before it degrades. Storage at scale also means the percentage likelihood of failure increases

Additional Jury Comments

There is really no excuse for using floppy disks for storage these days.  Tape is a different proposition since it allows high-density back up offline and nearline.  But there are challenges with backwards compatibility of popular and even relatively recent LTO versions.

Read More

Non-current Portable Optical Media

   Critically Endangered small

Materials saved to DVDs, CDs or other optical storage devices where the media is out of warranty and reader devices may no longer be supported or integrated easily into hardware infrastructure: typically more than five years old.

Group: Portable Media

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within three years, detailed assessment is in 12 months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to prevent losses in this group, including the development of new preservation tools or techniques

Examples

CDs, laser disc technologies, DVD, HDVD

‘Practically Extinct’ in the Presence of Aggravating Conditions

Poor storage; inability to access readers; no replication; encryption

‘Endangered’ in the Presence of Good Practice

Active management; dependable access to readers; strong documentation; documentation independent from the media

2019 Review

The judges have introduced this entry to ensure that the range of media storage is properly assessed and presented.  Optical media is in some senses a preferred option as it is typically more stable than magnetic or solid state media: but these characteristics have been over-sold.  The substrates of the disks can prove unstable and more importantly the reader technology tends to be proprietary and can become obsolete long before the disks degrade. Storage at scale also means the percentage likelihood of failure increases.

Additional Jury Comments

Highly dependent on who is looking after the portable media but made more difficult over time. There is a lack of granularity in the definition of media types so generic advice like "make sure you're moving your data to new forms of storage every 5 -10 years".

Early generations of this media are not as robust as the current generation and can deteriorate significantly if not stored appropriately.

Read More

Non-current Portable Solid State Media

 

 Critically Endangered small

Materials saved to flash or other solid-state storage devices where the media is out of warranty and reader devices may no longer be supported or integrated easily into hardware infrastructure: typically more than five years old.

Group: Portable Media

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within three years, detailed assessment is in 12 months

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to prevent losses in this group, including the development of new preservation tools or techniques

Examples

USB sticks and pen drives; Flash storage in cameras and phones; certain types of portable hard disk

‘Practically Extinct’ in the Presence of Aggravating Conditions

Poor physical storage; inability to access readers; no replication; encryption

‘Endangered’ in the Presence of Good Practice

Active management; dependable access to readers; strong documentation; documentation independent from the media

2019 Review

The judges have introduced this entry to ensure that the range of media storage is properly assessed and presented.  Solid state media – typically flash – provides very fast access to data but can fail without warning.  This is because it is typically subject to a limited number of program/erase cycles, as well as ‘read/disturb’ effects. Storage at scale also means the percentage likelihood of failure increases. 

Additional Jury Comments

Early generations of media and cheap giveaways are not robust and can deteriorate significantly over time.

Read More

Offline Gaming

   Critically Endangered small

Interactions and experiences of games and related interactive virtual worlds, representing a significant investment of skill and time by players, and significant elements of cultural output in the late 20th and early 21st century.

Group: Gaming

Trend: No change

Consensus Decision

Added to List: 2017

Last update: 2018

Previously: Critically Endangered

Imminence of Action

Action is recommended within twelve months, detailed assessment is now a priority

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

Loss seems likely: by the time tools or techniques have been developed the material will likely have been lost.

Examples

Single player games, especially those which   feature significant player-character customisation and player choice, including Role-Playing Games (RPGS) such as The Elder Scrolls V: Skyrim. Games that rely on unique peripherals to play, including rhythm action titles such as Guitar Hero.

‘Practically Extinct’ in the Presence of Aggravating Conditions

Complex hardware dependencies or bespoke hardware; dependence on obsolete, low usage operating systems; no emulation pathway; complex IPR; older magnetic media; free distribution on magazines; loss of underlying code or gaming engine; limited or no commercial interest.

‘Endangered’ in the Presence of Good Practice

emulation pathway; source code; trusted repository; large user community.

2019 Review

This is a subset of an entry made in 2017 for ‘Gaming’ which the Jury has split into four more discrete entries.  This entry for older games encourages greater consideration of the technical complexities which arise from preservation of software and hardware environments.  There is an active specialist market for older games which enable preservation but also skews it around commercial interests. The meaning of ‘older’ is open to interpretation but any game more than 10 years from release should be included here especially if there are more recent releases.

Additional Jury Comments

This is very closely related to the ‘Old or non-current video games’ item, but there are parallels with the online gaming platforms: how do we capture the experience of play? This is much harder than online gaming, where the social interactions occur in digital form. With appropriate selection, appraisal and preservation, we can capture some of this through ephemera such as gaming magazines of the 80s and 90s, for example, but much has already been lost. I’m focussing on the negative here – it is still possible to preserve design materials, developer interviews, sales data and games reviews, etc. but first-hand accounts of what it was actually like to play these games are rare.

By implication this category includes user generated content within games which is distinct from the game itself. For example, the National Library of Scotland was approached to preserve a Minecraft representation of Scottish cities. If you can preserve the game then preserving the user generated content should be straightforward. But it’s not clear who is doing that.

Read More

Old or non-current video games

   Critically Endangered small

Older video games designed and played on platforms and devices that are no longer supported, especially those with complex but obsolete rights management or system dependencies. This group includes also older editions of games still published in newer editions

Group: Gaming

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: New Entry

Previously: New Entry

Imminence of Action

Action is recommended within twelve months, detailed assessment is now a priority

Significance of Loss

The loss of tools, data or services within this group would impact on people and sectors around the world.

Effort to Preserve

Loss seems likely: by the time tools or techniques have been developed the material will likely have been lost.

Examples

Civilization; SimCity; Sonic the Hedgehog; EA Sports; Doom; Wolfenstein 3D; Lemmings

‘Practically Extinct’ in the Presence of Aggravating Conditions

Complex hardware dependencies or bespoke hardware; dependence on obsolete, low usage operating systems with no emulation pathway; complex intellectual property rights; use of older magnetic media; free distribution on magazines; loss of underlying code or gaming engine; limited or no commercial interest; dependency on remote servers that are closed.

‘Endangered’ in the Presence of Good Practice

Emulation pathway; source code; trusted repository; large user community;

2019 Review

This is a subset of an entry made in 2017 for ‘Gaming’ which the Jury has split into four discrete entries.  This entry for older games encourages greater consideration of the technical complexities which arise from preservation of software and hardware environments as well as the historic value which games have acquired.  There is an active specialist market for older games which enables, preservation but skews it to commercial interests. The meaning of ‘older’ is open to interpretation but certainly any version more than 10 years from release should be included here, even if and perhaps especially if there are more recent releases

Additional Jury Comments

Focussing on offline gaming here - it’s difficult to distinguish these from the related ‘Offline gaming’ entry but there are additional complexities in terms of systems dependencies that need to be considered. It is one thing to be able to get older software running on a different CPU architecture (emulation), but another thing entirely to recreate the peripherals (Wiimotes, plastic guitars, steering wheels, even bongo drums!) that are required to play the games as intended. There’s a sort of kinaesthetic issue: one could perhaps use the accelerometers in modern smartphones to simulate the functionality of a Wiimote, but it will feel different in the hand.

It might be possible to sub-divide between games that have a dependency on communication with servers that no longer exist (loss has already happened) and others where the dependencies are to do with old hardware/software where there is more potential to preserve the interaction.

It’s unclear who is responsible for preserving what and games may be falling between the cracks.

Read More

Web domains with no legal deposit

   Critically Endangered small

Web archiving is a recognized specialism within digital preservation, able to capture large quantities of material with routine and standards-based tools.  But there are significant issues of intellectual property rights associated with website capture and republication.  In many jurisdictions, but by no means all, those obstacles are overcome by regulations that enable a national library or other ‘legal deposit’ agency to copy and preserve content.  Where no such permission exists, there is a significant risk of loss.

Group: Web

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within 12 months, detailed assessment is now a priority

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

Loss seems likely: by the time tools or techniques have been developed the material will likely have been lost.

Examples

Domains registered without a country code; domains with a country code but weak or unenforceable legal deposit permission to harvest.

‘Practically Extinct’ in the Presence of Aggravating Conditions

Rapid churn of websites; lack of access to Internet Archive harvest; contentious content; encryption; digital rights management; non-standard content management

‘Endangered’ in the Presence of Good Practice

Permissive approach to Legal deposit;

2019 Review

This entry came about by Jury deliberation on entries to the open nomination process.  It is characterised by regulatory barriers rather than technical ones, though the pace of change in web technologies as well as the growth of web content mean that significant technical challenges still exist.  The Jury also notes that local conditions are also a significant factor.  For example, web sites often also fall under public records legislation or are important elements of corporate records: and so important parts of the web are harvested even when there is no explicit legal deposit legislation. Moreover the Jury particularly recognizes the work of the Internet Archive to capture and preserve content.  Even so there are significant gaps in web archiving and in too many cases it is regulation that is the barrier.

Additional Jury Comments

--

Read More

Unpublished research data from US Govt researchers

   Critically Endangered small

To prepare data for long-term archiving requires quite a bit of time, and offices being shut down or moved with little warning don't have time.

Group: Research Outputs

Trend: New Entry

Consensus Decision

Added to List: 2019

Last update: 2019

Previous category: New Entry

Imminence of Action

Action is recommended within 12 months, detailed assessment is now a priority

Significance of Loss

The loss of tools, data or services within this group would impact on many people and sectors.

Effort to Preserve

It would require a major effort to prevent losses in this group, such as the development of new preservation tools or techniques.

Examples

Agencies that have closed or have had funding withdrawn from research initiatives.

‘Practically Extinct’ in the Presence of Aggravating Conditions

Lack of access to archival services; sudden or un-anticipated closure; loss of implicit knowledge from destabilised or demoralised staff; encryption

‘Endangered’ in the Presence of Good Practice

Archival responsibility well developed; documentation; published through research channels.

2019 Review

This is unusual entry was submitted through open nomination process and assessed by Jury before being included.  It has significant overlaps with other entries in the research outputs group but has been retained to draw attention to two realities: firstly that research outputs are not simply a matter for academic institutions and that government is in fact a major producer of research data; and secondly that politically instability and threats to the continuity of government services are a significant preservation risk.  The significant fact that this entry relates to the US does not mean that other jurisdictions are immune from political instability. Instead it alludes to a concern expressed by the Jury that politically inconvenient research outputs face particular and immediate threats of which the digital preservation community should be cognisant.

Additional Jury Comments

The Jury offered no other comments on this entry, and a number of members did not participate in this assessment.

Read More

Smart Phone Apps

 

 Critically Endangered small

Apps created for smartphones. Many are deprecated quickly but others survive through multiple update cycles.  It is hard to maintain version control and often dependent upon the company that publishes them. There is no clear agency or mandate to record or collect.

Group: Apps

Trend: no change

Consensus Decision

Added to List: 2017

Last update: 2018

Previously: Critically Endangered

Imminence of Action

Action is recommended within 12 months, detailed assessment is now a priority

Significance of Loss

The loss of data, tools or services within this group would be felt globally.

Effort to Preserve

Loss seems likely: by the time tools or techniques have been developed the material will likely have been lost.

Examples

London 2012 app; BBC Olympic app; Apps published for Apple iOS 10 or earlier

‘Practically Extinct’ in the Presence of Aggravating Conditions

Device dependence; poor documentation; uncertainty over IPR; short term contracts; lack of skills, commitment or policy from corporate owners; rapid churn of OS; shifting business requirements of app resellers; dependence on exotic or obsolete formats or OS processes;

‘Endangered’ in the Presence of Good Practice

Strong documentation; version control for code and compiled app; emulation enabled; designated repository taking preservation responsibility and capable to deliver

2019 Review

This entry was submitted in 2017 and has been retained to draw attention to the challenges of software preservation and the extraordinary velocity of the market for apps. Given the speed of change it is hard to see how digital preservation efforts can keep pace.  It may help to split this entry into different groups based on the platform though the risks would be largely the same.

Additional Jury Comments

Old versions of apps are completely lost to most users: once you upgrade an app you typically can't go back.  Perhaps ios is more critical - at least with Android you can often get .apk from the internet separate from the marketplace. The NSRL contains hundreds of thousands of mobile applications which are not being actively preserved but could be if a mandate existed. An extension to Legal Deposit might be possible.

The faster we act the less we will lose. It is unlikely that there will ever be one agent with mandate to collect and different apps available in different countries so a network of national organisations would be needed. The companies that create these apps are the key to the licensing challenges and conversation with them is necessary, though would need to happen immediately in order to negotiate the right to preserve/escrow both apps, operating systems, documentation, and phone development emulators.

Apps often provide a secondary view on the primary data and in many cases that data is more important, and arguably easier to preserve.  So, although this is an eye-catching entry it is not as significant as others. 

Read More

Scroll to top