Obfuscate and Accumulate
Here’s an idea that comes out of a clear blue sky: we don’t talk enough about technical debt in digital preservation. It’s an important concept and one we could be working with.
Technical debt is a term used in software development to describe the costs that arise when an easy or cheap solution is adopted over a fully worked out, properly documented, long-term solution. Expedient or cheap solutions which don’t have forwards compatibility make future changes costly if not impossible. This cost accumulates "interest" as changes and upgrades are delayed and therefore become harder to achieve over time. The debt is a contingent liability: work can be deferred but cannot be avoided to maintain legacy systems, or to exit them in an orderly fashion. In worsening conditions, investment is diverted towards short term maintenance, which further increase the complexity of a legacy system, making migration harder and stifling genuine innovation. Left unchecked this ultimately reaches a bottom where either the system fails, or the business fails. Sometimes both.
My sense is that many agencies are accumulating technical debt which they don’t fully understand. Indeed, every agency that depends on technology for the delivery of products or services has a theoretical exposure to technical debt, and this has real world consequences.
A bold assertion follows – that, perhaps without fully realising it, the digital preservation community has been quietly and successfully building the tools and services that tackle and prevent the accumulation of technical debt. And so digital preservation is not a niche fixation for specialists, but a pervasive concern through the entire economy.
Blue Skies and Free Money
I live on a flight path into Glasgow Airport so am conditioned to the low-level rumble of air traffic. This time last year the tiny Kilbrides sat in the garden waving cheerfully as delegates arrived for iPres. A week or so ago there was nothing, only a clear blue sky.
A technical fault had knocked out the UK’s air traffic control system. A news flash nervously announced that UK airspace wasn’t exactly closed, just that it wasn’t entirely safe, so they had decided to stop planes taking off for a bit. By coincidence I heard this from colleagues in Dublin complaining that all their flights were grounded, the same colleagues who reported a technical glitch meant that Bank of Ireland cash machines were giving away (apparently) free money. Talk about IT problems with real world implications: blue skies and free money. What’s not to like?
Here's a useful rule of thumb when you read news reports like this. Largescale, time-served systems only normally misfire for two reasons: commission or omission. Perhaps someone is doing something that they shouldn’t be doing: sometimes a bungling misadventure; sometimes a crooked misdemeanour. Or perhaps something has not been done that needed to be done (or was done so cheaply and so poorly that time is up on the cheap fix): casual neglect leads to causal negligence.
I don’t know what happened at the Bank of Ireland or at Air Traffic Control, but I’ll bet senior managers are hunkered down trying to tease out omission from commission and promising it won’t happen again. I’ll also bet digital preservation will be getting pushed down their agenda (was it even on the agenda?) and that’s a shame. Here’s why: if you conceptualise obsolescence within the design phase of a system you will note and minimize the technical debts that the system accumulates, and so you will be better placed to avoid tipping points of failure.
The idea of technical debt has become current in software development. It’s been described as a sort of ‘dark matter’ which IT managers know exists and can detect its impact but find hard to measure or track. In July 2020 the business consultancy firm McKinsey estimated that as much as 20 percent of the budgets set aside for new products and services are diverted as a kind of interest payment on systems or data sets that were executed cheaply or poorly maintained. The same survey reported 60% of CIOs believe technical debt is increasing in their agencies.
Buy now pay later
The concept of technical debt can be applied to other contexts. Here’s a thought experiment which I’d love to try out.
My hunch is that agencies which plan for the long-term will reflect more or less continually on the total cost of ownership of data or systems, and they will budget accordingly. They will be incentivized to prefer systems and solutions which have credible claims to sustainability, because these will be cheaper and more effective in the short and long term. The total cost of ownership will drive their thinking, not the short-term costs of deployment. Therefore some aspect of preservation, whether through the migration of data or virtualization of a system, or even simply thorough documentation will be wired into the specifications at the design stage.
By contrast, agencies with short term planning horizons and short-term budget rounds will be encouraged to make cheap decisions in the moment without reflecting on the cost and effort that will arise from extracting data or business processes. In these circumstances, the data and systems will simply become unusable without additional cost. In many cases that will not actually be a problem. In many other cases the debt will be transferred down the line.
Groundhog Austerity
Why does this matter? It matters because it’s coming back: doing more with less.
I was invited recently to provide evidence to the ‘Funding for Culture’ hearings by the relevant committee in Parliament. The opening remarks from the committee included the ominous remarks that the funding crisis ‘provides an opportunity to accelerate innovative solutions to the budgetary pressures … at a time of limited resources, what other innovative approaches could the Scottish Government take forward to support the culture sector?’
I am not against innovation and goodness only knows how innovative the cultural sector can be and has been. Innovation is often-times interpreted as a pivot towards technologies that will simplify or streamline operations. It’s true that technology, effectively deployed, can have a significant and positive impact on budgets: but the longer-term use and deployment of digital resources has to be factored into the up-front costs. There is no evidence of additional support to the preservation of digital resources arising from the digital turn implied by the drive to innovation in the cultural sector. Therefore, when investing in technology there should be efforts to measure, manage and reduce technical debt, such as by better strategic investment in digital preservation.
Ignoring the future is the opposite of strategy
The heading is the message: ignoring the future is not an IT strategy. Ignoring the future is the opposite of an IT strategy. Just ask the CTO's at the Bank of Ireland and at the National Air Traffic Control Service.
Organizations that listen to the digital preservation community at the design and procurement phases of business systems will tend to avoid technical debt because they will have planned a pathway that avoids technical obsolescence or manage it more efficiently. Agencies that fail to listen to the digital preservation community will attempt to transfer that debt to them.
This is quite an abstract argument, but we can see it worked out in practice in very many cases. Digital preservation dependencies arise in the design, procurement, and operation of business systems. If we leave thinking about preservation to the end of life it’s often times already too late. None of this leads to anything genuinely new: we’ve been saying this for years and such invitations still seem too rare. Therefore, perhaps if we align ourselves with discussions about how to avoid or reduce technical debt, IT managers will stop seeing digital preservation as a separate exercise, and start seeing it as a requirement of every system and process they commission.
A message to CTO's everywhere: you can't ignore the future. It won’t go away.
Read more...