Where has all the AV content gone? How do we preserve it for the future? Jim Lindner Media Matters

Our Audio Visual Heritage How we will be understood by history will in part be based on what materials they have to study. We are in the early years of a technology evolution that will certainly take hundreds of years, maybe thousands. What is at risk are the documents of our time.

What are these Documents? Feature, Documentary, Independent Film Network, Independent, Art Television Industrial / Educational Film and Video Audio from Network to Soundtrack to Field Recordings and Oral Histories Community Recordings Home Recordings

Where has the content gone? We Have Lost Much More then most people suspect –All but a small fraction of the films of the silent film era are lost forever –Essentially all of television “pre history” is lost E.G. Broadcasts of J.L. Baird TV Before Tape – All lost except what was a kine or news film and has survived TV After Tape – A huge amount has been lost –1958 – news shows remain versus 11,000 Days

Where has all the content gone? –Landfill – space, cost, not “needed” anymore –Erasure – intentionally to save money and unintentionally –Into brown cardboard boxes after the edit sessions, unlabeled, sitting in warehouses or worse –Closets, Attics, Basements

Where has all the content gone? Some of what is left has found its way to an archive.

How Much of it survives? No one really knows - but there are estimates….. CBC – 300,000 tapes BBC – 600,000 hours MTV – 1.2 Million hours UNESCO estimate – 200 million hours of culturally important archives at risk

What do the Records in Archives look like? For the most part our A/V heritage is : A mixture of analog and digital formats including… For the most part our A/V heritage is : A mixture of analog and digital formats including…

A Vast Array of Media Types and Formats –Film and Film Elements Nitrate – Acetates – Polyester - Mixtures (Magnetic Full Coat) Nitrate – Acetates – Polyester - Mixtures (Magnetic Full Coat) –Audio Disc, Cylinder, Vinyl, Wire (steel), Polyester (PET), Acetate, CD-R Disc, Cylinder, Vinyl, Wire (steel), Polyester (PET), Acetate, CD-R –Video – Polyester (PET)

Media is EXTREMELY FRAGILE –Nitrate Highly Combustible –Acetates – Vinegar Syndrome –Binder issues – Sticky Shed –Laminates – Delaminate

Media is Subject to Disasters Natural….. –Chemical Deterioration –Physical/Mechanical Damage –Natural Disasters

… and Man Made WarTheftLoss Bad Economic Times Format Obsolescence Technological Evolution

“Digital” Media is subject to many of the same perils Format Type may be irrelevant in some situations

Storage as a file adds many levels of complexity that did not exist with Analog Media File Format Compatibility Application Level Compatibility Operating System Compatibility Firmware / BIOS Level Compatibility Hardware / Controller Level Compatibility

How?

What about the Content in the File? Content Management –Few standards across collections – no union catalog Let me see all the films that have Ford cars –Description of Sounds and Pictures with words –Primitive Searching tools

No real LONG term strategy: Best Practice Environmental Control –Short term postponement of the inevitable No single media type lasts forever No single format / technology lasts forever No single location lasts forever.

How long do we store the stuff anyhow – how long is forever? Manufacturers think 90 days – well OK – 10 Years We have paper documents for hundreds of years We have clay tablets for thousands of years We have cave paintings for many thousands of years

Can we realistically reformat forever? Consider the resources required to reformat every 25 years, physical space for each object, cost for media Consider the resources required to reformat every 25 years, physical space for each object, cost for media Consider the environmental cost to produce all that media Is it desirable even if you did?

What we are doing is not working very well! We need to rethink and try some new things… Can the past help us with the future?

The First Television was Mechanical! Mechanical television existed for quite some time – some systems scanning horizontally others vertically. John Logie Baird generally is given credit for the first working mechanical television system – as well as developing the first way of RECORDING his television signals.

This is what it looked like

This is what it took to record TV…

Magnetic Recording

In The Beginning: –Initial market was for for TIME SHIFTING – Not for Editing or anything else –No thought given to other markets or applications –It did not take very long before other uses were found for video recording and the manufacturers responded with new product

Magnetic Recording Different Markets / Different Needs –Broadcast – High Quality, High Cost, Low Quantity of Machines Sold, Flexibility –Industrial / Educational – Medium Quality, Medium Cost, Higher Volume of Machines Sold, Simplicity These Markets were later segmented Further and new markets like consumers came into being

Magnetic Recording Broadcast

Broadcast

Videotape Recording Military

Industrial / Educational

Maybe we can predict that… Change and new formats have always been a part of our AV Heritage and we need to think in those terms. Innovation will continue - and perhaps the best we can do is not mess it up for the next generation of innovation It will continue to get smaller, better and cheaper…. and

There WILL be more of it!

Here is the good news - it is getting cheaper and easier to store it!

Economic Viability of storage as FILES versus Videotape In Canberra – Today June 6 Using Sony Stock 1 Digital Betacam Tape costs $27.08, Records 1 hour, Obsolete Format ? 1 LTO3 Datatape costs $60.52

Economic Viability of storage as FILES versus Videotape Using MPEG2 50 Mbit or MJPEG hour of content requires 25 – 30 Gigabytes using MJPEG2000 (24 using MPEG2) 1 LTO3 tape stores 400 Gigabytes –400 / 25 = 16 1 LTO3 tape stores 16 Hours

Cost to store 1 hour of content Cost to store 1 hour of content LTO = $60.52 / 16 Cost = $3.78 Per Hour Digital Betacam = $27.08 Videotape is 7.16 TIMES more expensive!!!!! – storing the exact same quality.

How Much Will Storage Cost, What about small collections? 350 Hour Collection 350 x 30 Gigabytes each hour = 10,500 Gigabytes (10.5 Terabytes) Media $.15 per Gigabyte = $ LTO Tapes (versus 700 Umatic) 2333 DVD’s….

But what about the decks!? LTO3 $3600 (Tandberg Internal) DVW 2000 $40,500 UVW 1800 $9,00 (Used)

How is the Migration Performed? Video Videotape Audio Time Code Meta Data

Manual Migration Tedious and error prone Requires constant attention Difficult to maintain quality control Slow Team of 6 experts (3 per shift – 2 shifts per day) can migrate 5000 hrs/yr Expensive Requires highly skilled labor No economies of scale No Metadata No history of past actions or condition

Automated Migration Higher Quality Verification of tape condition and process Consistent quality control Faster Multiple streams at a time (up to 6) Runs 24/7 (up to 140 hours per day) Less Expensive 10 to 40% of the cost of manual migration

What about Compression ?

Compression is NOT ALWAYS BAD for archival applications Depends on the type of compression Depends on the application Depends on the implementation Depends on the user (using things for what they were designed to do)

Compression is NOT ALWAYS BAD for archival applications Must define terms –Preservation and/or Access? Must understand underlying issues –Technical Considerations –Marketing Considerations –Practical Considerations –Political Considerations (NTSC vs. PAL)

Many Different Standards - and it IS confusing… MPEG - Moving Picture Experts Group Several major categories use entirely different approaches MPEG 1 Samples 4:2:0 at 1.2 Mbits/sec at 352x240, 30 FIELDS / Second MPEG 2 Samples 4:2:0 at 1.2 to 15 Mbits/sec at 704x496, 60 Fields / Second MPEG 2 Professional Main Level Samples 4:2:2 at about 50-60Mbits/sec (variable) at 704x496, 60 fields/sec MPEG 4 MPEG 7 …. MetaData

Compression for A/V Materials In general terminology there are two classes of Compression – LossLESS, and Lossy Most Common formats in AV are Lossy –MPEG2, MPEG4, H.264, Windows Media LossLESS formats offer Uncompressed Quality, with storage savings of about 3:1 –MJPEG 2000

Determination of Quality cannot be “It looks fine to me” Do No Harm Area of Rapid Technology Development Real possibility of new class of “Orphan Video” by encoding type – Videocube What will Search Engines use in the future?

We Are Now in a Transition Phase – and We Will Be for Some Time Digital asset management is very new and it will take some time for the technology and standards to mature. Digital asset management is very new and it will take some time for the technology and standards to mature. Migration of analog media to digital systems is still very unclear – compression/formats/bitrates Migration of analog media to digital systems is still very unclear – compression/formats/bitrates Current production process still has “conventional” media used at times Current production process still has “conventional” media used at times Storage is still too expensive, networks are still too slow and unreliable, systems are still largely incompatible. Storage is still too expensive, networks are still too slow and unreliable, systems are still largely incompatible.

What Happens When This…..

Turns Into This

What will the role of the future A/V Archivist be? Archivist as Information “miner” Archivist as Information “miner” Archivist as architect of information standards and systems Archivist as architect of information standards and systems Archivist as Information Manager Archivist as Information Manager