Managing video data with finite resources Andrew Wild

Wikipedia describes this as... “The production of vast amounts of information in binary code, represented as words, texts, videos, photos and other mediums, expressed on computers and websites, and distributed across the internet.” What’s a Digital Exhaust? © 2011 Andy Wild

Post could describe this as... “The means of storing production data without sacrificing the effectiveness of your facilities infrastructure and assets” What’s a Digital Exhaust? Paraphrased...“You can’t store data indefinitely, so move it from expensive storage to cheaper more appropriate storage, but make sure you keep track of it!” © 2011 Andy Wild

YES... We all do now. If you are struggling for space to store data for the next production or are standing knee deep in G-Safe drives the answer for you is a combination of yes and no. Perhaps it would be fairer to say that you’re not managing your digital exhaust very well... If at all. Do I have a Digital Exhaust? © 2011 Andy Wild

What can I do about data? Data is inevitable, its here already. Deal with it! Manage your clients data carefully and efficiently. Invest in cost effective tools. Don’t make knee jerk purchases to fix issues. They can be difficult to unpick and expensive to solve. Talk to others and see what they have done. © 2011 Andy Wild

Capture Transport Duplicate Store Deliver Archive P.O.F. © 2011 Andy Wild Current working practice

Capture Transport Duplicate Store Deliver Archive Capture Transport Duplicate Store Deliver Archive Production 1 Production 2 © 2011 Andy Wild Multiple production streams

How much storage is enough? © 2011 Andy Wild

There’s never enough storage! © 2011 Andy Wild

Disks Its amazing they work at all! Disks can suffer from a range of strange things like... Wear out: In a 7200 RPM drive the disks are spinning 120 times per second. After a few years the motor can start to go. It may become erratic, causing some bits to be squeezed and others smeared. The arm that moves the heads can become loose and so can go off track and corrupt data on adjacent tracks. Flipped bits: read-only tracks sitting next to high write tracks can be weakened until your disk can’t read it. Dust and dirt can scratch the disk and/or create enough heat so the head stops reading. RAID controllers: small cpus running code subject to bugs, as well as all manner of electrical, connector and cable problems. Corruption of RAID 5 parity data can be possible and is irrecoverable. storagemojo.com/ Robin Harris – Storage GOD! © 2011 Andy Wild

Local StorageNetwork Storage  Attached directly to workstation.  Cheap to implement and expand.  Varying performance using USB – FireWire – SATA – SCSI – SAS – Fibre Channel.  Centrally held data available to all.  Expensive to implement and expand.  Accessed by standard / bespoke network infrastructure.  Varying performance from varying standards. CIFS – NFS – iSCSI – Bespoke etc. Storage woes. Where’s my disk? © 2011 Andy Wild

Local StorageNetwork Storage  “Sneaker Net” has issues... Disks get lost and dropped in transit etc.  Disks don’t travel too well. Failed drives.  Disks can be difficult to move from one system to another (MAC to PC for example.)  RAID5 systems are cheap.  Media stuck in one location and not available to all.  Expensive to implement and expand. Specialist to maintain.  At the mercy of your house network and limited by your connection speed.  Owing to high availability files appear to grow on the system at an exponential rate.  RAID6 offers increased redundancy performance, but still not bomb proof. Storage pros and cons © 2011 Andy Wild

Manage data before its an issue! © 2011 Andy Wild

Where to start? Start with some simple rules. Listen to your staff, they know more than you think. People inherently hate to delete things, but you have to delete at some stage. Make them delete it! © 2011 Andy Wild

If there’s too much...? Media Deletion Policy – Get One! Minimise files on your system, use fit for purpose NAS/SAN systems. Understand what you need to keep and what you don’t. Delete/backup data when its appropriate. © 2011 Andy Wild

If they won’t delete... If they won’t delete it... Move it... Till they delete it! Archiving is the official wording but this can take many forms. Have at least some room to move with your storage plans. Don’t fill everything! © 2011 Andy Wild

Best practices (Blue Sky) Capture/Ingest NLE Central Storage NAS / SAN Storage Cloud based service Deep archive server © 2011 Andy Wild

Best practices (cost effective) Capture/Ingest NLE Central Storage NAS / SAN Storage Blue Ray Disk archive © 2011 Andy Wild

Best practices (cost effective) Capture/Ingest NLE Central Storage NAS / SAN Storage © 2011 Andy Wild Deep archive server?

Tool Set - Hardware NASArchive  Security – Ensure you have a good RAID system (5 or 6 would be great)  Capacity – Make sure you have enough (you never will)  Connectivity – Make sure you have the right connection for the performance you require.  Price – There’s something for everyone out there!  Archive could be to disk*  Archive could be to data tape (LTO5 1.5TB per disk)  Archive could be to BlueRay (25GB to 50GB)  ALL assets that are archived need to have accountability no matter what medium is chosen  Archiving can be expensive! © 2011 Andy Wild

SAN/NAS Consumer QNAP/NetGear Turn key solutions easily deployed on an Ethernet network. Low to mid performance Capacity up to 8TB Server Drive server using hardware RAID controller and Enterprise class NAS software (Opensource ?) Mid to high performance Capacity up to 16TB Specialist Bespoke solution offering high storage capacity and density coupled with high performance. Specialist network possibly needed. Capacity 32TB + © 2011 Andy Wild

Archive BlueRay Disc Cheap to deploy and off the shelf tools allow for easy creation of data disks. Shelf life is currently unknown. Capacity : 50GB per DL disk BDXL is 100GB per disk. Data Tape Becoming much more common with LTO5 becoming ever more popular. Expensive to install and slow to write and restore from. Finding data can be sometimes difficult. Capacity : LTO4 800GB and LTO5 1.5TB Disk Archiving When retrieval of data is important and speed is of the essence specialist disk arrays with Nodal redundancy offers the greatest protection with easy scalability. Capacity: 32TB to PetaBytes © 2011 Andy Wild

Tool Set - Software SAN/NASArchive  Windows / Mac file shares.  Opensource. Openfiler – FreeNAS.  Propriety software.  Bespoke turn key solutions.  Home made solution.  OpenSource solutions. © 2011 Andy Wild

Look to the future Camera Acquisition Codecs (AVCHD-AVCI-MXF- QT etc) NLE Software developments. Workstation performances. Storage Capacity and speed needs. Evolving solutions for file based workflows. © 2011 Andy Wild

The Cloud and services Cloud based service ‘Pipe’ dependent – how quick is your connection? Purchased service, not owned by you. Archive? Review/approve? Streaming? Source downloads and stock shot purchase? © 2011 Andy Wild

Virtual Storage... Really? 500GB 3TB 12TB Virtual Volume Can be hosted locally but also accessed as a service (cloud) Still needs to exist on an actual physical disk! Expensive and complex to maintain. © 2011 Andy Wild

Moores Law for HDDs 3TB 1PB? For drives this is actually ‘Kryders Law’ and dictates the rate of progression in disk storage. Owing to the utilization of ECC, the magnetoresistive effect and the giant magnetoresistive effect the current rate of increase in hard drive capacity is doubling every 18months not 24, but with the event of SSD drives this should bring calculation back under Moores Law of 24 months © 2011 Andy Wild

Summary Install appropriate tool set and user skill sets. Manage your storage effectively. Develop a data management strategy and document it. Keep track of your customers needs of storage demands (what codec did they use and what shoot ratio etc) © 2011 Andy Wild

For further info please visit