Digital Library Storage Strategies Robert Cartolano, Director Library Information Technology Office November 14, 2008
Challenge Scale up efforts to catalog, digitize & publish to the Web unique, distinctive collection holdings that have significant value for teaching or research. Design & implement coherent & comprehensive preservation program for ensuring survival & continued accessibility of Libraries’ digital content. Develop & budget for long-term digital archiving strategy for content created by the Libraries, whether “born-digital” or converted from analog formats. Collaborate with other stakeholders to develop affordable cooperative solutions to ensure long-term preservation of licensed content. Libraries Strategic Plan,
Internal Workflow Management Systems Columbia Digital Library Columbia Academic Commons Internal Data Management Systems Fedora Platform and Library/Information Services Fedora Asset Repository Public facing Library facing Long Term Archive
Internal Workflow Management Systems Columbia Digital Library Columbia Academic Commons Internal Data Management Systems For example: Digitized collections Born-digital collections [Licensed collections] For example: Digitization workflow Preservation workflow Online exhibition workflow For example: Backup Data migration For example: Columbia-produced content Rich collaboration space (e.g., Sakai) Faculty profiles & citations Public facing Library facing Fedora Asset Repository Long Term Archive
Curation Complexity Complex Simple None $$$ $$ $ Cost High Effort Low None Full-Service Curation Assisted Curation Self-Service “Tagging” Automated Curation Slow Ingest Rate Fast Derived from: Goble, Carole, University of Manchester, “Curating Services and Workflows: the Good, the Bad and the Ugly”, Columbia Digital Library Columbia Academic Commons
Preservation Storage Technology Approach Risk Averse - use “tried and true” technologies –Use mature, commodity products as much as possible –Choose first-tier vendor (Sun) with sustainable support models, proven reliability, stability Multiple copies - 2 Disk, 2 Tape for Preservation Open as possible to maximize sustainability –Open Data, Open Formats, Open Source, Open Protocols, Open Community Entrance and Exit Strategy
Storage Technology Software - Storage Archive Manager (SAM) –Policy-based, tiered storage approach –Commercial support from Sun, released as open source –12 years in production, widely used –Solaris 10, ZFS, NFS/CIFS, TAR - standards Hardware - Sun Servers –Two SAM Front-End T5240 Solaris Servers –Tier I Disk Cache, 9.6 Terabytes (TB) expandable to 60TB –Tier II Disk Storage - 192TB Raw, 70TB estimated net storage –Tier II Tape Storage - 80TB, expandable to 460TB
Preservation Storage Architecture On-Site Tape in Campus Data Center On-Site Disk in Campus Data Center Off-Site Disk at NYSERNet Syracuse Off-Site, Off-line Tape to Iron Mountain Copy 1Copy 2Copy 3 SAM Servers Disk Cache Fedora Servers Copy 4 Columbia Digital Library Applications Columbia Academic Commons Applications
Questions