Coming revolutions in mass storage: implications for image archives Christopher D. Elvidge, Ph.D. NOAA-NESDIS National Geophysical Data Center E/GC2 325 Broadway, Boulder, Colorado USA And Dr. Mikhail ZHIZHIN Head of Information Technologies Lab Institute of Physics of the Earth and Geophysical Center Russian Academy of Science, Moscow, Russia APAN eScience Workshop – July 6, 2004

Image Archive Sizes Continuing to Grow Rapidly For example, from satellite image ingest at NOAA-NGDC runs from six to ten GB per day. For example, from satellite image ingest at NOAA-NGDC runs from six to ten GB per day. Once launched (~2007), NPP will produce about 2 TB of data per day, which NOAA will archive. Once launched (~2007), NPP will produce about 2 TB of data per day, which NOAA will archive. During the NPOESS era ( ) there will three satellites, each producing 2 TB of data per day, with NOAA responsible for the archive. During the NPOESS era ( ) there will three satellites, each producing 2 TB of data per day, with NOAA responsible for the archive. There are many other examples. There are many other examples.

Abridged History of Storage ( Punch cards – back when snakes had legs. Punch cards – back when snakes had legs. Ticker tape – faster than punch cards. Ticker tape – faster than punch cards. Magnetic tape – invented by IBM in Magnetic tape – invented by IBM in IBM introduces the 305 RAMAC (Random Access Method for Accounting and Control), the first magnetic hard disk storage system. The RAMAC stored 5 megabytes (MB) of data, was the size of two large refrigerators and cost $10,000 per MB; the device could store 5 million characters of data on 50 disks, each 24 inches in diameter. Each disk could hold an equivalent of 25,000 punch cards IBM introduces the 305 RAMAC (Random Access Method for Accounting and Control), the first magnetic hard disk storage system. The RAMAC stored 5 megabytes (MB) of data, was the size of two large refrigerators and cost $10,000 per MB; the device could store 5 million characters of data on 50 disks, each 24 inches in diameter. Each disk could hold an equivalent of 25,000 punch cards. RAMAC – the first hard drive – 1956

Abridged History of Storage ( 9-track tapes – workhorse of image archives in the 1960’s-early 1990’s. 50 mb at 1600 bpi. IBM MSS cartridge (1982) held 50 mb. 0.2 mb tape strip from IBM Data Cell (mid-1960’s)

Current Standard – Tape Library System Used by NASA, NOAA, USGS and many others. Used by NASA, NOAA, USGS and many others. Tape is widely regarded as the standard for at least another ten years. Tape is widely regarded as the standard for at least another ten years. Storage Technology 9310 robotic tape silo, can hold 6000 IBM 3590 tapes. At 20 GB each the silo can hold ~300 TB. Circa 1999.

LTO Tape Growth Path Already Planned ( Currently Available

Alternative to Tape Library Systems: Use “Local” Hard Drives Instead of Tape Approximate price parity between tape and hard drives. Approximate price parity between tape and hard drives. Allows faster access. Allows faster access. Several design options (SAN, NAS). Several design options (SAN, NAS). Hard drive capacity already in the 200 GB range and has been projected to reach 20 TB. Hard drive capacity already in the 200 GB range and has been projected to reach 20 TB. Data may be more easily corrupted. Data may be more easily corrupted.

Alternative to Tape Library Systems: Use “Local” Hard Drives Instead of Tape

Alternative to Tape Library Systems: Use “GRID” Hard Drives Instead of Tape Approximate price parity between tape and hard drives. Approximate price parity between tape and hard drives. Allows faster access. Allows faster access. Several design options. Several design options. Hard drive capacity already in the 200 GB range and has been projected to reach 20 TB. Hard drive capacity already in the 200 GB range and has been projected to reach 20 TB. Community ownership may lead to more collaborations? Community ownership may lead to more collaborations? Data may be more easily corrupted. Data may be more easily corrupted. Agencies may also choose to build stand alone archive to ensure long term data preservation. Agencies may also choose to build stand alone archive to ensure long term data preservation. See essay See essay

Nano-Storage-Technology Still Emerging Molecular-scale nanowire memory cells promises unprecedented data storage Molecular-scale nanowire memory cells promises unprecedented data storage Big Blue says breakthrough means millipede may crawl out of lab &document_id=7860 Big Blue says breakthrough means millipede may crawl out of lab &document_id= &document_id= &document_id=7860

InPhase Promotional Video Holographic Data Storage Still Emerging

Implementations of Nano and Holographic Data Storage Tape Tape CD like disks CD like disks Hard drives Hard drives Greater storage density – lower costs – but implementation routes likely to extend current forms.

Vision of Future Image Archives Data easily accessed – readily processed Data easily accessed – readily processed Combination of data from multiple sites / multiple sources Combination of data from multiple sites / multiple sources Copies of source data and processing tools kept on long term storage media Copies of source data and processing tools kept on long term storage media

Storage Options in Future Image Archives Raw Data, Metadata Processing Code, Higher Level Products Long term Survivable Storage A.K.A. Data Vault Network Storage Tape Library Systems Raw Data, Metadata, Processing Code GRID Storage Working Subsets Of Archive Raw Data, Metadata, Processing Code, Higher Level Products, Experimental Products, Assessments

Storage Options in Future Image Archives Raw Data, Metadata Processing Code, Higher Level Products Data Vault Open Storage Facility Raw Data, Metadata, Processing Code Widely Held Data Raw Data, Metadata, Processing Code, Higher Level Products, Experimental Products, Assessments Number of Users

Regional Resources Singapore Data Storage Institute: Agency for Science, Technology & Research, or A*STAR (then known as the National Science & Technology Board) and the National University of Singapore (NUS) Singapore Data Storage Institute: Agency for Science, Technology & Research, or A*STAR (then known as the National Science & Technology Board) and the National University of Singapore (NUS) Others? Others?

Conclusions - Advances in storage capacity & reductions in cost will allow archive storage to diversify – with copies held to meet specific objectives: Widely distributed collections used in current projects. Widely distributed collections used in current projects. Tape and hard drive media to provide operational access from data centers. Tape and hard drive media to provide operational access from data centers. Long term “survivable” storage – two or more copies on highly durable media to preserve data hundreds of years – ability to survive technological collapse – reengineering of read capacity. Long term “survivable” storage – two or more copies on highly durable media to preserve data hundreds of years – ability to survive technological collapse – reengineering of read capacity.