1. Coming revolutions in mass storage: implications for image archives Christopher D. Elvidge, Ph.D. NOAA-NESDIS National Geophysical Data Center E/GC2 325 Broadway, Boulder, Colorado 80305 USA Email: chris.elvidge@noaa.gov And Dr. Mikhail ZHIZHIN Head of Information Technologies Lab Institute of Physics of the Earth and Geophysical Center Russian Academy of Science, Moscow, Russia Email: jjn@wdcb.rujjn@wdcb.ru APAN eScience Workshop – July 6, 2004

2. Image Archive Sizes Continuing to Grow Rapidly For example, from 1992-2004 satellite image ingest at NOAA-NGDC runs from six to ten GB per day. For example, from 1992-2004 satellite image ingest at NOAA-NGDC runs from six to ten GB per day. Once launched (~2007), NPP will produce about 2 TB of data per day, which NOAA will archive. Once launched (~2007), NPP will produce about 2 TB of data per day, which NOAA will archive. During the NPOESS era (2010-2020+) there will three satellites, each producing 2 TB of data per day, with NOAA responsible for the archive. During the NPOESS era (2010-2020+) there will three satellites, each producing 2 TB of data per day, with NOAA responsible for the archive. There are many other examples. There are many other examples.

3. Abridged History of Storage (http://www.disk-tape-data-recovery.com/storage-history.htm) http://www.disk-tape-data-recovery.com/storage-history.htm Punch cards – back when snakes had legs. Punch cards – back when snakes had legs. Ticker tape – faster than punch cards. Ticker tape – faster than punch cards. Magnetic tape – invented by IBM in 1952. Magnetic tape – invented by IBM in 1952. 1956 - IBM introduces the 305 RAMAC (Random Access Method for Accounting and Control), the first magnetic hard disk storage system. The RAMAC stored 5 megabytes (MB) of data, was the size of two large refrigerators and cost $10,000 per MB; the device could store 5 million characters of data on 50 disks, each 24 inches in diameter. Each disk could hold an equivalent of 25,000 punch cards. 1956 - IBM introduces the 305 RAMAC (Random Access Method for Accounting and Control), the first magnetic hard disk storage system. The RAMAC stored 5 megabytes (MB) of data, was the size of two large refrigerators and cost $10,000 per MB; the device could store 5 million characters of data on 50 disks, each 24 inches in diameter. Each disk could hold an equivalent of 25,000 punch cards. RAMAC – the first hard drive – 1956

4. Abridged History of Storage (http://www.columbia.edu/acis/history/media.html) 9-track tapes – workhorse of image archives in the 1960’s-early 1990’s. 50 mb at 1600 bpi. IBM MSS cartridge (1982) held 50 mb. 0.2 mb tape strip from IBM Data Cell (mid-1960’s)

5. Current Standard – Tape Library System Used by NASA, NOAA, USGS and many others. Used by NASA, NOAA, USGS and many others. Tape is widely regarded as the standard for at least another ten years. Tape is widely regarded as the standard for at least another ten years. Storage Technology 9310 robotic tape silo, can hold 6000 IBM 3590 tapes. At 20 GB each the silo can hold ~300 TB. Circa 1999.

6. LTO Tape Growth Path Already Planned (http://www.lto-technology.com/newsite/index.html) Currently Available

7. Alternative to Tape Library Systems: Use “Local” Hard Drives Instead of Tape Approximate price parity between tape and hard drives. Approximate price parity between tape and hard drives. Allows faster access. Allows faster access. Several design options (SAN, NAS). Several design options (SAN, NAS). Hard drive capacity already in the 200 GB range and has been projected to reach 20 TB. Hard drive capacity already in the 200 GB range and has been projected to reach 20 TB. Data may be more easily corrupted. Data may be more easily corrupted.

8. Alternative to Tape Library Systems: Use “Local” Hard Drives Instead of Tape http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=43 http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=43 http://www.firingsquad.com/hardware/building_budget_storage_server/ http://www.firingsquad.com/hardware/building_budget_storage_server/ http://www.firingsquad.com/hardware/building_budget_storage_server/ http://www.archive.org/web/petabox.php http://www.archive.org/web/petabox.php http://www.archive.org/web/petabox.php http://nbd.sourceforge.net/ http://nbd.sourceforge.net/ http://www.storage.ibm.com/software/virtualization/sfs/ http://www.storage.ibm.com/software/virtualization/sfs/ http://www.microsoft.com/windowsserver2003/techinfo/overview/san.mspx http://www.microsoft.com/windowsserver2003/techinfo/overview/san.mspx http://www.microsoft.com/windowsserver2003/techinfo/overview/san.mspx http://www.enterprisestorageforum.com/technology/features/article.php/947551 http://www.enterprisestorageforum.com/technology/features/article.php/947551 http://www.enterprisestorageforum.com/technology/features/article.php/947551 http://www.enterprisestorageforum.com/technology/features/article.php/981191 http://www.enterprisestorageforum.com/technology/features/article.php/981191 http://www.enterprisestorageforum.com/technology/features/article.php/981191 http://www.cse.ohio-state.edu/~jain/refs/san_refs.htm http://www.cse.ohio-state.edu/~jain/refs/san_refs.htm http://www.cse.ohio-state.edu/~jain/refs/san_refs.htm http://www.brocade.com/san/pdf/whitepapers/SANvsNASWPFINAL3_01_01.pdf http://www.brocade.com/san/pdf/whitepapers/SANvsNASWPFINAL3_01_01.pdf http://www.brocade.com/san/pdf/whitepapers/SANvsNASWPFINAL3_01_01.pdf

9. Alternative to Tape Library Systems: Use “GRID” Hard Drives Instead of Tape Approximate price parity between tape and hard drives. Approximate price parity between tape and hard drives. Allows faster access. Allows faster access. Several design options. Several design options. Hard drive capacity already in the 200 GB range and has been projected to reach 20 TB. Hard drive capacity already in the 200 GB range and has been projected to reach 20 TB. Community ownership may lead to more collaborations? Community ownership may lead to more collaborations? Data may be more easily corrupted. Data may be more easily corrupted. Agencies may also choose to build stand alone archive to ensure long term data preservation. Agencies may also choose to build stand alone archive to ensure long term data preservation. See essay http://isec.pl/papers/juggling_with_packets.txt See essay http://isec.pl/papers/juggling_with_packets.txt

10. Nano-Storage-Technology Still Emerging Molecular-scale nanowire memory cells promises unprecedented data storage http://www.azonano.com/news_old.asp?newsID=122 Molecular-scale nanowire memory cells promises unprecedented data storage http://www.azonano.com/news_old.asp?newsID=122 Big Blue says breakthrough means millipede may crawl out of lab http://www.smalltimes.com/document_display.cfm?section_id=53 &document_id=7860 Big Blue says breakthrough means millipede may crawl out of lab http://www.smalltimes.com/document_display.cfm?section_id=53 &document_id=7860 http://www.smalltimes.com/document_display.cfm?section_id=53 &document_id=7860 http://www.smalltimes.com/document_display.cfm?section_id=53 &document_id=7860

11. InPhase Promotional Video Holographic Data Storage Still Emerging

12. Implementations of Nano and Holographic Data Storage Tape Tape CD like disks CD like disks Hard drives Hard drives Greater storage density – lower costs – but implementation routes likely to extend current forms.

13. Vision of Future Image Archives Data easily accessed – readily processed Data easily accessed – readily processed Combination of data from multiple sites / multiple sources Combination of data from multiple sites / multiple sources Copies of source data and processing tools kept on long term storage media Copies of source data and processing tools kept on long term storage media

14. Storage Options in Future Image Archives Raw Data, Metadata Processing Code, Higher Level Products Long term Survivable Storage A.K.A. Data Vault Network Storage Tape Library Systems Raw Data, Metadata, Processing Code GRID Storage Working Subsets Of Archive Raw Data, Metadata, Processing Code, Higher Level Products, Experimental Products, Assessments

15. Storage Options in Future Image Archives Raw Data, Metadata Processing Code, Higher Level Products Data Vault Open Storage Facility Raw Data, Metadata, Processing Code Widely Held Data Raw Data, Metadata, Processing Code, Higher Level Products, Experimental Products, Assessments Number of Users

16. Regional Resources Singapore Data Storage Institute: Agency for Science, Technology & Research, or A*STAR (then known as the National Science & Technology Board) and the National University of Singapore (NUS) http://www.dsi.a-star.edu.sg/research/spintronics.html Singapore Data Storage Institute: Agency for Science, Technology & Research, or A*STAR (then known as the National Science & Technology Board) and the National University of Singapore (NUS) http://www.dsi.a-star.edu.sg/research/spintronics.html http://www.dsi.a-star.edu.sg/research/spintronics.html Others? Others?

17. Conclusions - Advances in storage capacity & reductions in cost will allow archive storage to diversify – with copies held to meet specific objectives: Widely distributed collections used in current projects. Widely distributed collections used in current projects. Tape and hard drive media to provide operational access from data centers. Tape and hard drive media to provide operational access from data centers. Long term “survivable” storage – two or more copies on highly durable media to preserve data hundreds of years – ability to survive technological collapse – reengineering of read capacity. Long term “survivable” storage – two or more copies on highly durable media to preserve data hundreds of years – ability to survive technological collapse – reengineering of read capacity.