1. Sensor systems and large data sources Jim Myers, NCSA

2. Exponential Trends in Observational Technologies Decreasing costs Increasing rates/resolution Increasing automation Increasing dimensionality Increasing breadth of sources

3. Examples – Add your favorite here… – Home Monitoring – Health Monitoring – Environment Monitoring – Habitat Monitoring – Earthquake Monitoring – Battlefield Monitoring 3rd grade project - 70 MB

4. Challenges: Volume The ability to create data is outstripping storage… – locally and globally And Storage is growing faster than access speeds… “Over the last 10 years while disk sizes have increased by a factor of 1,000, the rotation speed of large disks used in disk arrays has only changed a factor of 2…”

5. Challenges: Discovery, Organization, Trust (Quality) Data is being collected – In multiple dimensions – On multiple subjects – In many locations File names don’t scale  Rich metadata and provenance is needed to allow discovery, organize it for use, and to support assessment of quality

6. Challenges: Analysis Whether physical or statistical, analysis methods often scale as powers of data size Research is requiring more sophisticated analysis, not less

7. Solutions/Trends: Innovation in storage/management HW to optimize data bandwidth (e.g. Graywolf) New forms of databases and content systems: – Streaming – Spatial – SciDB – Column Stores – Semantic stores – Big Table

8. Solutions/Trends: Innovation in Acquisition, Access, and Processing Adaptive Sensing Query Optimization/Parallelization Moving algorithms to data One-pass algorithms Analysis over compressed data

9. Summary Data Deluge Metadata Deluge Processing Deluge Innovation required across the life cycle Including development of new data organizations (e.g. DataNet)

10. References/Image Credits 1.Collins et al. (2003). Science 300, 286-290; Hugenholtz & Tyson (2008) Nature 455, 481-483. 2.http://www.ncbi.nlm.nih.gov/Genomes/http://www.ncbi.nlm.nih.gov/Genomes/ 3.Scientific Data Management in the Coming Decade, Jim Gray, David T. Liu, Maria Nieto- Santisteban, Alexander S. Szalay, David DeWitt, Gerd Heber, January 2005, Microsoft ResearchTechnical Report, MSR-TR-2005-10 4.The Sensor Spectrum: Technology, Trends, and Requirements, Joseph M. Hellerstein, Wei Hong, Samuel R. Madden, doi=10.1.1.87.2977, 2007 5.The Diverse and Exploding Digital Universe, IDC Whitepaper, March 2008 6.GrayWulf: Scalable Clustered Architecture for Data Intensive Computing, Alexander S. Szalay, Gordon Bell, Jan Vandenberg, Alainna Wonders, Randal Burns, Dan Fay, Jim Heasley, Tony Hey, Maria Nieto-SantiSteban, Ani Thakar, Catharine van Ingen, and Richard Wilton, 15 September 2008, MSR Tech Report MSR-TR-2008-187 7.Fran Berman, Ken Kennedy Award Presentation, SC 2009 8.Dick Crutcher, Gul Agha, Parya Moinzadeh, personal communication 9.http://cacm.acm.org/news/60825-wireless-smart-sensors-inspect-bridge/fulltexthttp://cacm.acm.org/news/60825-wireless-smart-sensors-inspect-bridge/fulltext 10.http://www.dailywireless.org/2009/08/28/smartphones-data-tsunami-coming/http://www.dailywireless.org/2009/08/28/smartphones-data-tsunami-coming/ 11.http://www.docstoc.com/docs/22818776/Trend-Summary-–-New-gene-sequencing-technologieshttp://www.docstoc.com/docs/22818776/Trend-Summary-–-New-gene-sequencing-technologies 12.http://www.economist.com/specialreports/displaystory.cfm?story_id=15557443http://www.economist.com/specialreports/displaystory.cfm?story_id=15557443 13.http://www.fastcompany.com/1548674/hp-joins-the-smarter-planet-sweepstakeshttp://www.fastcompany.com/1548674/hp-joins-the-smarter-planet-sweepstakes 14.http://www.sciencemag.org/cgi/content/full/323/5919/1297http://www.sciencemag.org/cgi/content/full/323/5919/1297