1. How much information? Adapted from a presentation by:
Jim Gray Microsoft Research
Alex Szalay
Johns Hopkins University

2. How much information is there in the world
What can we store.
What is stored.
Why are we interested.

3. Infinite Storage? The Terror Bytes are Here Petrified by Peta Bytes?
1 TB costs 1k$ to buy
1 TB costs 300k$/y to own
Management & curation are expensive
Searching 1TB takes minutes or hours
Petrified by Peta Bytes?
But… people can “afford” them so, – Even though they can never actually be seen in your lifetime
Automate the process
Yotta
Zetta
Exa
Peta
Tera
Giga
Mega
Kilo
We are here

4. How much information is there?
Yotta
Zetta
Exa
Peta
Tera
Giga
Mega
Kilo
Soon everything can be recorded and indexed
Most bytes will never be seen by humans.
Data summarization, trend detection anomaly detection are key technologies
See Mike Lesk: How much information is there:
See Lyman & Varian:
How much information
Everything!
Recorded
All Books MultiMedia
All books
(words)
.Movie
A Photo
A Book
24 Yecto, 21 zepto, 18 atto, 15 femto, 12 pico, 9 nano, 6 micro, 3 milli

5. First Disk 1956 IBM 305 RAMAC 4 MB 50x24” disks 1200 rpm 100 ms access
35k$/y rent
Included computer & accounting software (tubes not transistors)

6. Storage capacity beating Moore’s law
Improvements: Capacity 60%/y Bandwidth 40%/y Access time 16%/y
1000 $/TB today
100 $/TB in 2007
Moores law
58.70%
/year
TB growth
112.30%
/year since 1993
Price decline
50.70%
/year since 1993
Most (80%) data is personal (not enterprise) This will likely remain true.

7. Disk Storage Cheaper Than Paper
File Cabinet (4 drawer) 250$ Cabinet: Paper (24,000 sheets) 250$ Space 10€/ft2) 180$ Total 700$ $/sheet pennies per page
Disk: disk (250 GB =) 250$ ASCII: 100 m pages 2e-6 $/sheet(10,000x cheaper) micro-dollar per page Image: m photos 3e-4 $/photo (100x cheaper) milli-dollar per photo
Store everything on disk Note: Disk is 100x to 1000x cheaper than RAM

8. Trying to fill a terabyte in a year
Item
Items/TB
Items/day
300 KB JPEG
3 M
9,800
1 MB Doc
1 M
2,900
1 hour 256 kb/s MP3 audio
9 K 26
1 hour 1.5 Mbp/s MPEG video
290
0.8
Bottom line: we will be able to keep LOTS of video, and vast amounts of smaller data types (audio, photos, documents).
Note: probably not worth the time to delete an object

9. Portable Computer: 2006? 100 Gips processor 1 GB RAM 1 TB disk
1 Gbps network
“Some” of your software finding things is a data mining challenge

10. 80% of data is personal / individual. But, what about the other 20%?
Business
Wall Mart online: 1PB and growing….
Paradox: most “transaction” systems < 1 PB.
Have to go to image/data monitoring for big data
Government
Government is the biggest business.
Science
LOTS of data.

11. Q: Where will the Data Come From? A: Sensor Applications
Earth Observation
15 PB by 2007
Medical Images & Information + Health Monitoring
Potential 1 GB/patient/y  1 EB/y
Video Monitoring
~1E8 video 1E5 MBps  10TB/s  100 EB/y  filtered???
Airplane Engines
1 GB sensor data/flight,
100,000 engine hours/day
30PB/y
Smart Dust: ?? EB/y

12. Premise: DataGrid Computing
Store exabytes twice (for redundancy)
Access them from anywhere
Implies huge archive/data centers
Supercomputer centers become super data centers
Examples: Google, Yahoo!, Hotmail, BaBar, CERN, Fermilab, SDSC, …

13. Thesis
Most new information is digital (and old information is being digitized)
An Information Science Grand Challenge:
Capture
Organize
Summarize
Visualize
this information
Optimize Human Attention as a resource
Improve information quality

14. The Evolution of Science
Observational Science
Scientist gathers data by direct observation
Scientist analyzes data
Analytical Science
Scientist builds analytical model
Makes predictions.
Computational Science
Simulate analytical model
Validate model and makes predictions
Data Exploration Science Data captured by instruments Or data generated by simulator
Processed by software
Placed in a database / files
Scientist analyzes database / files

15. Computational Science Evolves
Historically, Computational Science = simulation.
New emphasis on informatics:
Capturing,
Organizing,
Summarizing,
Analyzing,
Visualizing
Largely driven by observational science, but also needed by simulations.
Too soon to say if comp-X and X-info will unify or compete.
BaBar, Stanford
P&E Gene Sequencer
From
Space Telescope

16. Next-Generation Data Analysis
Looking for
Needles in haystacks – the Higgs particle
Haystacks: Dark matter, Dark energy
Needles are easier than haystacks
Global statistics have poor scaling
Correlation functions are N2, likelihood techniques N3
As data and computers grow at same rate, we can only keep up with N logN
A way out?
Discard notion of optimal (data is fuzzy, answers are approximate)
Don’t assume infinite computational resources or memory
Requires combination of statistics & computer science

17. Smart Data (active databases)
If there is too much data to move around,
take the analysis to the data!
Do all data manipulations at database
Build custom procedures and functions in the database
Automatic parallelism guaranteed
Easy to build-in custom functionality
Databases & Procedures being unified
Example temporal and spatial indexing
Pixel processing
Easy to reorganize the data
Multiple views, each optimal for certain types of analyses
Building hierarchical summaries are trivial
Scalable to Petabyte datasets

18. Challenge: Make Data Publication & Access Easy
Augment FTP with data query: Return intelligent data subsets
Make it easy to
Publish: Record structured data
Find:
Find data anywhere in the network
Get the subset you need
Explore datasets interactively
Realistic goal:
Make it as easy as publishing/reading web sites today.

19. Data Federations of Web Services
Massive datasets live near their owners:
Near the instrument’s software pipeline
Near the applications
Near data knowledge and curation
Super Computer centers become Super Data Centers
Each Archive publishes a web service
Schema: documents the data
Methods on objects (queries)
Scientists get “personalized” extracts
Uniform access to multiple Archives
A common global schema
Challenge:
What is the object model for your science?
Federation

20. Web Services: The Key? Internet-scale distributed computing
Web SERVER:
Given a url + parameters
Returns a web page (often dynamic)
Web SERVICE:
Given a XML document (soap msg)
Returns an XML document
Tools make this look like an RPC.
F(x,y,z) returns (u, v, w)
Distributed objects for the web.
+ naming, discovery, security,..
Internet-scale distributed computing
Your
program
Web
Server
http
Web page
Your
program
Web
Service
soap
Data
In your address space
object in xml

21. Emerging technologies
Look at science
High end computation and storage