View from Experiment/Observation driven Applications Richard P. Mount May 24, 2004 DOE Office of Science Data Management Workshop

Richard P Mount View from Experiment/Observation Driven Science 2

3

4

5

6

7

8

9

10

Richard P Mount View from Experiment/Observation Driven Science 11

Richard P Mount View from Experiment/Observation Driven Science 12

Richard P Mount View from Experiment/Observation Driven Science 13

Richard P Mount View from Experiment/Observation Driven Science 14

Richard P Mount View from Experiment/Observation Driven Science 15

Richard P Mount View from Experiment/Observation Driven Science 16

Richard P Mount View from Experiment/Observation Driven Science 17

Richard P Mount View from Experiment/Observation Driven Science 18

Richard P Mount View from Experiment/Observation Driven Science 19

Richard P Mount View from Experiment/Observation Driven Science 20

Richard P Mount View from Experiment/Observation Driven Science 21

Richard P Mount View from Experiment/Observation Driven Science 22

Richard P Mount View from Experiment/Observation Driven Science 23

Richard P Mount View from Experiment/Observation Driven Science 24

Richard P Mount View from Experiment/Observation Driven Science 25

Richard P Mount View from Experiment/Observation Driven Science 26

Richard P Mount View from Experiment/Observation Driven Science 27

Richard P Mount View from Experiment/Observation Driven Science 28

Richard P Mount View from Experiment/Observation Driven Science 29

Richard P Mount View from Experiment/Observation Driven Science 30

Richard P Mount View from Experiment/Observation Driven Science 31

Richard P Mount View from Experiment/Observation Driven Science 32

Richard P Mount View from Experiment/Observation Driven Science 33

Richard P Mount View from Experiment/Observation Driven Science 34

Experiment/Observation Common Characterisitcs (Mildly Provocative)

Richard P Mount View from Experiment/Observation Driven Science 36 Experiment/Observation Common Characteristics Dominated by large, expensive devices and projects Correct project planning includes data- management hardware and software development –Not acceptable to build a $1Billion device and then face a Data-Management crisis –Development might be much more valuable if performed in a wider context Often hundreds or thousands of users Geographically distributed users

Richard P Mount View from Experiment/Observation Driven Science 37 Consequences of Common Characteristics Less worry about workflow management – part of the project from the start Multi-user concerns: –Keeping track of millions of data products (files?) created by people you barely know –Performance issues due to many concurrent queries –Data movement, grids and networks really matter to international collaborations Visualization can be a useful tool but rarely a major issue Responsiveness is a key issue –Taking months or years to answer a simple question is almost deadly

Final Comments and Pet Project Peddling

Richard P Mount View from Experiment/Observation Driven Science 39 Characterizing Scientific Data My petabyte is harder to analyze than your petabyte –Images (or meshes) are bulky but simply structured and usually have simple access patterns –Features are perhaps 1000 times less bulky, but often have complex structures and hard-to-predict access patterns

Richard P Mount View from Experiment/Observation Driven Science 40 Hydrogen Bubble Chamber Photograph 1970 CERN Photo

Richard P Mount View from Experiment/Observation Driven Science 41 Storage Issues Disks: –Random access performance is lousy, unless objects are megabytes or more independent of cost deteriorating with time at the rate at which disk capacity increases (Define random-access performance as time taken to randomly access entire contents of a disk)

Richard P Mount View from Experiment/Observation Driven Science 42 Latency and Speed – Random Access

Richard P Mount View from Experiment/Observation Driven Science 43 Latency and Speed – Random Access

The End