Big Data Architectures Panel on Exploiting Big Data in Collaboration Initiatives CTS 2012 Westminster (Denver) CO May 23 2012 Geoffrey Fox gcf@indiana.edu http://www.infomall.org http://www.salsahpc.org Director, Digital Science Center, Pervasive Technology Institute Associate Dean for Research and Graduate Studies,  School of Informatics and Computing Indiana University Bloomington

Architecture of Data Repositories? Traditionally governments set up repositories for data associated with particular missions For example EOSDIS, GenBank, NSIDC, IPAC for Earth Observation , Gene, Polar Science and Infrared astronomy respectively LHC/OSG computing grids for particle physics Focus has been on getting access to data with curation, provenance etc. Assumes analysis is dealt with separately as repositories have modest attached computing

Big Data Analysis Big data suggest that model of scientist browsing repository and downloading (petabytes of) data is flawed Data bandwidth too low and local compute resources too small The “Fourth Paradigm” (data oriented science) based on large scale data analysis Need to support repositories for large instruments (telescopes, accelerators, satellites) and pervasive distributed instruments/sensors (gene sequences)

Clouds as Support for Data Repositories? The data deluge needs cost effective computing Clouds are by definition cheapest Shared (Collaborative!) resources essential (to be cost effective and large) Can’t have every scientists downloading petabytes to personal cluster Need to reconcile distributed (initial source of ) data with shared computing Can move data to (disciple specific) clouds How do you deal with multi-disciplinary studies

Traditional File System? Data Compute Cluster C Archive Storage Nodes Typically a shared file system (Lustre, NFS …) used to support high performance computing Big advantages in flexible computing on shared data but doesn’t “bring computing to data” Cloud Object stores similar to this?

Hadoop/Google Data Parallel File System? C Data File1 Block1 Block2 BlockN …… Breakup Replicate each block No archival storage and computing brought to data