1. San Diego Supercomputer Center SDSC Storage Resource Broker SRB as data grid solution (Chinese version) Arun Jagatheesan arun@sdsc.edu San Diego Supercomputer Center SRB Workshop National Center for High-performance Computing (NCHC) Taiwan, August 3, 2004

2. San Diego Supercomputer Center SDSC Storage Resource Broker SRB as data grid solution (Chinese version) Arun Jagatheesan arun@sdsc.edu San Diego Supercomputer Center SRB Workshop National Center for High-performance Computing (NCHC) Taiwan, August 3, 2004

3. San Diego Supercomputer Center SDSC Storage Resource Broker 3 SRB? SRB = Storage Resource Broker

4. San Diego Supercomputer Center SDSC Storage Resource Broker 4 More Chinese Oops, don’t know any more Chinese to continue

5. San Diego Supercomputer Center SDSC Storage Resource Broker 5 Thanks to Layton Chen

6. San Diego Supercomputer Center SDSC Storage Resource Broker SRB as data grid solution (English Version) Arun Jagatheesan arun@sdsc.edu San Diego Supercomputer Center

7. SDSC Storage Resource Broker 7 Talk Outline Introduction to Problem statement(s) How SRB is the solution SRB Project History SRB Team SRB Architecture (from the Architect him self)

8. San Diego Supercomputer Center SDSC Storage Resource Broker 8 What problem, why SRB solution? Why are people using SRB? What problems did it solve for them? Who are these people? Did they use it because they liked Arun

9. San Diego Supercomputer Center SDSC Storage Resource Broker 9 Southern California Earthquake Center

10. San Diego Supercomputer Center SDSC Storage Resource Broker 10 Southern California Earthquake Center Build community digital library Manage simulation and observational data Anelastic wave propagation output 10 TBs, 1.5 million files Provide web-based interface Support standard services on digital library Manage data distributed across multiple sites USC, SDSC, UCSB, SDSU, SIO Provide standard metadata Community based descriptive metadata Administrative metadata Application specific metadata

11. San Diego Supercomputer Center SDSC Storage Resource Broker 11 SCEC Data Management Technologies Portals Knowledge interface to the library, presenting a coherent view of the services Knowledge Management Systems Organize relationships between SCEC concepts and semantic labels Process management systems Data processing pipelines to create derived data products Web services Uniform capabilities provided across SCEC collections Data grid Management of collections of distributed data Computational grid Access to distributed compute resources Persistent archive Management of technology evolution

12. San Diego Supercomputer Center SDSC Storage Resource Broker 12 NASA Data Grids NASA Information Power Grid NASA Ames, NASA Goddard Distributed data collection using the SRB ESIP federation Led by Joseph JaJa (U Md) Federation of ESIP data resources using the SRB NASA Goddard Data Management System Storage repository virtualization (Unix file system, Unitree archive, DMF archive) using the SRB NASA EOS Petabyte store Storage repository virtualization for EMC persistent store using the Nirvana version of SRB

13. San Diego Supercomputer Center SDSC Storage Resource Broker 13 NCSA 6+2 TF 4 TB Memory 400 TB disk SDSC 4.1 TF 2 TB Memory 500 TB SAN Caltech 0.5 TF.4 TB Memory 86 TB disk ANL 1 TF.25 TB Memory 25 TB disk 32 5 5 TeraGrid: 13.6 TF, 6.8 TB memory, 900 TB network disk, 10 PB archive HPSS 9 PB ESnet HSCC MREN/Abilene Starlight 32 24 8 32 24 8 4 Juniper M160 OC-12 OC-48 OC-12 574p IA-32 Chiba City 128p Origin HR Display & VR Facilities 256p HP X-Class 128p HP V2500 92p IA-32 Myrinet Chicago & LA DTF Core Switch/Routers Cisco 65xx Catalyst Switch (256 Gb/s Crossbar) OC-12 OC-3 vBNS Abilene MREN 1176p IBM SP 1.7 TFLOPs Blue Horizon OC-48 NTON 4 4 2 x Sun E10K 4 15xxp Origin UniTree 1024p IA-32 320p IA-64 2 14 8 vBNS Abilene Calren ESnet OC-12 OC-3 8 Sun Server 16 GbE 24 Extreme Blk Diamond OC-12 ATM Calren

14. San Diego Supercomputer Center SDSC Storage Resource Broker 14 NIH BIRN SRB Data Grid Biomedical Informatics Research Network Access and analyze biomedical image data Data resources distributed throughout the country Medical schools and research centers across the US Stable high performance grid based environment Coordinate data sharing Federate collections Support data mining and analysis

15. San Diego Supercomputer Center SDSC Storage Resource Broker 15 BIRN: Inter-organizational Data

16. San Diego Supercomputer Center SDSC Storage Resource Broker 16 SDSC SRB User Community (Major US) BaBar, Stanford Linear Accelerator Center (SLAC) California Digital Library (CDL) Center for Integrated Space Weather Modeling (CISM) CVC, Visualization Portal LDC Data Storage NIH Bio Informatics Research Network (BIRN) NSF Southern California Earthquake Center (SCEC) National Archives and Records Administration (NARA) National Aeronautics and Space Administration Centers (NASA) National Virtual Observatory (NVO) Npackage, NSF Middleware Initiative (NMI) National Science Digital Library (NSDL) National Optical Astronomy Observatory (NOAO) ROADNet Purdue University SCCOOS, USA Scientific Rich Media Archive Salk Institute Strand Map Service, USA UC Berkeley Library UCSD Library University of Houston Persistent Archives Test bed University of Wisconsin, Madison WebBase, Stanford University Yale University Library

17. San Diego Supercomputer Center SDSC Storage Resource Broker 17 SDSC SRB User Community Academia Sinica, Taiwan Australian National University Bio-Lab, University of Genoa, Italy Council for the Central Laboratory of the Research Councils (CCLRC), UK CC-IN2P3, France Distributed Framework, Singapore Distributed Aircraft Maintenance Environment (DAME), UK eMinerals Project, UK eScience, Belfast Center Fraunhofer ITWM, Germany High Energy Accelerator Organization, KEK, Japan K* Grid Computing, Korea KEK Computing Center, Japan Lyon, France NorGrid, Norway Nanyang Data Grid, Singapore Queensland University of Technology (QUT), Australia Rutherford Appleton Laboratory (RAL), UK T-Systems, Germany UK eScience Project, UK UniGrid, Poland UMK, Poland Virtual Laboratory for eScience, Netherlands

18. San Diego Supercomputer Center SDSC Storage Resource Broker 18 What problem, why SRB solution? Why are people using SRB? What problems did it solve for them? Who are these people? Did they use it because they liked Arun

19. San Diego Supercomputer Center SDSC Storage Resource Broker 19 Why they use SRB? Distributed unstructured data management Data Grids, Digital Libraries, Persistent Archives, Workflow/dataflow Pipelines, Knowledge Generation Distributed data storage provisioning Common logical namespace for data and storage Data publication Browsing and discovery of data in collections Data Preservation Management of technology evolution

20. San Diego Supercomputer Center SDSC Storage Resource Broker 20 324 TB 358 TB 682 TB Total data brokered by SDSC SRB

21. San Diego Supercomputer Center SDSC Storage Resource Broker 21 Countries actively using SDSC SRB

22. San Diego Supercomputer Center SDSC Storage Resource Broker 22 Countries actively using SDSC SRB

23. San Diego Supercomputer Center SDSC Storage Resource Broker 23 Looking back… 1995: MDAS Project by DARPA 1998: SRB Releases 2000: Arun joins SRB Only after that SRB becomes a hit – lucky guy (just kidding) 2000 ++: Multiple client interfaces, Many more functionalities, Multiple projects across the world 2005: NCHC demonstrates significant interest in SRB and also their end-users in Taiwan (through this workshop)

24. San Diego Supercomputer Center SDSC Storage Resource Broker 24 Physical Layer (Real World) Distributed digital entities Heterogeneous and distributed storage resources Autonomous Organizations Distributed Users, distributed authentication Heterogeneous authorization schemes Users; sub-organizations; organizations/enterprises; virtual organizations

25. San Diego Supercomputer Center SDSC Storage Resource Broker 25 Data Grid Transparencies/Virtualizations (bits,data,information,..) Storage Resource Transparency Storage Location Transparency E:\srbVault\image.jpg /users/srbVault/image.jpg Select … from srb.mdas.td where... Data Identifier Transparency image_0.jpg…image_100.jpg Data Replica Transparency image.sqlimage.cgiimage.wsdl Virtual Data Transparency patientRecordsCollectionmyActiveNeuroCollection Inter- organizational Information Storage Management

26. San Diego Supercomputer Center SDSC Storage Resource Broker 26 We are SRB Arun is here! - Shameless Self promotion Not in picture: Many students