Wong11/29/ SHARING DATA USING THE STORAGE RESOURCE BROKER (SRB) Ken Wong The Applied Research Laboratory (ARL) and The Department of Computer Science Washington University in St. Louis

Wong11/29/ OUTLINE OF TALK l SRB and HPSS Overview l SRB Concepts and Examples l Alternatives to SRB l Other SRB Projects l Our Experience

Wong11/29/ WU DATA CACHE AND THE SRB

Wong11/29/ WU DATA CACHE l 1.4 TB DEC Storage Works RAID (Level 5) –2-processor Sun Enterprise 450, 1 GB main memory –622 Mbps ATM interface, 10/100 Mbps Ethernet interface –1.7 TB (raw) = 48 x x x 36 GB l Backups –Incremental: Tue, Wed, Thu –Full: Mon, Fri, Sat l Data Volume –Used: 560 GB –Burn Rate: 7.0 GB/week (This Year); 5.5 GB/week (Lifetime)

Wong11/29/ INSTALLATION HISTORY l Jun/Jul 98 : Sun host and then 432 GB RAID –3 year extended warranty and 3 year maintenance on controllers l Sep 98 : SRB l Aug 99 : 24 x18.2 GB disks –3 year maintenance upgrade on controllers l Dec 99 : 24 x 36.4 GB disks

Wong11/29/ BRAINMAP DATA GROWTH

Wong11/29/ BRAINMAP DISK USAGE

Wong11/29/ STORAGE RESOURCE BROKER (SRB)

Wong11/29/ HIGH-PERFORMANCE STORAGE SYSTEM

Wong11/29/ HIGH-PERFORMANCE STORAGE SYSTEM l Current Usage –150 TB (terabytes; trillion) –15 million files l Current Capacity : 500 TBs of data (assuming a compression ratio of 1.5) l Projected Capacity : 1 PB (10^15) within a year

Wong11/29/ SRB CONCEPTS l SRB Server : Responds to SRB requests from clients l MCAT (Metadata Catalogue) –Information about data sets and collections (Oracle DB) l SRB Client l SRB Resource : A logical storage resource –Example: HPSS storage and container cache l Data Set : A file registered with the SRB l Collection : Group of registered data sets/collections l Container: Data sets stored as one physical unit –Container cache can be remote from HPSS

Wong11/29/ SRB SYSTEM CAPABILITIES l Collection-based management of data sets l Persistent identifiers for data sets l Management of data sets (copies or replicas) l Containers for aggregating data sets before archiving l Support for grid security infrastructure authentication –Uses public key certificates l Support for integrating data set collections across file systems, archives, and databases

Wong11/29/ SRB INTERFACES l Scommands (Unix commands) –Sinit/Sexit, Sput/Sget, Smkdir/Srmdir, Sls/Srm –Smkcont/Ssyncont, Slscont/Srmcont –SgetR/SgetU/SgetD l C-Programming API l Browser

Wong11/29/ PUBLISHING A DATA SET l Define the SRB environment (.srb/.MdasEnv file) mdasCollectionHome ‘/home/kenw.neurodb’ mdasDomainHome ‘neurodb’ srbUser ‘kenw’ srbHost ‘ghidorah.sdsc.edu’ defaultResource ‘cont-sdsc’ l Interact with SRB server %Sinit# Connect to SRB server %sls# See what is in my collection %Sput./mydata brain043# Copy file to SRB space %Schmod r public npaci brain043# Give read access %SgetD -a brain043# Check access permissions %Sexit# disconnect from SRB server

Wong11/29/ GETTING A DATA SET (SCOMMANDS) % Sinit % Scd /home/colin.neurodb# go to Colin's collection % Sls -l# see what is there % Sget colin_avg20_1.0mm_at0.5mm.mnc. # copy to this directory % Sexit

Wong11/29/ JINGHUA ZHOU'S WORK l Experiments –Test SRB functionality –Measures performance of basic SRB functions l Archiving (Perl Scripts) –Archive an arbitrary Unix directory to HPSS –Verify files were archived –Recover files from archival storage

Wong11/29/ RETRIEVAL EXPERIMENTS l Load 100 MB container with 1 MB files l Measure time required to retrieve N files l Divide time by N to get average time for each file l Repeat after container has been moved to tape l Repeat above steps for 10 MB container (instead of 100 MB)

Wong11/29/ AVERAGE RETRIEVAL TIME (OLD FILES)

Wong11/29/ AVERAGE RETRIEVAL TIME (FRESH FILES)

Wong11/29/ COMMENTS l SRB Overhead Per Object (File) –5-7 seconds (Early Measurements) –2-4 seconds (Recent Measurements) l Tape Overhead Per Object (File): 100 seconds l TCP Connection Needs Tuning –Assymetric routing, bottleneck,... –snoop and tcptrace analysis –Max Sget effective bandwidth is 8 Mbps –Max Sput effective bandwidth is 4 Mbps –Goal is 32 Mbps

Wong11/29/ ARCHIVING l Reflect Unix directory structure in SRB collection structure archiver NPACI/Unix account l Look for inactive files within a directory l Multiple versions handled by appending modification date to file name l Log all archival requests

Wong11/29/ CURRENT WORK l TCP Tuning and SRB Performance l Enhance Archival Scripts –Improve usability –Resilience to HPSS Blackouts –Parallel Archiving

Wong11/29/ RECENT SRB DEVELOPMENTS l Data Cutter l GSI authentication –UsesX.509 certificates l Container redesign – To handle multiple archival and cache resources l Remote proxy (Spcommand) l Textual annotation stored in MCAT

Wong11/29/ ALTERNATIVES TO SRB l Distributed Database –Do not deal with file data  Requires other means of accessing files –A heavyweight solution; i.e., expense (money, expertise) –Need instances running wherever you want to have storage –If it is only meta-data, then a case can be made but... Tied to a particular vendor at all sites Have to cross link all the databases l AFS (Andrew File System) –Doesn't have concept of application metadata SRB has some metadata facilities now and more to come Comments, annotations, user-controlled metadata –SRB provides a uniform authentication and authorization system

Wong11/29/ TOP SRB PROJECTS (SUMMARY) l 2-Micron All Sky Survey –10 TB of data from Caltech –5 million images sorted into 130,000 containers l Digital Embryo Project (NLM funded) –Digitizing existing slides for storage in HPSS l Particle Physics Data Grid (DOE funded) –Data mining l Information Power Grid (NASA funded) l Data Visualization Corridor (DOE funded) –Handles terabyte sized data sets for interactive viewing l Neuroscience Data Set Federation

Wong11/29/ TOP SRB PROJECTS l 2-Micron All Sky Survey (2MASS) –10 TB of data from Caltech (3 TB done) –5 million images sorted into 130,000 containers –SRB container technology used to manage the aggregation process on a disk cache –Replicate Caltech data l Digital Embryo Project (NLM funded) –Digitizing existing slides for storage in HPSS –SRB used to manage data movement, aggregation into containers, and metadata catalog –Queries against the collection l Particle Physics Data Grid (DOE funded) –Replicate data sets that are pulled into local disk caches

Wong11/29/ TOP SRB PROJECTS l Information Power Grid (NASA funded) –SRB used to support data mining against a distributed data set collection –Data transmission rate: 58 Mbps from SDSC to NASA Ames –Put collection management in front of storage archives through use of the MCAT l Data Visualization Corridor (DOE funded) –SRB has been integrated with the Data Cutter system For remote manipulation of data sets –Handles terabyte sized data sets for interactive viewing l Neuroscience Data Set Federation

Wong11/29/ CONCLUDING REMARKS l Documentation – – l Software –Follow SRB link –Get PGP key from SDSC –Can install subset (e.g., client only) l Applications?

Wong11/29/

Wong11/29/ WU DATA CACHE vBNS ghidorah(MCAT)hpssbrainmap (1.3 TB) stp, v1 (SUMS)petsun-23(Scanners) 622 Mbps 45 Mbps 155 Mbps ATM sdsc.eduwustl.edu UCSD, UCLA, John Hopkins, U. Montana, Caltech (12 Major Users)