Software Implementation Junwei Cao (曹军威) and Junwei Li (李俊伟) Tsinghua University Gravitational Wave Summer School Kunming, China, July 2009

Computing Challenges Large amount of data Parallel and distributed processing Clusters and grids Clusters: collection of workstations Grids: collection of clusters/supercomputers Software becomes the key LIGO data analysis software working group (DASWG) Future computing infrastructures

The LSC Data Grid (LDG) Cardiff Birmingham• AEI/Golm

The LDG Software Stack LDAS DMT LALApps Matlab End users & applications Application enabling LDM LDGreport Glue Onasys LSC Job management LSC Data management LDR LSCdataFind LSCsegFind The LSC Data Grid Client/Server Environment Version 4.0 (based on VDT 1.3.9) LSCcertUtils LSC CA LSC Security management Applications Infrastructures Condor-G Worklfow management / Condor DAGman VDS VOMS Catalog service / Globus Resource location service / Globus Metadata service Resource management / Globus GRAM Data transfer / GridFTP Job scheduling / Condor Grid security / Globus GSI Middleware / Services Operating Systems and … FC4 GCC Python Autotools MySQL

The LDG 4.0 Client/Server Written in Pacman 3 Based on VDT 1.3.9 …… platformGE( 'Linux' ); package( 'Client-Environment' ); cd( 'vdt' ); package( 'VDT_CACHE:Globus-Client' ); package( 'VDT_CACHE:CA-Certificates' ); package( 'VDT_CACHE:Condor' ); package( 'VDT_CACHE:Fault-Tolerant-Shell' ); package( 'VDT_CACHE:GSIOpenSSH' ); package( 'VDT_CACHE:KX509' ); package( 'VDT_CACHE:MyProxy' ); package( 'VDT_CACHE:PyGlobus' ); package( 'VDT_CACHE:PyGlobusURLCopy' ); package( 'VDT_CACHE:UberFTP' ); package( 'VDT_CACHE:EDG-Make-Gridmap' ); package( 'VDT_CACHE:Globus-RLS-Client' ); package( 'VDT_CACHE:VDS' ); package( 'VDT_CACHE:VOMS-Client' ); cd(); package( 'Client-FixSSH' ); package( 'Client-RLS-Python-Client' ); package( 'Client-Cert-Util' ); package( 'Client-LSC-CA' ); OR platformGE( 'Sun' ); package( 'SolarisPro' ); platformGE( 'MacOS' ); package( 'Mac' ); Written in Pacman 3 Based on VDT 1.3.9 Support LDG:Client and LDG:ClientPro Support multiple platforms: FC4, Solaris and Darwin at client and only FC4 at server Support both 32bit and 64bit machines Server includes client Online documentation for step-by-step installation

Data Discovery (LSCdataFind) Metadata observatory = L dataType = RDS_R_L3 startGPS = 751658016 endGPS = 751658095 Metadata Service LFNs L-RDS_R_L3-751658016-16.gwf L-RDS_R_L3-751658032-16.gwf L-RDS_R_L3-751658048-16.gwf L-RDS_R_L3-751658064-16.gwf L-RDS_R_L3-751658080-16.gwf Resource Location Service @ldas.mit.edu PFNs (for both local and remote) file://localhost/data/node10/frame/S3/L3/LLO/L-RDS_R_L3-751658016-16.gwf gsiftp://ldas.mit.edu/data/node10/frame/S3/L3/LLO/L-RDS_R_L3-751658016-16.gwf file://localhost/data/node11/frame/S3/L3/LLO/L-RDS_R_L3-751658032-16.gwf gsiftp://ldas.mit.edu/data/node11/frame/S3/L3/LLO/L-RDS_R_L3-751658032-16.gwf file://localhost/data/node12/frame/S3/L3/LLO/L-RDS_R_L3-751658048-16.gwf gsiftp://ldas.mit.edu/data/node12/frame/S3/L3/LLO/L-RDS_R_L3-751658048-16.gwf file://localhost/data/node13/frame/S3/L3/LLO/L-RDS_R_L3-751658064-16.gwf gsiftp://ldas.mit.edu/data/node13/frame/S3/L3/LLO/L-RDS_R_L3-751658064-16.gwf file://localhost/data/node14/frame/S3/L3/LLO/L-RDS_R_L3-751658080-16.gwf gsiftp://ldas.mit.edu/data/node14/frame/S3/L3/LLO/L-RDS_R_L3-751658080-16.gwf

Data Replication (LDR) 24/7 Operation LHO (Tier 0) LDRMaster LDRMetadataService LDRSchedule LDRTransfer (gridftp) Caltech (Tier 1) RLS Local Storage Module mysql LDRMaster LDRMetadataService LDRSchedule LDRTransfer (gridftp) MIT (Tier 2) RLS Local Storage Module mysql 1. update metadata lfn 2. get lfn pfn@caltech(remote) 3. generate lfn pfn@mit 4. gridftp pfn@caltech(remote) pfn@mit(local) 5. rlsadd lfn pfn@mit LLO (Tier 0)

Data Monitoring Toolkit (DMT) SMP client gui Name Server Server lsmp DMT Monitors Single data stream lmsg DMT Online Use Scenario – control-room type DMT Offline Use Scenario – standalone or grid enabled MonServer Multiple data streams Stdout Trigger files Alarm files Trend files …… base container xml /data/node10/frame/S3/L3/LHO/H-RDS_R_L3-751658016-16.gwf /data/node11/frame/S3/L3/LHO/H-RDS_R_L3-751658032-16.gwf /data/node12/frame/S3/L3/LHO/H-RDS_R_L3-751658048-16.gwf /data/node13/frame/S3/L3/LHO/H-RDS_R_L3-751658064-16.gwf /data/node14/frame/S3/L3/LHO/H-RDS_R_L3-751658080-16.gwf /data/node15/frame/S3/L3/LHO/H-RDS_R_L3-751658096-16.gwf /data/node16/frame/S3/L3/LHO/H-RDS_R_L3-751658112-16.gwf sigp html ezcalib xsil /data/node10/frame/S3/L3/LLO/L-RDS_R_L3-751658016-16.gwf /data/node11/frame/S3/L3/LLO/L-RDS_R_L3-751658032-16.gwf /data/node12/frame/S3/L3/LLO/L-RDS_R_L3-751658048-16.gwf /data/node13/frame/S3/L3/LLO/L-RDS_R_L3-751658064-16.gwf /data/node14/frame/S3/L3/LLO/L-RDS_R_L3-751658080-16.gwf /data/node15/frame/S3/L3/LLO/L-RDS_R_L3-751658096-16.gwf /data/node16/frame/S3/L3/LLO/L-RDS_R_L3-751658112-16.gwf dmtenv event …… trig …… frameio DMT Libraries

standalone run of rmon DMT offline monitor An Example DMT Monitor filelist1.txt multilist.txt /data/node10/frame/S3/L3/LLO/L-RDS_R_L3-751658016-16.gwf /data/node11/frame/S3/L3/LLO/L-RDS_R_L3-751658032-16.gwf /data/node12/frame/S3/L3/LLO/L-RDS_R_L3-751658048-16.gwf /data/node13/frame/S3/L3/LLO/L-RDS_R_L3-751658064-16.gwf /data/node14/frame/S3/L3/LLO/L-RDS_R_L3-751658080-16.gwf /data/node15/frame/S3/L3/LLO/L-RDS_R_L3-751658096-16.gwf /data/node16/frame/S3/L3/LLO/L-RDS_R_L3-751658112-16.gwf filelist1.txt filelist2.txt filelist2.txt rmon /data/node10/frame/S3/L3/LHO/H-RDS_R_L3-751658016-16.gwf /data/node11/frame/S3/L3/LHO/H-RDS_R_L3-751658032-16.gwf /data/node12/frame/S3/L3/LHO/H-RDS_R_L3-751658048-16.gwf /data/node13/frame/S3/L3/LHO/H-RDS_R_L3-751658064-16.gwf /data/node14/frame/S3/L3/LHO/H-RDS_R_L3-751658080-16.gwf /data/node15/frame/S3/L3/LHO/H-RDS_R_L3-751658096-16.gwf /data/node16/frame/S3/L3/LHO/H-RDS_R_L3-751658112-16.gwf standalone run of rmon DMT offline monitor [jcao@ldaspc1 rmon]$ export LD_LIBRARY_PATH=/opt/lscsoft/dol/lib [jcao@ldaspc1 rmon]$ ./rmon -opt opt -inlists multilist.txt Processing multi list file: multilist.txt Number of lists added: 2 Total data streams: 2 Processing frame list file: /home/jcao/rmon/filelist1.txt Number of files added: 1188 Total frame files: 1188 Processing frame list file: /home/jcao/rmon/filelist2.txt channel[1]=H1:LSC-AS_Q channel[2]=L1:LSC-AS_Q startgps=751658000 stride=16 r-statistic=-0.00251782 startgps=751658016 stride=16 r-statistic=-0.0122699 startgps=751658032 stride=16 r-statistic=0.0168868 …… opt stride 16.0 channel_1 H1:LSC-AS_Q channel_2 L1:LSC-AS_Q

The LDM Modules and Flowchart Other tools client server QUEUED REJECTED SCHEDULED LSCdataFind Server ldm_submit ldm_locate_script LSCdataFind LOCATING ldm_agent RELEASED condor_master LOCATED Globus Job Manager ldm_q ldm_exec_script condor_submit RUNNING ldm_rm FINISHED LDM_SITES ldm_agent [MIT] lscdatafindserver = ldas-gridmon.mit.edu globusscheduler = ldas-grid.mit.edu/jobmanager-condor environment = LD_LIBRARY_PATH=/dso-test/home/jcao/dol/lib [CIT] lscdatafindserver = ldas-gridmon.ligo.caltech.edu globusscheduler = ldas-grid.ligo.caltech.edu/jobmanager-condor environment = LD_LIBRARY_PATH=/dso-test/jcao/dol/lib [LHO] lscdatafindserver = ldas-gridmon.ligo-wa.caltech.edu globusscheduler = ldas-grid.ligo-wa.caltech.edu/jobmanager-condor [LLO] lscdatafindserver = ldas-gridmon.ligo-la.caltech.edu globusscheduler = ldas-grid.ligo-la.caltech.edu/jobmanager-condor environment = LD_LIBRARY_PATH=/data2/jcao/dol/lib LDM_CONFIG Condor [AGENT] RESOURCES = @MIT@CIT@LHO@LLO SITES = /home/jcao/ldm/etc/LDM_SITES EXEC = /home/jcao/ldm/bin/ldm_exec_script LOCATE = /home/jcao/ldm/bin/ldm_locate_script PID = /home/jcao/ldm/var/ldm.pid LOG = /home/jcao/ldm/var/ldm.log LDG = /home/jcao/ldg-3.0/ Modules developed or deployed Modules designed and underdeveloped

Data Monitoring Environment (LDM) grid-enabled run of rmon DMT offline monitor using LDM ldm.sub [jcao@ldaspc1 ~]$ cd ldm [jcao@ldaspc1 ldm]$ source setup.sh [jcao@ldaspc1 ldm]$ cd ../rmon [jcao@ldaspc1 rmon]$ ldm_agent [jcao@ldaspc1 rmon]$ ldm_submit ldm.sub Job test has been submitted. [jcao@ldaspc1 rmon]$ more ldm_test_condor.out Processing multi list file: ldm_test_CIT_multilist.txt Number of lists added: 2 Total data streams: 2 …… startgps=751658000 stride=16 r-statistic=-0.00251782 [job] id = test monitor = rmon args = -opt opt input = opt [data] observatory = @H@L type = @RDS_R_L3@RDS_R_L3 start = 751658000 end = 751676993 automatically generated Condor-G submission file Users are interfaced with a LIGO friendly language. universe = globus globusscheduler = ldas-grid.ligo.caltech.edu/jobmanager-condor log = ldm_test_condor.log output = ldm_test_condor.out error = ldm_test_condor.err should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = ldm_test_CIT_multilist.txt, ldm_test_CIT_filelist1.txt, ldm_test_CIT_filelist2.txt, /home/jcao/rmon/opt arguments = -inlists ldm_test_CIT_multilist.txt -opt opt environment = LD_LIBRARY_PATH=/dso-test/jcao/dol/lib executable = /home/jcao/rmon/rmon Queue Users do not bother with technical details of LSC data grid services. Data are located and file lists are generated automatically

Open Science Grid (OSG) High energy physics Bioinformatics Nanotechnology …… Grid of Grids 20 thousand CPUs Petabytes of data storage

Thank You ! Junwei Cao jcao@tsinghua.edu.cn http://ligo.org.cn