1. Gregory Mendell LIGO Hanford Observatory
S6 Data Production/LSC Data Grid Plans:
RDS HOFT LDG
March 2008 LSC/DASWG F2F Meetings
Gregory Mendell
LIGO Hanford Observatory
LIGO-Gxxxxxxxxx-00-W
LIGO-G W
LIGO-G W

2. S5 Reduced Data Sets (RDS)
RDS frames are subsets of the raw data, with some channels downsampled
Level 1 RDS (/archive/frames/S5/L1; type == RDS_R_L1)
separate frames for LHO and LLO:
LHO: 131 H1 channels and 143 H2 channels
LLO: 181 L1 channels
Level 3 RDS (/archive/frames/S5/L3; type == RDS_R_L3)
DARM_ERR, STATE_VECTOR, and SEGNUM; separate frames for LHO and LLO.
Level 4 RDS (/archive/frames/S5/L4; type == [H1|H2|L1]_RDS_R_L1)
DARM_ERR downsampled by a factor 4, STATE_VECTOR, and SEGNUM; separate frame for H1, H2 and L1 data.

3. LIGO S5 Data Products Disk Space Tape Archiving @ 186 GBs/tape;
Dual Copy
Data Rate Yr Yr Comp. Ratio
LHO raw MB/s TBs tapes on tape
LHO Level 1 RDS MB/s TBs tapes in files
LHO Level 3 RDS MB/s TBs tapes in files
LHO Level 4 RDS MB/s TBs tapes in files
LHO Level 1 h(t) MB/s TBs tapes --
LHO Level 2 h(t) MB/s TBs tapes
LHO SFTs MB/s TBs tapes --
LLO raw MB/s TBs tapes on tape
LLO Level 1 RDS MB/s TBs tapes in files
LLO Level 3 RDS MB/s TBs tapes in files
LLO Level 4 RDS MB/s TBs tapes in files
LLO Level 1 h(t) MB/s TBs tapes
LLO Level 2 h(t) MB/s TBs tapes --
LLO SFTs MB/s TBs tapes --
Totals: MB/s TBs tapes as above
* All of Level 1 RDS will not necessary fit on cluster node disks and/or inside the tape library at the sites. A copy of all data of all types is stored at the sites on tape, either in the tape library or off-line in tape cabinets.

4. A5/S6 Change to RDS compression
The compression of the RDS frames has changed from gzip to zero_suppress_int_float_otherwise_gzip in the A5 Astrowatch data. This will continue with the S6 RDS data.
The new compression method is already in the C and C++ frame libraries, but not in the fast frame library used by dtt. A request to add this has been sent to Daniel Sigg.
Tests that show that zero_suppress_int_float_otherwise_gzip reduces the size of the Level 1 RDS frames by 30% and speeds up output (and probably input) by around 20%.

5. Data Products For S6 Raw Full Frame (L0) Data
Compress raw data in the frames? (test by John Zweizig indicate can reduce size by factor of 2)
Change frame duration from 32 s?
Level 1 RDS (RDS_R_L1) Data
Requires 65 TBs of disk space per year (LDAS Plans to keep this on disk. All IFOs at both sites?)
Need to finalize S6 channel list with input from DASWG/GlitchGroup/DetChar
Remove channels in Level 1 h(t) frames from Level 1 RDS frames?
Change 64 s frame duration?
Split LSC, ASC, PEM, SUS, FSR into separate frames?
Make Level 2 RDS of most used channels? (e.g., request by Soma)
Level 3 RDS (RDS_R_L3) Data
Change 256 s frame duration?
Level 4 RDS ([H1|H2|L1_RDS_R_L4) Data
Discontinued
Timing RDS (requested by Szabi)
93 Timing channels not in Level 1 RDS data at 31 MBs per 64 s frame file, for 15 TBs/yr of data.
E.G,: H0:DAQ-FB0_UPTIME SECONDS, H0:DAQ-FB1_UPTIME_SECONDS, H0:TIM-MSR_BYVAT_TIME_DIFF, H0:TIM-MSR_H2VAT_TIME_DIFF, H1:DAQ-GPS_RAMP_EX, H1:GDS-IRIGB_LVEA, H1:GDS-TIME_MON, H1:GPS-EX_ATOMIC_FAIL…
Online h(t) (HOFT) Data
LDAS will generate using Xavi’s code. See next slides.

6. S5/S6 RDS Generation Publishing/LDR LDAS RDS frames transfer
get segments from
LDAS diskcacheAPI
RDS frames transfer
to CIT via LDR
Driver Script
createrds.tcl uses
Ligotools LDASjob
package to submit jobs
run LDAS
createRDS jobs
MIT
UWM
PSU
AEI
RDS frames
are written to
archive filesystem

7. Will start testing during astrowatch!!!
NEW S6 h(t) Generation
Publishing/LDR
LDAS
get segments from
LDAS diskcacheAPI and LSCsegFind
NEW Driver Script run by LDAS:
h(t) frames transfer
to CIT via LDR
createhoft.tcl driver script has been written and simple tests run
Run LSCdataFind and Xavi’s code
Advantages:
Sys Admins run scripts, like RDS generation
Same monitoring/gap checking that makes RDS generation robust
Sys Admins aware of ldas down-time, problems
Local Sys Admins close to Calibration (Greg & Mike, Igor & Brian)
MIT
UWM
PSU
AEI
h(t) frames
are written to
archive filesystem
Will start testing during astrowatch!!!

8. RDS monitoring will include h(t) monitoring!
Click here to get directory listing; browse for scripts and adc*.txt file to get channel lists.

9. LSC Data Grid (LDG) LDG Client*: LDG ClientPro: LDG Server:
package( 'Client-Environment' );
package( 'VDT_CACHE:Globus-Client' );
package( 'VDT_CACHE:CA-Certificates' );
package( 'VDT_CACHE:Condor' );
package( 'VDT_CACHE:GSIOpenSSH' );
package( 'VDT_CACHE:KX509' );
package( 'VDT_CACHE:UberFTP' );
package( 'VDT_CACHE:EDG-Make-Gridmap' );
package( 'VDT_CACHE:VOMS-Client' );
package( 'Client-FixSSH' );
package( 'Client-Cert-Util' );
package( 'Client-LSC-CA' );
LDG ClientPro:
Same as Client, plus…
package( 'VDT_CACHE:MyProxy' );
package( 'VDT_CACHE:PyGlobus' );
package( 'VDT_CACHE:PyGlobusURLCopy' );
package( 'VDT_CACHE:Globus-RLS-Client' );
package( 'VDT_CACHE:Pegasus' );
package( 'VDT_CACHE:VOMS-Client' );
package( 'VDT_CACHE:Globus-Base-WSGRAM-Client' );
package( 'VDT_CACHE:Globus-WS-Client' );
package( 'VDT_CACHE:TclGlobus-Client' );
LDG Server:
package( 'Server-Environment' )
package( 'VDT_CACHE:Globus' )
package( 'VDT_CACHE:CA-Certificates' )
package( 'VDT_CACHE:CA-Certificates-Updater' )
package( 'VDT_CACHE:Condor' )
package( 'VDT_CACHE:GSIOpenSSH' )
package( 'VDT_CACHE:KX509' )
package( 'VDT_CACHE:MyProxy' )
package( 'VDT_CACHE:UberFTP' )
package( 'VDT_CACHE:EDG-Make-Gridmap' )
package( 'VDT_CACHE:Globus-RLS')
package( 'VDT_CACHE:Globus-Core')
package( 'VDT_CACHE:Globus-Condor-Setup' )
package( 'VDT_CACHE:PyGlobus' )
package( 'VDT_CACHE:PyGlobusURLCopy' )
package( 'VDT_CACHE:Pegasus' )
package( 'VDT_CACHE:VOMS-Client' )
package( 'VDT_CACHE:Globus-WS' )
package( 'VDT_CACHE:Tomcat-5.5' )
package( 'VDT_CACHE:TclGlobus' )
package( 'Server-FixSSH' )
package( 'Server-RLS-Python-Client' )
package( 'Server-Cert-Util' )
package( 'Server-LSC-CA' )
*Note that a generic install that builds globus client from source is available for platforms not supported by VDT, e.g., for Solaris & PowerPC

10. LDG Plans Support for CentOS 5 (done) and Debian Etch (next release)
Further support for Mac OS X on Intel (Leopard)
Most Frequent User Requests:
Make it work on my platform. Problems:
Missing dependencies, so should work towards improved dependency checking
Minor bug with work around (so should document fixes in easy to find location)
Lite version that just installs gsi-enabled ssh, certificate management tools
Help getting/renewing certificates (See Authentication Committee’s work)

11. LDG Middleware Packaging
from Stuart:
As suggested by Scott in the Feb 28 CompComm meeting I would like to initiate the CompComm Middleware Packaging Group to address how LIGO should manage the distribution of grid middleware. The initial group is to consist of Greg and Scott, though they are encouraged to recruit other expertise and opinions as needed. The primary questions to address are:
Should the LDG server bundle be replaced with the OSG stack? If so, what about sites, such as the Observatories, that are not running OSG jobs.
2) If we stick with LDG as a rebundle of the VDT:
a) Do we have the right split between client, clientpro, and server
(more or less splits?)
b) How should the Server bundle be enhanced to effectively support
WS-GRAM services (aka GT4)--e.g., VDT enhancement to support
distinct LDG_SERVER_LOCATION and VDT_LOCATION or modify LDG
to make these equal?
c) Should we actively push for the replacement of Pacman? and
if so what should we request, e.g., RPM, Solaris packages, DEB, ...
3) What LIGO applications should be modified to use standard LDG middleware
installations, e.g., LDAS is currently investigating the feasibility of
using TclGlobus and Globus from LDG Server rather than a parallel install
in /ldcg, what about LDR, others?

12. END

13. Channel Lists
Level 1:
Level 3:
Level 4:
L1:LSC-AS_Q
L1:LSC-AS_I
L1:LSC-POB_Q
...
L1:LSC-DARM_CTRL
L1:LSC-DARM_ERR
L0:PEM-EY_BAYMIC
L0:PEM-EX_BAYMIC
Note that one fast channel for 1 yr of S5 data can take up ~ 2 TBs of disk space.
Level 3:
H2:LSC-DARM_ERR
H2:IFO-SV_STATE_VECTOR
H2:IFO-SV_SEGNUM
H1:LSC-DARM_ERR
H1:IFO-SV_STATE_VECTOR
H1:IFO-SV_SEGNUM
Level 4:
H2:LSC-DARM_ERR