July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 1 DØ RACE Status Report Introduction Software Distribution (DØRACE Setup) Hardware Infrastructure –DØRAM Architecture –Regional Analysis Centers Software Infrastructure(SAM-Grid) Conclusions Jae Yu July 10, 2002 DØ IB, Oklahoma Workshop

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 2 DØRACE ( R emote A nalysis C oordination E ffort ) Computing hardware is rather inexpensive –CPU and storage media are inexpensive –Small institutions can afford to own reasonable size computing facilities DØ collaboration is larger and more international –Most the collaborating institutions are remote –Code development can occur at remote stations Promote contribution of available human resources for software development Give ownership to collaborators from remote institutes –Optimal and efficient access to data is of utmost importance to expedite analyses –Minimize travel around the globe for data access –Exploit existing but scattered computing resources –Sociological issue of HEP people at the home institutions Sharing a 15-20fb -1 worth of raw and reconstructed data/MC (~9PB) efficiently is a big issue Primary goal is empowering individual desktop users

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 3 Efficient remote DØ software distribution –Allow remote participation for code development is a bottleneck in expediting physics results –Allow remote analysis for histogram production –Allow remote reconstruction or production environment Sufficient compute and storage hardware infrastructure Optimized resource management tools –Allow to maximally utilize offsite resources –Allow participation of remote resources for collaboration’s needs Efficient and transparent data delivery and sharing –Allow location independent access to sufficiently large data sets throughout the entire network of collaboration –Minimize central data storage dependence –Alleviate load for central data storage and servers What do we need?

Software Distribution (DØRACE Setup)

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 5 Identified Difficulties –Having hard time setting up initially Lack of updated documentation Rather complicated set up procedure Lack of experience  No forum to share experiences –OS version differences (RH6.2 vs 7.1), let alone OS –Most the established sites have easier time updating releases –Network problems affecting successful completion of large size releases (4GB) takes a couple of hours (SA) –No specific responsible persons to ask questions –Availability of all necessary software via UPS/UPD –Time difference between continents affecting efficiencies From the Nov. Survey

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 6 Categorized remote analysis system set up by the functionality –Desk top only –A modest analysis server –Linux installation –UPS/UPD Installation and deployment –External package installation via UPS/UPD CERNLIB Kai-lib Root –Download and Install a DØ release Tar-ball for ease of initial set up? Use of existing utilities for latest release download –Installation of cvs –Code development –KAI C++ compiler –SAM station setup DØRACE Setup Strategy Phase I Rootuple Analysis Phase 0 Preparation Phase II Executables Phase III Code Dev. Phase IV Data Delivery

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 7 Regular bi-weekly meetings (9am, on-week Thursdays, in 9 th circle and in the Saturn VR) –Updates for releases –Sites report in the meetings Status Difficulties/Issues Solutions –Featured topics of common interests –Slides 100% posted on the web before the meeting –Meeting 100% on VRVS since June 5, 2002 (thanks to VCTF&SVCC) Instructions for setup regularly updated and posted on the DØRACE web page  Setup is easier Automatic release ready notification system in place Releases are split in two packages (binaries and sources) to alleviate network dependencies

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 8 Progressive

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 9 DØRACE Deployment Map (US and EU only) Processing Center Analysis Site w/ SAM Analysis Site w/o SAM No DØRACE Excuse me of my poor geography and missing continents!! You are welcome to provide me updates on the maps. Incomplete

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 10 distribution list used for exchange of information from –Help each other –Find bugs and report for fixes –Accumulate expertise Given the number of sites with setup, the load to releases managers (2) is not large Run-time environment packages for consistent executable McFarm control software distributed to a few new farms and tested running this afternoon with basic grid tools A total of 38 institutions ready for code development –About a dozen SAM sites active  Must use them now!!! We have established rather stable software distribution system What next???

DØRAM Hardware Infrastructure (DØRAC)

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 12 DØ RAC Working Group Formed after the Feb. DØRACE workshop –fully characterize RACs –Address issues Members: –Iain Bertram (Lancaster), Chip Brock (MSU), Frank Filthaut (NIKHEF), Lee Lueking (FNAL), Peter Maettig (Wuppertal), Meena Narain (Boston), Bruno Thooris (Saclay), Jae Yu (UTA), Christian Zeitnitz (Meinz) The group met every week via video for about 1.5 months prior to the Computing review for a timely input to the management –The review committee commended our aggressive approach to RAM The hard work of the group result in …. –Proposal for DØ Regional Analysis Centers (DØ Note #3984) –

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 13 Characterize use of RAC in a representative analysis Proposed DØRAM Characteristics of the data to be stored Services to be provided by DØRACs Requirements for DØRACs –Storage space justification –Compute resource justification Possible candidate sites and their capabilities Prototype RAC project Organizational and bureaucratic issues Implementation time scale Conclusions with lots of issues need to be addressed The DØRAC Proposal

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 14 Central Analysis Center (CAC) Desktop Analysis Stations DAS …. DAS …. Regional Analysis Centers Provide various services Institutional Analysis Centers IAC... IAC …. IAC Normal Interaction Communication Path Occasional Interaction Communication Path DØ Remote Analysis Model (DØRAM) RAC ….

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 15 What is a DØRAC? An institute with large concentrated and available computing resources An institute willing to provide services to a few small institutes (IAC) in the region and to the collaboration An institute willing to provide increased infrastructure as the data from the experiment grows An institute willing to provide support personnel

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 16 What services does DØRAC provide? Service to IAC’s: –Accept and execute analysis batch job requests –Provide cache and storage space –Store and provide access to desired data sets –Provide database access –Provide intermediary code distribution Services to Collaboration –Generate and reconstruct MC data set –Participate in re-reconstruction of data Provide manpower support for the above activities

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 17 Regional Analysis Center Requirements Located in geographically and infrastructure sensible place Sufficiently large bandwidth to FNAL, other RACs, and IACs Large storage space (robotic and/or) disk to store –100% TMB in each RAC –100% DST in the sum of all RACs, distributed randomly –Store MC data set Sufficiently large compute resources Support for the infrastructure and maintenance

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 18 Chip’s Sears Model of Categorization Best RACs: –Gbit or better network bandwidth –Robotic tape storage: ~170TB –Disk storage space: ~110TB –Compute resources: ~50 cpu/year/RAC –Provide database proxy service –Cost: ~$1M/year Good RACs: –Gbit or better network bandwidth –Disk storage: ~60TB –Compute resources:~50cpu/year/RAC –Provide database proxy service –Cost: $300k~$1M Better

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 19 Other Issues Obtaining personnel support commitment Serious MOU structure to spell out commitment Sharing resources with other experiments and discipline –Emergency resource loans –Technical conflicts, such as difference in OS Need a world-wide management structure –How do we resolve and allocate resources? –How is the priority within the experiment between physics groups determined? –How do we address issues that affect other experiment and discipline?

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 20 Proposed DØRAC Implementation Timescale Dec. 1, 2002: Implement Prototype RAC (pRAC project) –Cluster associated IAC’s –Transfer Thumbnail data set constantly from CAC to the RAC –Implement services to IAC’s –Monitor activities Jan. 2003: Workshop on RAC Mar. – Aug. 2003: Establish and initiate site selection process Mar. – Oct. 2003: Establish and negotiate MOU agreements with RAC institutes Jan. 31, 2004: Fully deploy and activate RACs with sufficient capacity

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 21 Funding…. The proposal is a good place to point back at but –It needs to be adopted something the collaboration wants to be implemented –This process must be completed swiftly (by Oct. 1?) so that we can tap into the funding agencies with a few proposals European, South America, and Asian countries seem to already have established national policies for funding HEP computing, independent of experiments U.S. needs some 3 or so RACs –UTA has won a ~$1.35M MRI through CSE primarily for DØRAC  Barely enough for Run IIa –More funding necessary to establish two more sites and software development support MRI and ITR funding

DØRAM Software Infrastructure (SAM-Grid) Project to include Job and Information Management with the SAM Data Management System Project started in 2001 as part of the PPDG collaboration to handle DØ’s expanded needs. Recently included CDF Current SAM-Grid team includes: –Andrew Baranovski, Gabriele Garzoglio, Lee Lueking, Dane Skow, Igor Terekhov, Rod Walker (Imperial College), Jae Yu (UTA), Drew Meyer (UTA), Tomasz Wlodek in Collaboration with U. Wisconsin Condor team.

July 10, 2002DØRACE Status Report, Jae Yu DØ IB Meeting, OU Workshop 23 Conclusions DØ software setup deployed (38 sites w/ code development)  Must start using the setup DØRAC Working group has completed its work and submitted a proposal (DØ Note #3984) to the collaboration Time for the collaboration to act upon it and start implementation  –A committee to look into and evaluate the proposal w/ recommendations Need a volunteer site as the pilot RAC site  Does not have to be a full site later on, though desirable –Need to prove implementation of concepts –Work out issues prior to full site implementations Must write proposals to acquire funds for the next sites –UTA MRI is a good starting point but… –We need more sites in Europe, S. America, Asia and 2-3 more US sites –Need funds for software and personnel supports Need serious participation in software infrastructure development