9 February 2000CHEP2000 Paper 3681 CDF Data Handling: Resource Management and Tests E.Buckley-Geer, S.Lammel, F.Ratnikov, T.Watts Hardware and Resources Organization of Data User View of Access to Data Batch queues Disk Management Tests

9 February 2000CHEP2000 Paper 3682 Hardware Resources; Organization of Data. Mixed flavor unix cluster (CPU resource). Fibre channel disk arrays on each node of cluster currently (disk resource). Tape drives and robot tape library (tape drives resource). Drives connected directly on each node. Concentrate in talk on resources during reading of data. Datasets, filesets, files of 1GB. Datasets: raw, primary, secondary… Tapes store a group of filesets. Associations in Datafile Catalog (see Paper 367).

9 February 2000CHEP2000 Paper 3683 User View

9 February 2000CHEP2000 Paper 3684 User View of Access to Data Batch queues to manage cpu cycles Access data only from disk, not tape. Staging jobs in parallel. Disk inventory manager package for shared disk space. Batch queues to manage tape drives.

9 February 2000CHEP2000 Paper 3685 Batch Queues LSF (Platform Computing) proposed. Fairshare scheduling. Combined quotas across queues desirable. CPU queues for analysis jobs: Allocate CPU cycles by group, by user, by special project. I/O queues for staging jobs: input, output, event pick. 1 tape drive per queue slot. I/O job cpu use is proportional to data volume. Allocate drives and data volume by group, user, project.

9 February 2000CHEP2000 Paper 3686 Disk Management By fileset (reduce bookkeeping overhead) Allow static filesets for important datasets Filesets remain on disk until space is needed. Use-reservation prevents deletion of fileset. Delete algorithm looks at frequency of use and time since last use-reservation. Allocate space algorithm uses quotas by group and user.

9 February 2000CHEP2000 Paper 3687 User Job and Disk Management User gives dataset Dataset converted to list of filesets Stager manages list and returns next fileset when asked. Stager Part of User Job: Maintains small buffer of use-reservations to keep ahead of analysis job Adds use-reservations for filesets on disk or spawns input staging jobs to maintain buffer Releases use-reservations when fileset processed

9 February 2000CHEP2000 Paper 3688 Effects of Disk Management Job processes filesets on disk first (different orders, different times) Multiple jobs using same fileset share staging jobs Fast analysis job gets multiple staging jobs Only a fraction of a dataset is present on disk at one time (conserves disk space).

9 February 2000CHEP2000 Paper 3689 Prototype Tests Set of basic queues on workstation (LSF) Basic staging software Simulated analysis jobs which process dummy data Set of big and small dummy datasets Basic CDF Data Catalog software with contents for this simulation Purpose is to test ideas on resource management, and evaluate how analysis jobs interact in a resource limited environment.

9 February 2000CHEP2000 Paper Prototype Scaled Down Environment Single cpu workstation, b0ib04 Staging disk 9 GB Filesets of size 0.5 GB 4 small 1GB i.e. 10% of disk 4 large 10 GB i.e. 100% of disk 2 cpu queues, short & long Analysis jobs with variable cpu time 4 execution slots for each cpu queue 2 simulated tape drives (2 slots in io queue) 1 real tape drive in Emass robot

9 February 2000CHEP2000 Paper Simulation Scenarios Purpose: Investigate effect of patterns of use by collaboration (CDF “spin” jobs, repetitive small dataset jobs) Exercise data access features Choosing scenarios: A. Short vs long job competition B. Several jobs using same big dataset (CDF “spin” jobs) C. Competition for tape drives and disk space

9 February 2000CHEP2000 Paper Some scenarios studied: One long job vs a stream of short jobs Three long jobs on same dataset, see figure Ten long jobs on same dataset Mixed set of different long jobs and users (6 jobs, 6 users, 4 datasets). Stream of short jobs vs 4 different long jobs The disk allowed 4 different big datasets to be processed together, as expected for this simulation. Extra staging jobs for the streams of short jobs occurred when expected (when contesting against 4 or more different big datasets).

9 February 2000CHEP2000 Paper Trial 45 Three Long Jobs

9 February 2000CHEP2000 Paper Trial 29 Stream of shorts vs 1 long, 1 short

9 February 2000CHEP2000 Paper Conclusions from Prototype Tests DIM/Stager worked well. Stager functions appropriate, simple. Gave guidance for full implementation (client/server structure, cleanup, admin functions) Limited test of LSF (batch queues) worked well.

9 February 2000CHEP2000 Paper Mock Data Challenge 1 During December 99 and January 00, CDF successfully tested the movement of MC simulated data from the online Level 3 trigger farm of processors to the tape library, and through the offline reconstruction farm back to the tape library. Many sub-groups were involved. The resource management methods discussed here were implemented and used but will not be stressed until the rate tests of Challenge 2 in Spring 2000.

9 February 2000CHEP2000 Paper Summary Resource management methods were explained. Prototype tests were extolled. Full implementation of methods is underway. More tests to come. CDF Engineering run occurs in August 00.