Batch Software at JLAB Ian Bird Jefferson Lab CHEP February, 2000

Ian Bird / Jefferson LabCHEP Introduction Environment –Farms –Data flows –Software Batch systems –JLAB software –LSF vs. PBS Scheduler Tape software –File pre-staging/caching

Ian Bird / Jefferson LabCHEP Environment Computing facilities were designed to: –Handle data rate of close to 1 TB/day –1 st level reconstruction only (2 passes) Match average data rate –Some local analysis but mainly export of vastly reduced summary DSTs Originally estimated requirements: –~ 1000 SI95 –3 TB online disk –300 TB tape storage – 8 RedWood drives

Ian Bird / Jefferson LabCHEP Environment - real After 1 year of production running of CLAS (largest experiment) –Detector is far cleaner than anticipated, which means: Data volume is less ~ 500 GB/day Data rate is 2.5x anticipated (2.5 kHz) Fraction of good events larger DST sizes are same as Raw data (!) –Per event processing time is much longer than original estimates –Most analysis is done locally – no-one is really interested in huge data exports Other experiments also have large data rates (for short periods)

Ian Bird / Jefferson LabCHEP Computing implications CPU requirement is far greater –Current farm is 2650 SI95 and will double this year Farm has a big mixture of work –Not all production – “small” analysis jobs too –We make heavy use of LSF hierarchical scheduling Data access demands are enormous –DSTs are huge, many people, frequent accesses –Analysis jobs want many files Tape access became a bottleneck –Farm can no longer be satisfied

JLab Farm Layout Gigabit Ethernet Quad SUN E4000 STK Redwood Tape Drives Fast Ethernet Gigabit Ethernet SCSI2 FWD SCSI2 UWD/S Fast Ethernet Dual PII 450MHz Qty GB UWS Dual PII 400MHz Qty GB UWS Cisco Cat 5500 Quad SUN E GB stage 150GB 200GB stage MetaStor SH7400 File Server 3TB UWD work MetaStor SH7400 File Server 3TB UWD work Plan - FY 2000 STK 9840 Tape Drives FARM SYSTEMS MASS STORAGE MASS STORAGESERVERS WORK FILE SERVERS WORK FILE SERVERS Cisco 2900 Gigabit Ethernet Cisco 2900 Dual PIII 500MHz Qty GB UWS Dual PIII 650MHz Qty GB UWS Dual Sun Ultra2 400GB UWD Dual Sun Ultra2 400GB UWD CACHE FILE SERVERS CACHE FILE SERVERS Dual Sun Ultra2 400GB UWD Dual Sun Ultra2 400GB UWD Cisco 2900 Dual PII 300MHz Qty GB FWD

Ian Bird / Jefferson LabCHEP Other farms Batch farm –180 nodes -> 250 Lattice QCD –20 node Alpha (Linux) cluster –Parallel application development –Plans (proposal) for large 256 node cluster Part of larger collaboration Group want a “meta-facility” –Jobs run on least loaded cluster (wide area scheduling)

Ian Bird / Jefferson LabCHEP Additional requirements Ability to handle and schedule parallel jobs (MPI) Allow collaborators to “clone” the batch systems and software –Allow inter-site job submission –LQCD is particularly interested in this Remote data access

Ian Bird / Jefferson LabCHEP Components Batch software –Interface to underlying batch system Tape software –Interface to OSM, overcome limitations Data caching strategies –Tape staging –Data caching –File servers

Ian Bird / Jefferson LabCHEP Batch software A layer over the batch management system –Allow replacement of batch system LSF, PBS (DQS) –Constant user interface no matter what the underlying system is –Batch farm can be managed by the management system (e.g. LSF) –Build in a security infrastructure (e.g GSI) Particularly to allow remote access securely

Batch control system LSF, PBS, DQS, etc. Job submission system Submission interface Database Query interface User processes Submission, query, statistics Batch processors Batch system - schematic

Ian Bird / Jefferson LabCHEP Existing batch software Has been running for 2 years –Uses LSF –Multiple jobs – parameterized jobs (LSF now has job arrays, PBS does not have this) –Client is trivial to install on any machine with a JRE – no need to install LSF, PBS etc. Eases licensing issues Simple software distribution Remote access –Standardized statistics and bookkeeping outside of LSF MySQL based

Ian Bird / Jefferson LabCHEP Existing software cont. Farm can be managed by LSF –Queues, hosts, scheduler etc. Rewrite in progress to: –Add PBS interface (and DQS?) –Security infrastructure to permit authenticated remote access –Clean up

Ian Bird / Jefferson LabCHEP PBS as alternative to LSF PBS (Portable Batch System – NASA) –Actively developed –Open, freely available –Handles MPI (PVM) –User interface very familiar to NQS/DQS users –Problem (for us) was the (lack of a good) scheduler PBS provides only a trivial scheduler, but Provides mechanism to plug in another We were using hierarchical scheduling in LSF

Ian Bird / Jefferson LabCHEP PBS scheduler Multiple stages (6), can be used or not as required, in arbitrary order –Match making – matches requirements to system resources –System priority (e.g. data available) –Queue selection (which queue runs next) –User priority –User share: which user runs next, based on user and group allocations and usage –Job age Scheduler has been provided to PBS developers for comments – and is under test

Ian Bird / Jefferson LabCHEP Mass storage Silo – 300 TB Redwood capacity –8 Redwood drives –5 (+5) 9840 drives –Managed by OSM Bottleneck: –Limited to a single data mover –That node has no capacity for more drives –1 TB tape staging RAID disk 5 TB of NFS work areas/caching space

Ian Bird / Jefferson LabCHEP Solving tape access problems Add new drives – 9840’s –Requires 2 nd OSM instance Transparent to user Eventual replacement of OSM Transparent to user File pre-staging to the farm Distributed data caching (not NFS) Tools to allow user optimization Charge for (prioritize) mounts

Ian Bird / Jefferson LabCHEP OSM OSM has several limitations (and is no longer supported) –Single mover node is most serious No replacement possible yet Local tapeserver software solves many of these problems for us –Simple remote clients (Java based) – do not need OSM except on server

Ian Bird / Jefferson LabCHEP Tape access software Simple put/get interface, –Handles multiple files, directories etc. Can have several OSM instances, but a unique file catalog, transparent to user –System fails over between servers Only way to bring 9840’s on line Data transfer is network (socket) copy in Java Allows a scheduling/user allocation algorithm to be added to tape access Will permit “transparent” replacement of OSM

Ian Bird / Jefferson LabCHEP Data pre-fetching & caching Currently –Tape – stage disk – network copy to farm node local disk –Tape – stage disk – NFS cache – farm But this can cause NFS server problems Plan: –Dual solaris nodes with ~ 350 GB disk (RAID 0) Gigabit ethernet Provides large cache for farm input –Stage out entire tapes to cache Cheaper than staging space, better performance than NSF Scaleable as the farm grows

JLab Farm Layout Gigabit Ethernet Quad SUN E4000 STK Redwood Tape Drives Fast Ethernet Gigabit Ethernet SCSI2 FWD SCSI2 UWD/S Fast Ethernet Dual PII 450MHz Qty GB UWS Dual PII 400MHz Qty GB UWS Cisco Cat 5500 Quad SUN E GB stage 150GB 200GB stage MetaStor SH7400 File Server 3TB UWD work MetaStor SH7400 File Server 3TB UWD work Plan - FY 2000 STK 9840 Tape Drives FARM SYSTEMS MASS STORAGE MASS STORAGESERVERS WORK FILE SERVERS WORK FILE SERVERS Cisco 2900 Gigabit Ethernet Cisco 2900 Dual PIII 500MHz Qty GB UWS Dual PIII 650MHz Qty GB UWS Dual Sun Ultra2 400GB UWD Dual Sun Ultra2 400GB UWD CACHE FILE SERVERS CACHE FILE SERVERS Dual Sun Ultra2 400GB UWD Dual Sun Ultra2 400GB UWD Cisco 2900 Dual PII 300MHz Qty GB FWD

Ian Bird / Jefferson LabCHEP File pre-staging Scheduling for pre-staging is done by the job server software –Splits/groups jobs by tape (could be done by user) –Makes a single tape request –Holds jobs while files are staged –Implemented by batch jobs that release held jobs –Released jobs with data available get high priority –Reduces job slots blocked by jobs waiting for data

Ian Bird / Jefferson LabCHEP Conclusions PBS is a sophisticated and viable alternative to LSF Interface layer permits –use of same jobs on different systems – user migration –Add features to batch system