17/09/2004 John Kewley Grid Technology Group Introduction to Condor

John Kewley Grid Technology 17 th September 2004 Outline oWhat is Condor? oWhat can it be used for? oStatus of DL Condor Pool(s)

John Kewley Grid Technology 17 th September 2004 What is Condor? oA job submission framework which utilises spare computing power within a heterogeneous computer network (Condor pool) oIt supports High-Throughput Computing (HTC), maximising the amount of processing capacity that is utilised over long periods of time. oDeveloped over many years at University of Wisconsin – Madison

John Kewley Grid Technology 17 th September 2004 Basic Features oA Condor pool is a set of resources (clusters, servers and networked workstations), managed by a Central Manager oThe Central Manager matches requests for resources with those resources available within the pool oUser does not need account on machine where job runs, but may submit jobs to the pool from his/her workstation. oHighly extensible resource description and job requirements language which is used to classify/advertise the resources in the pool. oAvailable on multiple platforms.

John Kewley Grid Technology 17 th September 2004 Supported platforms ArchitectureOperating System Hewlett Packard PA-RISC (PA PA8000)HPUX Sun SPARC Sun4m,Sun4c, Sun UltraSPARCSolaris 2.6, 2.7, 2.8, 2.9 Silicon Graphics MIPS (R5000, R8000, R10000)IRIX 6.5 (clipped) Intel x86Red Hat Linux 7.1, 7.2, 7.3, 8.0 Red Hat Linux 9 Windows 2000 Prof + Server, 2003 Server (clipped) Windows XP Professional (clipped) ALPHADigital Unix 4.0 Red Hat Linux 7.1, 7.2, 7.3 (clipped) Tru (clipped) PowerPCMacintosh OS X (clipped) AIX 5.2L (clipped) ItaniumRed Hat Linux 7.1, 7.2, 7.3 (clipped) SuSE Linux Enterprise 8.1 (clipped)

John Kewley Grid Technology 17 th September 2004 Execute MachineSubmit Machine Job Startup Submit Schedd Starter Job Shadow Condor Syscall Lib Startd Central Manager CollectorNegotiator Slide courtesy of University of Wisconsin-Madison

John Kewley Grid Technology 17 th September 2004 Additional Features oCheckpointing and migration of jobs oShared filestore is not required, but can be utilised oInterworking with Globus, oSecurity: GSI, Kerberos oUse of MPI and PVM oWorkflow using DAGMan (Directed Acyclic Graph Manager). oWindows + Unix + Linux + …

John Kewley Grid Technology 17 th September 2004 Execution Environments standard oMust be relinked under condor oSystem calls occur on the submitting resource oJobs may checkpoint and hence be stopped and later restarted from its last checkpoint, and may migrate to another resource oNot available on some platforms (e.g. Windows) oSome restrictions on what can be run. vanilla oAny executable or script, no need for relinking or access to object files oSystem calls happen on the executing resource oNo checkpointing, not so good for long-running jobs. If a job is stopped it will be rescheduled (i.e. compute time is lost). oWorks on all supported platforms (incl Windows) oSome opening of file permissions may be required

John Kewley Grid Technology 17 th September 2004 Possible Uses oUse vanilla universe for jobs which comprise many small (comparatively), independent tasks. oUse standard universe for jobs which will run for long periods. oUtilise the “odds and ends” of the pool for compilation and build tests.

John Kewley Grid Technology 17 th September 2004 Condor Pools at DL oInternal Pool  5 Windows 3x Windows XP Professional 2x Windows 2000 Professional  18 Linux 6x SuSE Linux 9.0 2x SuSE Linux 8.0 5x White Box Enterprise Linux 3.0 3x Red Hat Linux 9 1x Mandrake Linux x Gentoo Linux 1.4 oExternal Pool  6 Linux 2x Red Hat Linux 7.3 4x White Box Enterprise Linux 3.0

John Kewley Grid Technology 17 th September 2004 Build and Test oOur External Pool is being used by the OMII (Open Middleware Infrastructure Institute) for building and testing their latest Grid middleware. oWe intend extending the use of this pool for use as a build and test pool for other Institutions on the UK Grid. oOur internal users are also keen to utilise this build technology to build release packages of their software for many different platforms.

John Kewley Grid Technology 17 th September 2004 User Status We are currently at an early stage with our user community and are helping them setup their code so that it can be run conveniently under Condor. These users are from the following computational science communities: o CCP1 - The electronic structure of molecules o CCP4 - Protein crystallography

John Kewley Grid Technology 17 th September 2004 Summary oCondor can utilise otherwise unused resources (e.g. Windows workstations overnight) oUse vanilla universe for jobs which comprise many small (comparatively), independent tasks oUse standard universe for jobs which will run for long periods (although not on Windows) oCan be used for compilation and build tests