Download presentation
Presentation is loading. Please wait.
Published byPatience Harrell Modified over 8 years ago
1
Gridengine Configuration review ● Gridengine overview ● Our current setup ● The scheduler ● Scheduling policies ● Stats from the clusters
2
Gridengine Overview ● Accepts jobs from the outside world ● Puts jobs in a holding area until they can be run ● Sends jobs from the holding area to an execution device ● Manages running jobs ● Records details about finished job
3
Gridengine Overview (2) ● Four types of hosts – Execution: runs jobs. – Submit: allowed to submit jobs from – Master: schedules jobs. – Admin: allowed to run admin cluster from. ● Hosts can be many types but only one master (hot spare). ● Could run everything on one host...silly but possible.
4
Queues (Cluster Queues) ● Container for a class of jobs ● Can define specific resources – large memory machines – specific processor – architecture – time restricted (runtime or time of day/week) ● Contain one or more execution hosts ● Can be preemptive ● Can contain subqueues
5
Queues(2) ● Queue instance – Each queue is bound to an included execution host via a queue instance – Each execution host can have multiple queue instances attached. – Can have one or more job slots.
6
Simple configuration ● One cluster queue ● Each execution host has one queue instance ● Jobs are scheduled in FIFO. ● This is the default configuration gridengine ships with.
7
Our Hardware ● 4 clusters running gridengine – Lion: 64+ nodes (GX240) – Lutzow: 16 nodes (PE530) – Townhill: 34 nodes (PE1425 dual CPU) – Hermes: 24 nodes (PE1425 single CPU) ● 4 head nodes (1 per cluster) ● 1Tb local home directories ● 1Tb “scratch” space
8
Current setup ● All hosts are admin hosts ● Single “head node” configured as submit/master ● Execution hosts have ssh blocked ● Users ssh onto head node and submit jobs. – Actually they tend to run scripts which submit jobs – Lots of jobs – Not all of them will run properly.
9
The Scheduler Process
10
Prioritisation ● Prioritisation based on – Entitlement – Urgency – Custom ● Generates a Dispatch priority ● Real number based on combination of above.
11
Entitlement ● Priority based on users/groups ● Can be explicit(user A jobs before user B) ● Can allocate ratio of resources (group A get 60% CPU usage over, group B get 40%) ● Share tree allows the allocation to be spread over a defined time period. ● Need to configure information for users/groups
12
Share tree example
13
Urgency ● Deadline contribtion – Priority rises closer to deadline specified at submission ● Wait time contribution – Priority rises with time ● Resource contribution – Can assign urgency to a resource (Maltab licenses)
14
Custom ● Allows for prioritisation based on site specific requirements ● Run arbitrary script which alters priority. ● Defaults to posix priority (like nice) – Users can lower priority – Admin can raise priority
15
Summary ● Can control job execution based on – Queues: assign specific execution hosts for specific tasks or users/groups. Queues can be calendar controlled. – Scheduler: prioritise jobs based on who submitted them or what resources they require.
16
Current setup ● Single queue containing all nodes in a cluster ● Limited user/group support (FC5) ● Allocates equal priority to each user with jobs in pending queue.
17
It's mostly downhill from here
18
Gathering job data ● Sun dbwriter ● Java script runs on accounting/reporting file and populates postgresql database (42GB footprint). ● Data from Dec/Jan until yesterday “with holes” ● Difficult to analyse some jobs (parallel,stopped jobs)
19
How many jobs
20
Hmm thats a lot of short jobs
21
That's really a lot of short jobs ● Remember all those scripts? ● How many of these jobs actually run for any length of time?
22
How many jobs (>3min)
23
Remove the <3min jobs
29
Tentative conclusions ● Could add more submit hosts/backup scheduler for redundancy (virtualisation). ● Need to set up queue to handle short jobs with quick turnaround ● Also need preempted queue for longer running jobs. ● User scripts can muddy the water, can't assume quiet time for system admin tasks
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.