Testing the dynamic per-query scheduling (with a FIFO queue) Jan Iwaszkiewicz.

Slides:



Advertisements
Similar presentations
Scheduling Introduction to Scheduling
Advertisements

CPU Scheduling Tanenbaum Ch 2.4 Silberchatz and Galvin Ch 5.
Process Control Management Prepared by: Dhason Operating Systems AICT
UC Berkeley Job Scheduling for MapReduce Matei Zaharia, Dhruba Borthakur *, Joydeep Sen Sarma *, Scott Shenker, Ion Stoica 1 RAD Lab, * Facebook Inc.
Distributed Indexed Outlier Detection Algorithm Status Update as of March 11, 2014.
Parasol Architecture A mild case of scary asynchronous system stuff.
UC Berkeley Job Scheduling with the Fair and Capacity Schedulers Matei Zaharia Wednesday, June 10, 2009 Santa Clara Marriott.
Operating Systems 1 K. Salah Module 2.1: CPU Scheduling Scheduling Types Scheduling Criteria Scheduling Algorithms Performance Evaluation.
1 Traditional Unix Scheduling “Traditional” means –No consideration on realtime processes. –Typically 4.3BSD and SVR3 MFQ-based –Preemptive (time quantum.
The new The new MONARC Simulation Framework Iosif Legrand  California Institute of Technology.
Chapter 5: CPU Scheduling
Operating Systems (CSCI2413) Lecture 3 Processes phones off (please)
Distributed Process Management1 Learning Objectives Distributed Scheduling Algorithms Coordinator Elections Orphan Processes.
Statistics of CAF usage, Interaction with the GRID Marco MEONI CERN - Offline Week –
New CERN CAF facility: parameters, usage statistics, user support Marco MEONI Jan Fiete GROSSE-OETRINGHAUS CERN - Offline Week –
1 Status of the ALICE CERN Analysis Facility Marco MEONI – CERN/ALICE Jan Fiete GROSSE-OETRINGHAUS - CERN /ALICE CHEP Prague.
Staging to CAF + User groups + fairshare Jan Fiete Grosse-Oetringhaus, CERN PH/ALICE Offline week,
ADLB Update Recent and Current Adventures with the Asynchronous Dynamic Load Balancing Library Rusty Lusk Mathematics and Computer Science Division Argonne.
PROOF: the Parallel ROOT Facility Scheduling and Load-balancing ACAT 2007 Jan Iwaszkiewicz ¹ ² Gerardo Ganis ¹ Fons Rademakers ¹ ¹ CERN PH/SFT ² University.
Silberschatz, Galvin and Gagne  Operating System Concepts Chapter 6: CPU Scheduling Basic Concepts Scheduling Criteria Scheduling Algorithms.
Copyright © 2006 by The McGraw-Hill Companies, Inc. All rights reserved. McGraw-Hill Technology Education Lecture 5 Operating Systems.
CSC 360- Instructor: K. Wu CPU Scheduling. CSC 360- Instructor: K. Wu Agenda 1.What is CPU scheduling? 2.CPU burst distribution 3.CPU scheduler and dispatcher.
CPU Scheduling Introduction to Operating Systems: Module 13.
OPERATING SYSTEMS CPU SCHEDULING.  Introduction to CPU scheduling Introduction to CPU scheduling  Dispatcher Dispatcher  Terms used in CPU scheduling.
Implementing Processes and Process Management Brian Bershad.
Scheduling Strategies Operating Systems Spring 2004 Class #10.
ICS Principles of Operating Systems Lecture 5 - CPU Scheduling Prof. Nalini Venkatasubramanian
CPU Scheduling CSCI 444/544 Operating Systems Fall 2008.
Silberschatz and Galvin  Operating System Concepts Module 5: CPU Scheduling Basic Concepts Scheduling Criteria Scheduling Algorithms Multiple-Processor.
Development of the distributed monitoring system for the NICA cluster Ivan Slepov (LHEP, JINR) Mathematical Modeling and Computational Physics Dubna, Russia,
ROOT-CORE Team 1 PROOF xrootd Fons Rademakers Maarten Ballantjin Marek Biskup Derek Feichtinger (ARDA) Gerri Ganis Guenter Kickinger Andreas Peters (ARDA)
Matchmaking: A New MapReduce Scheduling Technique
Silberschatz, Galvin and Gagne ©2009 Operating System Concepts – 8 th Edition, Chapter 3: Process-Concept.
Super Scaling PROOF to very large clusters Maarten Ballintijn, Kris Gulbrandsen, Gunther Roland / MIT Rene Brun, Fons Rademakers / CERN Philippe Canal.
Operating Systems (CS 340 D) Dr. Abeer Mahmoud Princess Nora University Faculty of Computer & Information Systems Computer science Department.
Processor Scheduling Hank Levy. 22/4/2016 Goals for Multiprogramming In a multiprogramming system, we try to increase utilization and thruput by overlapping.
PROOF tests at BNL Sergey Panitkin, Robert Petkus, Ofer Rind BNL May 28, 2008 Ann Arbor, MI.
Introduction Contain two or more CPU share common memory and peripherals. Provide greater system throughput. Multiple processor executing simultaneous.
Latest Improvements in the PROOF system Bleeding Edge Physics with Bleeding Edge Computing Fons Rademakers, Gerri Ganis, Jan Iwaszkiewicz CERN.
Latest Improvements in the PROOF system Bleeding Edge Physics with Bleeding Edge Computing Fons Rademakers, Gerri Ganis, Jan Iwaszkiewicz CERN.
Alien and GSI Marian Ivanov. Outlook GSI experience Alien experience Proposals for further improvement.
CPU Scheduling Operating Systems CS 550. Last Time Deadlock Detection and Recovery Methods to handle deadlock – Ignore it! – Detect and Recover – Avoidance.
Name : Mamatha J M Seminar guide: Mr. Kemparaju. GRID COMPUTING.
Basic Concepts Maximum CPU utilization obtained with multiprogramming
1 Chapter 5: CPU Scheduling. 2 Basic Concepts Scheduling Criteria Scheduling Algorithms.
Review of PARK Reflectometry Group 10/31/2007. Outline Goal Hardware target Software infrastructure PARK organization Use cases Park Components. GUI /
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
TensorFlow– A system for large-scale machine learning
OpenPBS – Distributed Workload Management System
Copyright ©: Nahrstedt, Angrave, Abdelzaher
Operating Systems (CS 340 D)
Lecture Topics: 11/1 Processes Process Management
QoS in the Tier1 batch system(LSF)
Operating Systems (CS 340 D)
THINGS YOU SHOULD KNOW ABOUT JOB SCHEDULING. One such automation tool to run the essence of any organization or enterprise is Job scheduling. It is the.
Lecture 23: Process Scheduling for Interactive Systems
CS 143A - Principles of Operating Systems
Module 5: CPU Scheduling
Support for ”interactive batch”
3: CPU Scheduling Basic Concepts Scheduling Criteria
Process scheduling Chapter 5.
CPU SCHEDULING.
Lecture Topics: 11/1 General Operating System Concepts Processes
Operating System Concepts
Processes Hank Levy 1.
Processor Scheduling Hank Levy 1.
Concurrency and Threading: CPU Scheduling
Module 5: CPU Scheduling
Processes Hank Levy 1.
Module 5: CPU Scheduling
Presentation transcript:

Testing the dynamic per-query scheduling (with a FIFO queue) Jan Iwaszkiewicz

Per-Job vs. Per-Session Per-Session scheduling –Default in PROOF Per-Job –Start a new session only on the master –Call GetWorkers just when the user calls TProof::Process. –Possibility of keeping the inactive workers after the query (avoid startup times) while other jobs can use the CPUs.

Load-based scheduling Works per-session or per-query Assigns the workers based on: – current load of the cluster – relative priority of the user (static or dynamic) Parameters –Fraction (of free CPUs) –Optimal no. workers per CPU core Formula: n = (#free slots) * fraction * p –p: weighted priority of the user/group –#free: no. free CPU slots

Queuing A global queue in the scheduler To be used in case of heavy load and to optimize the average service time Implementation uses the PROOF asynchronous mode. First version with FIFO algorithm in SVN! Enable by: schedparm queue:fifo

Sending a query in the dynamic load-based mode Call GetWorkers If the list is empty swich to the asychronous mode and wait for a kPROOF_RESUME message. Start workers and process the query

Tests: input Workload file format: –Time –Dataset –No. Events –Selector –Username Generating the workload –e.g. Downey 1997

Tests: running Script to start a query and save result Script to execute the workload –Start jobs at appropriate times

Tests: Analysis Read results from DB or ROOT files Calculate cost function of the schedule –Convex in respect to service time –Proportional to DS size? –Proportional to user priority