BQS Update Architecture Scheduling Job Types New Users Needs More users & machines, Scalability issues Needs for more sophisticated monitoring and control.

Slides:

Advertisements

Similar presentations

This course is designed for system managers/administrators to better understand the SAAZ Desktop and Server Management components Students will learn.

Advertisements

Nadia LAJILI IN2P3 Computing Center Testbed Status IN2P3 Computing Center Testbed Status Lyon, February 2003.

Building Portals to access Grid Middleware National Technical University of Athens Konstantinos Dolkas, On behalf of Andreas Menychtas.

Generic MPI Job Submission by the P-GRADE Grid Portal Zoltán Farkas MTA SZTAKI.

SUS Feature Pack for SMS Michel Jouvin LAL / IN2P3

CERN LCG Overview & Scaling challenges David Smith For LCG Deployment Group CERN HEPiX 2003, Vancouver.

Resource Management with YARN: YARN Past, Present and Future

CHEP 2012 – New York City 1.  LHC Delivers bunch crossing at 40MHz  LHCb reduces the rate with a two level trigger system: ◦ First Level (L0) – Hardware.

DataGrid is a project funded by the European Union 22 September 2003 – n° 1 EDG WP4 Fabric Management: Fabric Monitoring and Fault Tolerance

Workload Management Massimo Sgaravatto INFN Padova.

Default: Zoom 65% then screenshot. DB XStudio tests Launcher DB XStudio XAgent tests Backup DB tests Launcher.

Enabling Grids for E-sciencE Medical image processing web portal : Requirements analysis. An almost end user point of view … H. Benoit-Cattin,

CC - IN2P3 Site Report Hepix Fall meeting 2009 – Berkeley

7/2/2003Supervision & Monitoring section1 Supervision & Monitoring Organization and work plan Olof Bärring.

Scalable Systems Software Center Resource Management and Accounting Working Group Face-to-Face Meeting June 13-14, 2002.

1 Dynamic Application Installation (Case of CMS on OSG) Introduction CMS Software Installation Overview Software Installation Issues Validation Considerations.

CHEP 2000 Smart Resource Management Software in High Energy Physics Wolfgang Gentzsch and Lothar Lippert Gridware GmbH & Inc. Padua, 9 February 2000.

OSG Middleware Roadmap Rob Gardner University of Chicago OSG / EGEE Operations Workshop CERN June 19-20, 2006.

Job Submission Condor, Globus, Java CoG Kit Young Suk Moon.

February 20, AgentCities - Agents and Grids Prof Mark Baker ACET, University of Reading Tel:

Grid Computing I CONDOR.

A.Guarise – F.Rosso 1 Enabling Grids for E-sciencE INFSO-RI Comprehensive Accounting Views on large computing farms. Andrea Guarise & Felice Rosso.

IMDGs An essential part of your architecture. About me

Aug 13 th 2003Scheduler Tutorial1 STAR Scheduler – A tutorial Lee Barnby – Kent State University Introduction What is the scheduler and what are the advantages?

Overview of day-to-day operations Suzanne Poulat.

11/30/2007 Overview of operations at CC-IN2P3 Exploitation team Reported by Philippe Olivero.

Database Administration COMSATS INSTITUTE OF INFORMATION TECHNOLOGY, VEHARI.

Batch Scheduling at LeSC with Sun Grid Engine David McBride Systems Programmer London e-Science Centre Department of Computing, Imperial College.

The huge amount of resources available in the Grids, and the necessity to have the most up-to-date experimental software deployed in all the sites within.

Stuart Wakefield Imperial College London Evolution of BOSS, a tool for job submission and tracking W. Bacchi, G. Codispoti, C. Grandi, INFN Bologna D.

MAGDA Roger Jones UCL 16 th December RWL Jones, Lancaster University MAGDA  Main authors: Wensheng Deng, Torre Wenaus Wensheng DengTorre WenausWensheng.

Condor: High-throughput Computing From Clusters to Grid Computing P. Kacsuk – M. Livny MTA SYTAKI – Univ. of Wisconsin-Madison

PROOF Cluster Management in ALICE Jan Fiete Grosse-Oetringhaus, CERN PH/ALICE CAF / PROOF Workshop,

Manchester HEP Desktop/ Laptop 30 Desktop running RH Laptop Windows XP & RH OS X Home server AFS using openafs 3 DB servers Kerberos 4 we will move.

A Brief Documentation.  Provides basic information about connection, server, and client.

Architecture and ATLAS Western Tier 2 Wei Yang ATLAS Western Tier 2 User Forum meeting SLAC April

Derek Wright Computer Sciences Department University of Wisconsin-Madison MPI Scheduling in Condor: An.

SAN DIEGO SUPERCOMPUTER CENTER Inca TeraGrid Status Kate Ericson November 2, 2006.

JBQS - Bernard CHAMBON - HEPIX, Nov JBQS presentation IN2P3 Computer Center Campus de la DOUA 27, Boulevard du 11 Novembre.

CASTOR evolution Presentation to HEPiX 2003, Vancouver 20/10/2003 Jean-Damien Durand, CERN-IT.

Derek Wright Computer Sciences Department University of Wisconsin-Madison Condor and MPI Paradyn/Condor.

Cluster Configuration Update Including LSF Status Thorsten Kleinwort for CERN IT/PDP-IS HEPiX I/2001 LAL Orsay Tuesday, December 08, 2015.

D. Duellmann - IT/DB LCG - POOL Project1 The LCG Pool Project and ROOT I/O Dirk Duellmann What is Pool? Component Breakdown Status and Plans.

RHIC/US ATLAS Tier 1 Computing Facility Site Report Christopher Hollowell Physics Department Brookhaven National Laboratory HEPiX Upton,

Magda Distributed Data Manager Prototype Torre Wenaus BNL September 2001.

Master Cluster Manager User Interface (API Level) User Interface (API Level) Query Translator Avro NTA Query Engine NTA Query Engine Job Scheduler Avro.

Gennaro Tortone, Sergio Fantinel – Bologna, LCG-EDT Monitoring Service DataTAG WP4 Monitoring Group DataTAG WP4 meeting Bologna –

EGEE is a project funded by the European Union under contract IST Experiment Software Installation toolkit on LCG-2

VOX Project Status T. Levshina. 5/7/2003LCG SEC meetings2 Goals, team and collaborators Purpose: To facilitate the remote participation of US based physicists.

CMS Experience with the Common Analysis Framework I. Fisk & M. Girone Experience in CMS with the Common Analysis Framework Ian Fisk & Maria Girone 1.

Operation team at Ccin2p3 Suzanne Poulat –

OSG Status and Rob Gardner University of Chicago US ATLAS Tier2 Meeting Harvard University, August 17-18, 2006.

A System for Monitoring and Management of Computational Grids Warren Smith Computer Sciences Corporation NASA Ames Research Center.

VO Box discussion ATLAS NIKHEF January, 2006 Miguel Branco -

DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.

Preservation Data Services Persistent Archive Research Group Reagan W. Moore October 1, 2003.

Alessandro De Salvo Mayuko Kataoka, Arturo Sanchez Pineda,Yuri Smirnov CHEP 2015 The ATLAS Software Installation System v2 Alessandro De Salvo Mayuko Kataoka,

Magda Distributed Data Manager Torre Wenaus BNL October 2001.

Vendredi 27 avril 2007 Management of ATLAS CC-IN2P3 Specificities, issues and advice.

Jean-Philippe Baud, IT-GD, CERN November 2007

Real Time Fake Analysis at PIC

How to connect your DG to EDGeS? Zoltán Farkas, MTA SZTAKI

BOSS: the CMS interface for job summission, monitoring and bookkeeping

Sergio Fantinel, INFN LNL/PD

Grid Computing.

PES Lessons learned from large scale LSF scalability tests

AliEn central services (structure and operation)

University of Technology

OPS235 PACKAGE MANAGEMENT

Presentation transcript:

BQS Update Architecture Scheduling Job Types New Users Needs More users & machines, Scalability issues Needs for more sophisticated monitoring and control over the system HEPIX Edinburgh 26/5/04

Architecture Update Client Worker MySQL DB BQS Scheduler Worker DB Agent results spawn report submit query DB Agent HEPIX Edinburgh 26/5/04

Scheduler Resources Quasi Interactive Jobs Scheduling Update HEPIX Edinburgh 26/5/04

More Control for Operation and Administration: Weight of Past Resource Usage And Group Objectives Max Job Duration Small Jobs Bias Scheduler HEPIX Edinburgh 26/5/04

Beyond Traditional Resources E.G. Disk, Time, Memory Logical Resources Name Max Available Restricted Flag Admin Defined Resources E.G. HPSS Logical Resources HEPIX Edinburgh 26/5/04

Created & managed by Users: Decide of the Name: u_XXX Receive Privilege bqs.u_XXXadmin Set Max Available and Restricted Flag Grant/deny bqs.u_XXXusage privilege Logical U_Resource HEPIX Edinburgh 26/5/04

A General Service, APIs, Commands To: Grant, Deny, Check & List Privileges Given to Users, Groups and Machines EG in BQS applid: bqs.admin, bqs.oper, bqs.spawn_forbidden Privilege Management HEPIX Edinburgh 26/5/04

Parallel Jobs Arborescent Jobs GRID Jobs New Job Types HEPIX Edinburgh 26/5/04

2 new submit options: proc, ptype proc: Number of WorkPoints ptype: PVM, MPICH, LAM-MPI Parallel Jobs HEPIX Edinburgh 26/5/04

Parallel Jobs Client Worker MySQL DB BQS Master Worker BQS DB Agent results spawn parallel job report submit … query spawn task DB DB Agent global report HEPIX Edinburgh 26/5/04

SNOVAE: many related small tasks need short global response time Schedule and spawn as one Job to reduce BQS latency Runs on a number of WorkPoints User must describe tasks dependencies Arborescent Job HEPIX Edinburgh 26/5/04

Real and Generic Accounts AFS Tokens and Certificates Specific RH LCG Soft Full Production Farm (Currently a Specific lcg Logical Test Farm for Validation) GRID Jobs HEPIX Edinburgh 26/5/04

BIO: Quasi Interactives Jobs Installation and documentation for LCG Other Projects HEPIX Edinburgh 26/5/04