CMS Week, June 7-11, 20041 CMS Production in Wisconsin Status of recent developments. Dan Bradley Sridhara Dasu Vivek Puttabuddhi Wesley Smith The Condor.

Slides:

Advertisements

Similar presentations

Andrew McNab - Manchester HEP - 17 September 2002 Putting Existing Farms on the Testbed Manchester DZero/Atlas and BaBar farms are available via the Testbed.

Advertisements

GXP in nutshell You can send jobs (Unix shell command line) to many machines, very fast Very small prerequisites –Each node has python (ver or later)

Condor use in Department of Computing, Imperial College Stephen M c Gough, David McBride London e-Science Centre.

Current methods for negotiating firewalls for the Condor ® system Bruce Beckles (University of Cambridge Computing Service) Se-Chang Son (University of.

Dan Bradley Computer Sciences Department University of Wisconsin-Madison Schedd On The Side.

CERN LCG Overview & Scaling challenges David Smith For LCG Deployment Group CERN HEPiX 2003, Vancouver.

Overview of Wisconsin Campus Grid Dan Bradley Center for High-Throughput Computing.

Condor-G: A Computation Management Agent for Multi-Institutional Grids James Frey, Todd Tannenbaum, Miron Livny, Ian Foster, Steven Tuecke Reporter: Fu-Jiun.

Condor and GridShell How to Execute 1 Million Jobs on the Teragrid Jeffrey P. Gardner - PSC Edward Walker - TACC Miron Livney - U. Wisconsin Todd Tannenbaum.

WP 1 Grid Workload Management Massimo Sgaravatto INFN Padova.

GRID workload management system and CMS fall production Massimo Sgaravatto INFN Padova.

Workload Management Workpackage Massimo Sgaravatto INFN Padova.

GRID Workload Management System Massimo Sgaravatto INFN Padova.

Workload Management Massimo Sgaravatto INFN Padova.

First steps implementing a High Throughput workload management system Massimo Sgaravatto INFN Padova

Condor Overview Bill Hoagland. Condor Workload management system for compute-intensive jobs Harnesses collection of dedicated or non-dedicated hardware.

Evaluation of the Globus GRAM Service Massimo Sgaravatto INFN Padova.

CVMFS: Software Access Anywhere Dan Bradley Any data, Any time, Anywhere Project.

Jaeyoung Yoon Computer Sciences Department University of Wisconsin-Madison Virtual Machines in Condor.

The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.

Zach Miller Computer Sciences Department University of Wisconsin-Madison What’s New in Condor.

SCD FIFE Workshop - GlideinWMS Overview GlideinWMS Overview FIFE Workshop (June 04, 2013) - Parag Mhashilkar Why GlideinWMS? GlideinWMS Architecture Summary.

Volunteer Computing and Hubs David P. Anderson Space Sciences Lab University of California, Berkeley HUBbub September 26, 2013.

S. Dasu, CHEP04, Interlacken, Switzerland1 Use of Condor and GLOW for CMS Simulation Production What are Condor & GLOW? What is special about Condor &

Vladimir Litvin, Harvey Newman Caltech CMS Scott Koranda, Bruce Loftis, John Towns NCSA Miron Livny, Peter Couvares, Todd Tannenbaum, Jamie Frey Wisconsin.

Todd Tannenbaum Computer Sciences Department University of Wisconsin-Madison What’s New in Condor.

Open Science Grid Software Stack, Virtual Data Toolkit and Interoperability Activities D. Olson, LBNL for the OSG International.

1 port BOSS on Wenjing Wu (IHEP-CC)

The Glidein Service Gideon Juve What are glideins? A technique for creating temporary, user- controlled Condor pools using resources from.

Ashok Agarwal 1 BaBar MC Production on the Canadian Grid using a Web Services Approach Ashok Agarwal, Ron Desmarais, Ian Gable, Sergey Popov, Sydney Schaffer,

Workload Management WP Status and next steps Massimo Sgaravatto INFN Padova.

STAR scheduling future directions Gabriele Carcassi 9 September 2002.

1 Evolution of OSG to support virtualization and multi-core applications (Perspective of a Condor Guy) Dan Bradley University of Wisconsin Workshop on.

Wenjing Wu Andrej Filipčič David Cameron Eric Lancon Claire Adam Bourdarios & others.

Part 6: (Local) Condor A: What is Condor? B: Using (Local) Condor C: Laboratory: Condor.

1 The Roadmap to New Releases Todd Tannenbaum Department of Computer Sciences University of Wisconsin-Madison

Use of Condor on the Open Science Grid Chris Green, OSG User Group / FNAL Condor Week, April

ETICS All Hands meeting Bologna, October 23-25, 2006 NMI and Condor: Status + Future Plans Andy PAVLO Peter COUVARES Becky GIETZEL.

Condor: High-throughput Computing From Clusters to Grid Computing P. Kacsuk – M. Livny MTA SYTAKI – Univ. of Wisconsin-Madison

WLCG Service Report ~~~ WLCG Management Board, 1 st September

NGS Innovation Forum, Manchester4 th November 2008 Condor and the NGS John Kewley NGS Support Centre Manager.

1 The Roadmap to New Releases Todd Tannenbaum Department of Computer Sciences University of Wisconsin-Madison

Todd Tannenbaum Computer Sciences Department University of Wisconsin-Madison Condor RoadMap.

The Roadmap to New Releases Derek Wright Computer Sciences Department University of Wisconsin-Madison

TeraGrid Advanced Scheduling Tools Warren Smith Texas Advanced Computing Center wsmith at tacc.utexas.edu.

July 11-15, 2005Lecture3: Grid Job Management1 Grid Compute Resources and Job Management.

Review of Condor,SGE,LSF,PBS

Dan Bradley University of Wisconsin-Madison Condor and DISUN Teams Condor Administrator’s How-to.

Proposal for a IS schema Massimo Sgaravatto INFN Padova.

GLIDEINWMS - PARAG MHASHILKAR Department Meeting, August 07, 2013.

Peter Couvares Associate Researcher, Condor Team Computer Sciences Department University of Wisconsin-Madison

GRID activities in Wuppertal D0RACE Workshop Fermilab 02/14/2002 Christian Schmitt Wuppertal University Taking advantage of GRID software now.

John Kewley e-Science Centre All Hands Meeting st September, Nottingham GROWL: A Lightweight Grid Services Toolkit and Applications John Kewley.

Jaime Frey Computer Sciences Department University of Wisconsin-Madison What’s New in Condor-G.

Dan Bradley Condor Project CS and Physics Departments University of Wisconsin-Madison CCB The Condor Connection Broker.

Douglas Thain, John Bent Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Miron Livny Computer Sciences Department, UW-Madison Gathering at the Well: Creating.

EDG - WP1 (Grid Work Scheduling) Status and plans Massimo Sgaravatto INFN Padova.

Status of Globus activities Massimo Sgaravatto INFN Padova for the INFN Globus group

Grid Workload Management (WP 1) Massimo Sgaravatto INFN Padova.

Jaime Frey Computer Sciences Department University of Wisconsin-Madison Condor and Virtual Machines.

CVMFS: Software Access Anywhere Dan Bradley Any data, Any time, Anywhere Project.

Andrew McNab - Globus Distribution for Testbed 1 Status of the Globus Distribution for Testbed 1 Andrew McNab, University of Manchester

Five todos when moving an application to distributed HTC.

Matt Lemons Nate Mayotte

Condor Glidein: Condor Daemons On-The-Fly

Basic Grid Projects – Condor (Part I)

Grid Laboratory Of Wisconsin (GLOW)

Condor-G Making Condor Grid Enabled

GLOW A Campus Grid within OSG

Condor-G: An Update.

Presentation transcript:

CMS Week, June 7-11, CMS Production in Wisconsin Status of recent developments. Dan Bradley Sridhara Dasu Vivek Puttabuddhi Wesley Smith The Condor Team +

CMS Week, June 7-11, Investigating User Mode Linux Designed a UML job wrapper for CMS Provides “blessed” linux environment on demand. Works transparently with almost any software stack. But there’s no such thing as a free linux 15-20% performance drop for Oscar Host “skas” kernel patch only gains ~3% I/O intensive jobs will be even worse without skas UML only ported to Linux x86 (no Windows) 80 MB tarball + install time, but this is easily cached

CMS Week, June 7-11, Condor Glidein What is it? Provides Condor job management under other batch systems (e.g. on the grid) Matchmaking, checkpointing, job migration, etc. Some improvements & experiments. MDS schema requirements now optional More automation in setup and installation With Vladimir Litvin and Edward Walker testing GridShell as Glidein submission agent. No such thing as a free Condor? Difficult to work across a firewall.

CMS Week, June 7-11, Better Condor Preemption Limitations of claim-based preemption Very flexible policy expressions, but… Not sensitive to job boundaries. Policy: “machine X should never kill job Y within bound Z” interferes with preemption of claims. Fair-sharing problems noticed on Grid Added claim “retirement” Claim retires on job boundary or limit. Uniform negotiation of retirement preferences and requirements. Works with any preemption policy.

CMS Week, June 7-11, Orphaned Jobs Trouble with orphans on Grid3 Jobs with no Globus jobmanager Affecting all batch systems, but most annoying in Condor because it doesn’t give up unless told to do so. Jobs with missing GASS files try forever. Globus jobmanager for Condor patched Runs job with time-to-live, periodically renewed Automatically halts failing or queued orphans. Jobmanager may still reattach within longer- term garbage collection cycle.

CMS Week, June 7-11, CMS on Grid Laboratories of Wisconsin 1st round of GLOW now commissioned GHz zeons 1.2 TB/rack cache 2 more rounds in the pipeline…