NorduGrid Grid Manager developed at NorduGrid project.

Slides:

Advertisements

Similar presentations

Generic MPI Job Submission by the P-GRADE Grid Portal Zoltán Farkas MTA SZTAKI.

Advertisements

EGC 2005, CrossGrid technical achievements, Amsterdam, Feb. 16th, 2005 WP2-3 New Generation Environment for Grid Interactive MPI Applications M igrating.

The Anatomy of the Grid: An Integrated View of Grid Architecture Carl Kesselman USC/Information Sciences Institute Ian Foster, Steve Tuecke Argonne National.

FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.

Grid Resource Allocation Management (GRAM) GRAM provides the user to access the grid in order to run, terminate and monitor jobs remotely. The job request.

CERN LCG Overview & Scaling challenges David Smith For LCG Deployment Group CERN HEPiX 2003, Vancouver.

Plateforme de Calcul pour les Sciences du Vivant SRB & gLite V. Breton.

Lightweight Preservation Environment Gary Jackson.

A Computation Management Agent for Multi-Institutional Grids

WP 1 Grid Workload Management Massimo Sgaravatto INFN Padova.

The DataGrid Project NIKHEF, Wetenschappelijke Jaarvergadering, 19 December 2002

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.

USING THE GLOBUS TOOLKIT This summary by: Asad Samar / CALTECH/CMS Ben Segal / CERN-IT FULL INFO AT:

The EDG Testbed Introduction and Setup The European DataGrid Project Team

GRID workload management system and CMS fall production Massimo Sgaravatto INFN Padova.

Towards the Design and Implementation of the DAME prototype: OGSA Compliant Grid Services on the White Rose Grid Sarfraz A Nadeem University of Leeds.

Workload Management Workpackage Massimo Sgaravatto INFN Padova.

GRID Workload Management System Massimo Sgaravatto INFN Padova.

Globus activities within INFN Massimo Sgaravatto INFN Padova for the INFN Globus group

Workload Management Massimo Sgaravatto INFN Padova.

First steps implementing a High Throughput workload management system Massimo Sgaravatto INFN Padova

Grids and Globus at BNL Presented by John Scott Leita.

Globus Computing Infrustructure Software Globus Toolkit 11-2.

Evaluation of the Globus GRAM Service Massimo Sgaravatto INFN Padova.

The NorduGrid project: Using Globus toolkit for building Grid infrastructure presented by Aleksandr Konstantinov Mattias Ellert Aleksandr Konstantinov.

The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.

OSG End User Tools Overview OSG Grid school – March 19, 2009 Marco Mambelli - University of Chicago A brief summary about the system.

The EDG Testbed Deployment Details The European DataGrid Project

BaBar WEB job submission with Globus authentication and AFS access T. Adye, R. Barlow, A. Forti, A. McNab, S. Salih, D. H. Smith on behalf of the BaBar.

High Performance Louisiana State University - LONI HPC Enablement Workshop – LaTech University,

Workload Management WP Status and next steps Massimo Sgaravatto INFN Padova.

Grid Computing - AAU 14/ Grid Computing Josva Kleist Danish Center for Grid Computing

Grid Resource Allocation and Management (GRAM) Execution management Execution management –Deployment, scheduling and monitoring Community Scheduler Framework.

NorduGrid Architecture and tools CHEP2003 – UCSD Anders Wäänänen

Computational grids and grids projects DSS,

QCDGrid Progress James Perry, Andrew Jackson, Stephen Booth, Lorna Smith EPCC, The University Of Edinburgh.

Grid Workload Management & Condor Massimo Sgaravatto INFN Padova.

1 st December 2003 JIM for CDF 1 JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow.

DataGrid WP1 Massimo Sgaravatto INFN Padova. WP1 (Grid Workload Management) Objective of the first DataGrid workpackage is (according to the project "Technical.

Grid Workload Management Massimo Sgaravatto INFN Padova.

- Distributed Analysis (07may02 - USA Grid SW BNL) Distributed Processing Craig E. Tull HCG/NERSC/LBNL (US) ATLAS Grid Software.

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.

International Workshop on HEP Data Grid Nov 9, 2002, KNU Data Storage, Network, Handling, and Clustering in CDF Korea group Intae Yu*, Junghyun Kim, Ilsung.

Andrew McNabETF Firewall Meeting, NeSC, 5 Nov 2002Slide 1 Firewall issues for Globus 2 and EDG Andrew McNab High Energy Physics University of Manchester.

Author - Title- Date - n° 1 Partner Logo EU DataGrid, Work Package 5 The Storage Element.

Author - Title- Date - n° 1 Partner Logo WP5 Summary Paris John Gordon WP5 6th March 2002.

Quick Introduction to NorduGrid Oxana Smirnova 4 th Nordic LHC Workshop November 23, 2001, Stockholm.

Production Tools in ATLAS RWL Jones GridPP EB 24 th June 2003.

First attempt for validating/testing Testbed 1 Globus and middleware services WP6 Meeting, December 2001 Flavia Donno, Marco Serra for IT and WPs.

June 24-25, 2008 Regional Grid Training, University of Belgrade, Serbia Introduction to gLite gLite Basic Services Antun Balaž SCL, Institute of Physics.

July 11-15, 2005Lecture3: Grid Job Management1 Grid Compute Resources and Job Management.

What is SAM-Grid? Job Handling Data Handling Monitoring and Information.

Part Five: Globus Job Management A: GRAM B: Globus Job Commands C: Laboratory: globusrun.

MTA SZTAKI Hungarian Academy of Sciences Introduction to Grid portals Gergely Sipos

Globus Toolkit Massimo Sgaravatto INFN Padova. Massimo Sgaravatto Introduction Grid Services: LHC regional centres need distributed computing Analyze.

Review of Condor,SGE,LSF,PBS

AliEn AliEn at OSC The ALICE distributed computing environment by Bjørn S. Nilsen The Ohio State University.

Grid Compute Resources and Job Management. 2 How do we access the grid ?  Command line with tools that you'll use  Specialised applications Ex: Write.

ATLAS Data Challenge on NorduGrid CHEP2003 – UCSD Anders Wäänänen

Workload Management System Jason Shih WLCG T2 Asia Workshop Dec 2, 2006: TIFR.

Grid Compute Resources and Job Management. 2 Grid middleware - “glues” all pieces together Offers services that couple users with remote resources through.

Jaime Frey Computer Sciences Department University of Wisconsin-Madison What’s New in Condor-G.

Grid Workload Management (WP 1) Massimo Sgaravatto INFN Padova.

The DataGrid Project NIKHEF, Wetenschappelijke Jaarvergadering, 19 December 2002

Grid Activities in CMS Asad Samar (Caltech) PPDG meeting, Argonne July 13-14, 2000.

Current Globus Developments Jennifer Schopf, ANL.

Parallel Computing Globus Toolkit – Grid Ayaka Ohira.

Peter Kacsuk – Sipos Gergely MTA SZTAKI

Globus Job Management. Globus Job Management Globus Job Management A: GRAM B: Globus Job Commands C: Laboratory: globusrun.

Presentation transcript:

NorduGrid Grid Manager developed at NorduGrid project

NorduGrid Grid Manager (essentials) The Target Provide job control and data handling functionality required by HEP applications. Limitation: Data is handled only at that beginning and end of the job. User must provide information about input and output data. Comparisons Globus gatekeeper Provides more functionality for controling job parameters and data handling EDG WP1 Moves part of Logging and Bookkeeping functionality from the centralized server. Hence gives better fault tolerance. Job is run in more restrictive environment (maybe not already??) No special requirements for working nodes of the cluster

NorduGrid Grid Manager (Globus is cornerstone) The Grid Manager is based on Globus Toolkit TM libraries and services. The following parts of Globus are used: GridFTP - fast and reliable data access for Grid GASS Copy interface - support for different data access protocols Replica Catalog - metadata storage GRAM - resource request (obsolete) RSL - expandable Resource Specification Language The advantages of using Globus: Widely accepted de-facto standard for Grid computing No need to develop low level services Security built in Continiously evolving (??)

NorduGrid Grid Manager (internals) LRMS (PBS) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) Grid Manager Globus Gatekeeper GRAM Jobmanager NG Globus GridFTP server NorduGrid GridFTP file access plugin job control plugin

NorduGrid Grid Manager (job submission) LRMS (PBS) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager

NorduGrid Grid Manager (job submission) LRMS (PBS) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager RSL

NorduGrid Grid Manager (job submission) LRMS (PBS) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager RSL

NorduGrid Grid Manager (job submission) LRMS (PBS) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager RSL Job directory Inputfiles prodscript atlsim.makefile atlas.kumac gen0016_1.root <- rc:///gen0016_1.root

NorduGrid Grid Manager (job preparation) LRMS (PBS) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager Job directory Inputfiles prodscript atlsim.makefile atlas.kumac gen0016_1.root <- rc:///gen0016_1.root prodscript atlsim.makefile atlas.kumac prodscript atlsim.makefile atlas.kumac

NorduGrid Grid Manager (job preparation) LRMS (PBS) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager Job directory Inputfiles prodscript atlsim.makefile atlas.kumac gen0016_1.root rc:///gen0016_1.root prodscript atlsim.makefile atlas.kumac prodscript atlsim.makefile atlas.kumac gen0016_1.root Downloader rc:///gen0016_1.root

NorduGrid Grid Manager (job execution) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager Job directory executable = prodscript runtimeEnvironment = ATLAS arguments = prodscript atlsim.makefile atlas.kumac gen0016_1.root LRMS (PBS) prodscript

NorduGrid Grid Manager (job post-processing) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager Job directory prodscript atlsim.makefile atlas.kumac gen0016_1.root stdout + stderr atlas.001.his atlas.001.zebra LRMS (PBS) prodscript stdout + stderr

NorduGrid Grid Manager (job post-processing) Job session directories Gatekeeper LRMS (PBS) Computing nodes LRMS (PBS) NorduGrid GridFTP file access plugin job control plugin Grid Manager Job directory stdout + stderr atlas.001.his atlas.001.zebra LRMS (PBS) stdout + stderr atlas.001.zebra Outputfiles stdout + stderr atlas.001.his atlas.001.zebra rc:///results/atlas.001.zebra Uploader

NorduGrid Grid Manager (Functionality) Stage in input data and executables. Possible sources: Job submission machine GridFTP (preffered), FTP, HTTP or HTTPS servers Files registered in Globus Replica Catalog. Secure authentication. Destination is chosen automatically or can be forced. Stage out output data. Possible destinations: Keep on Grid cluster till user downloads GridFTP, FTP, HTTP or HTTPS servers Files can be registered in Globus Replica Catalog. Destination and protocol are obtained from Location information notification of job status changes Support for software runtime environment configuration Interface to the information about job (used by NorduGrid Information System)

NorduGrid Grid Manager (Things To Do) Better support for data handling: On-site data caching (preferably partial caching) On-site data sharing Better handling of failures More firewall friendly Better support for Local Resource Management System More flexible configuration suitable for demanding site managers Better support for running under non-root account Improve security More strict permissions Possible chroot environment