Job Management with DATA

Slides:



Advertisements
Similar presentations
EGEE is a project funded by the European Union under contract IST EGEE Tutorial Turin, January Hands on Job Services.
Advertisements

INFSO-RI Enabling Grids for E-sciencE Workload Management System and Job Description Language.
SEE-GRID-SCI User Interface (UI) Installation and Configuration Branimir Ackovic Institute of Physics Serbia The SEE-GRID-SCI.
EGEE is funded by the European Union under contract IST Elena Slabospitskaya IHEP NA3 manager for Russia An inroduction to services provided.
Riccardo Bruno, INFN.CT Sevilla, 10-14/09/2007 GENIUS Exercises.
DIRAC Tutorial By R. Graciani Lisbon, Nov 8 th 2012.
E-infrastructure shared between Europe and Latin America 12th EELA Tutorial for Users and System Administrators Architecture of the gLite.
INFSO-RI Enabling Grids for E-sciencE EGEE Middleware The Resource Broker EGEE project members.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Job Submission Fokke Dijkstra RuG/SARA Grid.
Job Submission The European DataGrid Project Team
Enabling Grids for E-sciencE gLite training at Sinaia '06 Victor Penso Kilian Schwarz GSI Darmstadt Germany.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Luciano Díaz ICN-UNAM Based on Domenico.
Enabling Grids for E-sciencE EGEE-II INFSO-RI BG induction to GRID Computing and EGEE project – Sofia, 2006 Practical: Porting applications.
K. Harrison CERN, 20th April 2004 AJDL interface and LCG submission - Overview of AJDL - Using AJDL from Python - LCG submission.
The gLite API – PART I Giuseppe LA ROCCA INFN Catania ACGRID-II School 2-14 November 2009 Kuala Lumpur - Malaysia.
INFSO-RI Enabling Grids for E-sciencE GILDA Praticals GILDA Tutors INFN Catania ICTP/INFM-Democritos Workshop on Porting Scientific.
:: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: GridKA School 2009 MPI on Grids 1 MPI On Grids September 3 rd, GridKA School 2009.
Enabling Grids for E-sciencE Workload Management System on gLite middleware Matthieu Reichstadt CNRS/IN2P3 ACGRID School, Hanoi (Vietnam)
Nadia LAJILI User Interface User Interface 4 Février 2002.
E-science grid facility for Europe and Latin America Marcelo Risk y Juan Francisco García Eijó Laboratorio de Sistemas Complejos Departamento.
INFSO-RI Enabling Grids for E-sciencE Workload Management System Mike Mineter
Workflow Template Generation Haw to. Steps Make proper set of WF examples on Portal Save it to a local directory Decompress it Make a Template Directory.
1 Esther Montes Prado CIEMAT 10th EELA Tutorial Madrid, Hands-on on WMS (Review and Summary)
- Distributed Analysis (07may02 - USA Grid SW BNL) Distributed Processing Craig E. Tull HCG/NERSC/LBNL (US) ATLAS Grid Software.
Group 1 : Grid Computing Laboratory of Information Technology Supervisors: Alexander Ujhinsky Nikolay Kutovskiy.
EGEE-III INFSO-RI Enabling Grids for E-sciencE Feb. 06, Introduction to High Performance and Grid Computing Faculty of Sciences,
Jan 31, 2006 SEE-GRID Nis Training Session Hands-on V: Standard Grid Usage Dušan Vudragović SCL and ATLAS group Institute of Physics, Belgrade.
Architecture of the gLite WMS (Workload Management System) Hands-on Paola Celio Universita’ Roma TRE INFN Roma TRE Sevilla Septembre 2007.
Job Management DIRAC Project. Overview  DIRAC JDL  DIRAC Commands  Tutorial Exercises  What do you have learned? KEK 10/2012DIRAC Tutorial.
E-science grid facility for Europe and Latin America MAVs-Study Biologically Inspired, Super Maneuverable, Flapping Wing Micro-Air-Vehicles.
FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.
INFSO-RI Enabling Grids for E-sciencE Αthanasia Asiki Computing Systems Laboratory, National Technical.
Enabling Grids for E-sciencE Workload Management System on gLite middleware - commands Matthieu Reichstadt CNRS/IN2P3 ACGRID School, Hanoi.
High-Performance Computing Lab Overview: Job Submission in EDG & Globus November 2002 Wei Xing.
INFSO-RI Enabling Grids for E-sciencE Job Workflows with gLite Emidio Giorgio INFN NA4 Generic Applications Meeting 10 January 2006.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Workload management in gLite 3.x - MPI P. Nenkova, IPP-BAS, Sofia, Bulgaria Some of.
INFSO-RI Enabling Grids for E-sciencE Job Submission Tutorial (material from INFN Catania)
Workload Management System Jason Shih WLCG T2 Asia Workshop Dec 2, 2006: TIFR.
1 DIRAC Job submission A.Tsaregorodtsev, CPPM, Marseille LHCb-ATLAS GANGA Workshop, 21 April 2004.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Moisés Hernández Duarte UNAM FES Cuautitlán.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Command Line Grid Programming Spiros Spirou Greek Application Support Team NCSR “Demokritos”
INFSO-RI Enabling Grids for E-sciencE Job Description Language (JDL) Giuseppe La Rocca INFN First gLite tutorial on GILDA Catania,
INFSO-RI Enabling Grids for E-sciencE GILDA Praticals Giuseppe La Rocca INFN – Catania gLite Tutorial at the EGEE User Forum CERN.
E-infrastructure shared between Europe and Latin America FP6−2004−Infrastructures−6-SSA Special Jobs Valeria Ardizzone INFN - Catania.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks WMS tricks & tips – further scripting Giuseppe.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Practical using WMProxy advanced job submission.
Biomed tutorial 1 Enabling Grids for E-sciencE INFSO-RI EGEE is a project funded by the European Union under contract IST JDL Flavia.
June 24-25, 2008 Regional Grid Training, University of Belgrade, Serbia Hands-on: Compiling MPI codes with PGI Dušan Vudragović SCL,
User Interface UI TP: UI User Interface installation & configuration.
LCG2 Tutorial Viet Tran Institute of Informatics Slovakia.
Istituto Nazionale di Astrofisica Information Technology Unit INAF-SI Job with data management Giuliano Taffoni.
Job Management Beijing, 13-15/11/2013. Overview Beijing, /11/2013 DIRAC Tutorial2  DIRAC JDL  DIRAC Commands  Tutorial Exercises  What do you.
Workload Management Status DIRAC User Meeting Marseille, Oct 2012.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) Advanced Job Riccardo Rotondo
Introduction to Computing Element HsiKai Wang Academia Sinica Grid Computing Center, Taiwan.
FESR Consorzio COMETA - Progetto PI2S2 Jobs with Input/Output data Fabio Scibilia, INFN - Catania, Italy Tutorial per utenti e.
Enabling Grids for E-sciencE Work Load Management & Simple Job Submission Practical Shu-Ting Liao APROC, ASGC EGEE Tutorial.
EGI-InSPIRE RI EGI Hands On Training for AEGIS Users EGI-InSPIRE N G I A E G I S EGI Hands On Training for AEGIS Users Institute of Physics.
Information System testing for LCG-1
Workload Management System on gLite middleware
EGEE tutorial, Job Description Language - more control over your Job Assaf Gottlieb Tel-Aviv University EGEE is a project.
Gridifying the LHCb Monte Carlo simulation system
5. Job Submission Grid Computing.
The gLite Workload Management System
gLite Job Management Christos Theodosiou
Job Description Language
GENIUS Grid portal Hands on
Job Description Language (JDL)
Hands on Session: DAG Job Submission
Job Submission M. Jouvin (LAL-Orsay)
Presentation transcript:

Job Management with DATA Beijing 13-15/11/2013

Overview Sandboxes vs. Data JDLs options

Example Simple JDL JobName = "Simple_Job"; Executable = "/bin/ls"; * Arguments = "-ltr"; StdOutput = "StdOut"; * StdError = "StdErr"; * OutputSandbox = {"StdOut","StdErr"}; * * Mandatory attributes DIRAC Tutorial Beijing, 13-15/11/2013

Sandboxes vs. Data Sandbox files are stored in the DIRAC Workload Management Service and are only available for the time of life of the corresponding job. Volatile storage. Some grace period, typically 1 week for keeping sandboxes to allow their retrieval. Data are stored in the Grid Storage Permanent storage The size of the Sandboxes is limited to ~10MG per job Submitting a job with a large sandbox will result as an error Large output sandboxes are stored in the Grid Storage This is the user responsibility to clean up this storage DIRAC Tutorial Beijing, 13-15/11/2013

Example Input and Output Sandbox Simple.jdl Executable = "testJob.sh"; StdOutput = "StdOut"; StdError = "StdErr"; InputSandbox = {"testJob.sh"}; OutputSandbox = {"StdOut","StdErr"}; testJob.sh #!/bin/bash /bin/hostname /bin/date DIRAC Tutorial Beijing, 13-15/11/2013

Example Input and Output Data testJob.jdl Executable = "testJob.sh"; StdOutput = "StdOut"; StdError = "StdErr"; InputSandbox = {"testJob.sh”,”LFN:/vo.france- asia.org/user/v/vhamar/test.txt"}; OutputSandbox = {"StdOut","StdErr"}; OutputSE = ”CPPM-disk"; testJob.sh #!/bin/bash /bin/hostname cat test.txt DIRAC Tutorial Beijing, 13-15/11/2013

Example Input and Output Data DIRAC Tutorial Beijing, 13-15/11/2013

Wiki https://github.com/DIRACGrid/DIRAC/wiki/JobManag ement