Pablo Saiz CAF and Grid User Forum

Slides:

Advertisements

Similar presentations

DataTAG WP4 Meeting CNAF Jan 14, 2003 Interfacing AliEn and EDG 1/13 Stefano Bagnasco, INFN Torino Interfacing AliEn to EDG Stefano Bagnasco, INFN Torino.

Advertisements

WP 1 Members of Wp1: INFN Cesnet DATAMAT PPARC. WP 1 What does WP1 do? Broker Submission mechanism JDL/JCL and other UIs Logging computational economics.

WEB DESIGN TABLES, PAGE LAYOUT AND FORMS. Page Layout Page Layout is an important part of web design Why do you think your page layout is important?

Copyright © 2008 SAS Institute Inc. All rights reserved. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks.

Slide 1 of 10 Job Event Basics A Job Event is the name for the collection of components that comprise a scheduled job. On the iSeries a the available Job.

The LEGO Train Framework

SEE-GRID-SCI Hands-On Session: Workload Management System (WMS) Installation and Configuration Dusan Vudragovic Institute of Physics.

Batch Process Overview. Batch Processes 2 What Do You Mean By Batch Processing? Allows for Mass Entry of Data All Processing of the Data Happens Without.

Department of Computer Science 1 CSS 496 Business Process Re-engineering for BS(CS)

Asynchronous Web Services Approach Enrique de Andrés Saiz.

AliEn Tutorial MODEL th May, May 2009 Installation of the AliEn software AliEn and the GRID Authentication File Catalogue.

The ATLAS Production System. The Architecture ATLAS Production Database Eowyn Lexor Lexor-CondorG Oracle SQL queries Dulcinea NorduGrid Panda OSGLCG The.

© 2011 Delmar, Cengage Learning Chapter 9 Collecting Data with Forms.

ALICE Offline meeting 27 March 2008 Artem Harutyunyan

AliEn uses bbFTP for the file transfers. Every FTD runs a server, and all the others FTD can connect and authenticate to it using certificates. bbFTP implements.

Workload Management WP Status and next steps Massimo Sgaravatto INFN Padova.

CERN IT Department CH-1211 Geneva 23 Switzerland t The Experiment Dashboard ISGC th April 2008 Pablo Saiz, Julia Andreeva, Benjamin.

Copyright © 2007, Oracle. All rights reserved. Managing Concurrent Requests.

RISICO on the GRID architecture First implementation Mirko D'Andrea, Stefano Dal Pra.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES P. Saiz (IT-ES) AliEn job agents.

David Adams ATLAS AJDL: Analysis Job Description Language David Adams BNL December 15, 2003 PPDG Collaboration Meeting LBL.

INFSO-RI Enabling Grids for E-sciencE Workload Management System Mike Mineter

Software Sustainability Institute Online reconstruction (Manchego) Status report 09/02/12 Mike Jackson

Ganga A quick tutorial Asterios Katsifodimos Trainer, University of Cyprus Nicosia, Feb 16, 2009.

Version Control with SVN Images from TortoiseSVN documentation

ACCUPLACER Duplicate Student IDs Indiana Department Of Education

Overview of ALICE monitoring Catalin Cirstoiu, Pablo Saiz, Latchezar Betev 23/03/2007 System Analysis Working Group.

TORQUE Kerry Chang CCLS December 13, O UTLINE Torque How does it work? Architecture MADA Demo Results Problems Future Improvements.

Andrei Gheata, Mihaela Gheata, Andreas Morsch ALICE offline week, 5-9 July 2010.

Working with AliEn Kilian Schwarz ALICE Group Meeting April

4 AliEn Artem Harutyunyan Presented by: Predrag Buncic Tackling job.

CERN – Alice Offline – Thu, 20 Mar 2008 – Marco MEONI - 1 Status of Cosmic Reconstruction Offline weekly meeting.

Pipeline Basics Jared Crossley NRAO NRAO. What is a data pipeline?  One or more programs that perform a task with reduced user interaction.  May be.

AliEn AliEn at OSC The ALICE distributed computing environment by Bjørn S. Nilsen The Ohio State University.

January 2006Colby College ITS Setting Up Course Pages.

Davisware GlobalEdge 2008 Payroll Main Menu Time Entry and Payroll Processing.

Korea Workshop May GAE CMS Analysis (Example) Michael Thomas (on behalf of the GAE group)

AliRoot survey: Analysis P.Hristov 11/06/2013. Are you involved in analysis activities?(85.1% Yes, 14.9% No) 2 Involved since 4.5±2.4 years Dedicated.

HUBbub 2013: Developing hub tools that submit HPC jobs Rob Campbell Purdue University Thursday, September 5, 2013.

Session: 4. © Aptech Ltd. 2Creating Hyperlinks and Anchors / Session 4  Describe hyperlinks  Explain absolute and relative paths  Explain how to hyperlink.

Pavel Nevski DDM Workshop BNL, September 27, 2006 JOB DEFINITION as a part of Production.

STAR Scheduling status Gabriele Carcassi 9 September 2002.

Condor Project Computer Sciences Department University of Wisconsin-Madison Condor Job Router.

1 P-GRADE Portal hands-on Gergely Sipos MTA SZTAKI Hungarian Academy of Sciences.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES L. Betev, A. Grigoras, C. Grigoras, P. Saiz, S. Schreiner AliEn.

STAR Scheduler Gabriele Carcassi STAR Collaboration.

CERN - IT Department CH-1211 Genève 23 Switzerland t Grid Reliability Pablo Saiz On behalf of the Dashboard team: J. Andreeva, C. Cirstoiu,

+ AliEn status report Miguel Martinez Pedreira. + Touching the APIs Bug found, not sending site info from ROOT to central side was causing the sites to.

David Adams ATLAS AJDL: Abstract Job Description Language David Adams BNL June 29, 2004 PPDG Collaboration Meeting Williams Bay.

AliEn Tutorial ALICE workshop Sibiu 20 th August, 2008 Pablo Saiz.

Active-HDL Server Farm Course 11. All materials updated on: September 30, 2004 Outline 1.Introduction 2.Advantages 3.Requirements 4.Installation 5.Architecture.

Institute of Informatics & Telecommunications NCSR “Demokritos” Spidering Tool, Corpus collection Vangelis Karkaletsis, Kostas Stamatakis, Dimitra Farmakiotou.

Enabling Grids for E-sciencE LRMN ThIS on the Grid Sorina CAMARASU.

Best 20 jobs jobs sites.

ANALYSIS TRAIN ON THE GRID Mihaela Gheata. AOD production train ◦ AOD production will be organized in a ‘train’ of tasks ◦ To maximize efficiency of full.

EDGI European Desktop Grid Initiative Have you ever submitted jobs to gLite in one run? If not, I will show.

Scientific Data Processing Portal and Heterogeneous Computing Resources at NRC “Kurchatov Institute” V. Aulov, D. Drizhuk, A. Klimentov, R. Mashinistov,

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES A. Abramyan, S. Bagnasco, L. Betev, D. Goyal, A. Grigoras, C.

U.S. ATLAS Grid Production Experience

Running a job on the grid is easier than you think!

Running a job on the grid is easier than you think!

Using Paraguin to Create Parallel Programs

ALICE FAIR Meeting KVI, 2010 Kilian Schwarz GSI.

LHCb Computing Model and Data Handling Angelo Carbone 5° workshop italiano sulla fisica p-p ad LHC 31st January 2008.

Introduction to P-GRADE Portal hands-on

5. Job Submission Grid Computing.

The Condor JobRouter.

Michael P. McCumber Task Force Meeting April 3, 2006

gLite Job Management Christos Theodosiou

Product Training Program

Presentation transcript:

Pablo Saiz CAF and Grid User Forum AliEn Job Merging Pablo Saiz CAF and Grid User Forum

Job Merging Next logical step after job splitting See http://indico.cern.ch/conferenceDisplay.py?confId=31167 Concatenate the result of all subjobs of a given masterjob New status if a masterjob needs merge: INSERTED SPLITTING SPLITMERGINGDONE The ‘merging’ is another job It will wait in the queue like any other job 13 September 2018 pablo.saiz@cern.ch

1 image is better than 1000 words histo.root analysis.log Subjob 1 Merge Histo AllHisto.root Subjob 2 histo.root analysis.log User JDL Subjob 3 Merge Logs ERROR!! Alllogs.txt … histo.root analysis.log Subjob n Time INSERTED SPLIT MERGING DONE 13 September 2018 pablo.saiz@cern.ch

How to specify Merging In the JDL of the masterJob: AliEn will do: Merge={“<input>:<jdl>:<output>” (,“<input2>:<jdl2>:<output2>”)* } MergeOutputDir=“/path/where/you/want/the/output”; Default /proc/<user>/<masterid>/merge AliEn will do: submit <jdl> <masterJobId> <input> <output> <user> <procdir> <outputdir> 13 September 2018 pablo.saiz@cern.ch

How to start the merging Automatically: When all the subjobs are in a final state, AliEn sends the merging masterJob <id> merge Force the merging of the subjobs that have finished By hand: submit <jdl> <masterJobId> <input> <output> <user> <procdir> <outputdir> 13 September 2018 pablo.saiz@cern.ch

Existing merging JDLs /alice/jdl/mergerootfile.jdl /alice/jdl/mergerootfile-sequential.jdl No requirements. The merging can be executed anywhere! User defined Variations of the previous jdl There is no merging for text files: Needed? 13 September 2018 pablo.saiz@cern.ch

ToDo Given a masterjobid, follow up the status of the merging Automatically put requirements on the execution site for the merging. More documentation in the bible ? 13 September 2018 pablo.saiz@cern.ch

Conclusions Merging collects the output of subjobs into a single file Performed when all the subjobs are in a final state: ERROR or DONE Can also be trigger manually Documentation will be added to the bible 13 September 2018 pablo.saiz@cern.ch