GETTING STARTED ON THE GRID: W. G. SCOTT (RAL/PPD) RAL PHYSICS MEETING TUES 15 MAY 2007. GENERATED 10K SAMPLES IN EACH CHANNEL ON LXPLUS (IN 2006) SIMULATED/DIGITISDED.

Slides:



Advertisements
Similar presentations
GANGA Overview Germán Carrera, Alfredo Solano (CNB/CSIC) EMBRACE COURSE Monday 19th of February to Friday 23th. CNB-CSIC Madrid.
Advertisements

Computing Lectures Introduction to Ganga 1 Ganga: Introduction Object Orientated Interactive Job Submission System –Written in python –Based on the concept.
Submitting jobs to the grid Argonne Jamboree January 2010 R. Yoshida Esteban Fullana.
Israel Cluster Structure. Outline The local cluster Local analysis on the cluster –Program location –Storage –Interactive analysis & batch analysis –PBS.
DB Audit Expert v1.1 for Oracle Copyright © SoftTree Technologies, Inc. This presentation is for DB Audit Expert for Oracle version 1.1 which.
Shell Script Examples.
1 SEEM3460 Tutorial Unix Introduction. 2 Introduction What is Unix? An operation system (OS), similar to Windows, MacOS X Why learn Unix? Greatest Software.
Using Macs and Unix Nancy Griffeth January 6, 2014 Funding for this workshop was provided by the program “Computational Modeling and Analysis of Complex.
Enabling Grids for E-sciencE gLite training at Sinaia '06 Victor Penso Kilian Schwarz GSI Darmstadt Germany.
Unix Primer. Unix Shell The shell is a command programming language that provides an interface to the UNIX operating system. The shell is a “regular”
26 April 2005Steve Armstrong1 High Level Trigger Software Tutorial Steve Armstrong Brookhaven National Laboratory 26 April 2005 ATLAS Trigger/DAQ Week.
Setup your environment : From Andy Hass: Set ATLCURRENT file to contain "none". You should then see, when you login: $ bash Doing hepix login.
A Project about: Molecular Dynamic Simulation (MDS) Prepared By Ahmad Lotfy Abd El-Fattah Grid Computing Group Supervisors Alexandr Uzhinskiy & Nikolay.
Don Quijote Data Management for the ATLAS Automatic Production System Miguel Branco – CERN ATC
David Adams ATLAS ATLAS Distributed Analysis David Adams BNL March 18, 2004 ATLAS Software Workshop Grid session.
03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio.
Unix Tutorial for FreeSurfer Users. Helpful To Know FreeSurfer Tutorial Wiki:
Launch SpecE8 and React from GSS. You can use the chemical analyses in a GSS data sheet to set up and run SpecE8 and React calculations. Analysis → Launch…
Additional UNIX Commands. 222 Lecture Overview  Multiple commands and job control  More useful UNIX utilities.
Unix Tutorial for FreeSurfer Users. Helpful To Know FreeSurfer Tutorial Wiki:
Introduction to Linux OS (IV) AUBG ICoSCIS Team Prof. Volin Karagiozov March, 09 – 10, 2013 SWU, Blagoevgrad.
David Adams ATLAS DIAL status David Adams BNL July 16, 2003 ATLAS GRID meeting CERN.
Essential Shell Programming by Prof. Shylaja S S Head of the Dept. Dept. of Information Science & Engineering, P.E.S Institute of Technology, Bangalore
UNIX Shell Script (1) Dr. Tran, Van Hoai Faculty of Computer Science and Engineering HCMC Uni. of Technology
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) GISELA Additional Services Diego Scardaci
Beyond sh Not everyone is as fond of UNIX as most other people. The tutorial talks about the dark side of UNIX.
Bookkeeping Tutorial. Bookkeeping & Monitoring Tutorial2 Bookkeeping content  Contains records of all “jobs” and all “files” that are created by production.
Generation of Radiative B-Decays Decay Channels Bs  Phi + Gamma Bd  K * + Gamma Gareth Brown, RAL PPD5/10/06 Thanks to, Bill, John, Monica and Julie.
July 28' 2011INDIA-CMS_meeting_BARC1 Tier-3 TIFR Makrand Siddhabhatti DHEP, TIFR Mumbai July 291INDIA-CMS_meeting_BARC.
Submitting jobs to the grid Argonne Jamboree January 2010 R. Yoshida (revised March 2010) Esteban Fullana.
Ganga A quick tutorial Asterios Katsifodimos Trainer, University of Cyprus Nicosia, Feb 16, 2009.
GRID. Register Fill the form. Your IP (Juanjo) signature is needed and the one from the.
Introduction to Ganga Karl Harrison (University of Cambridge) ATLAS Distributed Analysis Tutorial Milano, 5-6 February 2007
Nurcan Ozturk University of Texas at Arlington US ATLAS Transparent Distributed Facility Workshop University of North Carolina - March 4, 2008 A Distributed.
Using UNIX Shell Scripts Michael Griffiths Corporate Information and Computing Services The University of Sheffield
Karsten Köneke October 22 nd 2007 Ganga User Experience 1/9 Outline: Introduction What are we trying to do? Problems What are the problems? Conclusions.
Ganga 4 Basics - Tutorial Jakub T. Moscicki ARDA/LHCb Ganga Tutorial, November 2005.
Ganga 4 Basics - Tutorial Jakub T. Moscicki ARDA/LHCb Ganga Tutorial, September 2006.
Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,
Distributed Data Analysis with GANGA (Tutorial) Alexander Zaytsev Budker Institute of Nuclear Physics (BudkerINP), Novosibirsk On the basis of GANGA EGEE.
Using Ganga for physics analysis Karl Harrison (University of Cambridge) ATLAS Distributed Analysis Tutorial Milano, 5-6 February 2007
2 June 20061/17 Getting started with Ganga K.Harrison University of Cambridge Tutorial on Distributed Analysis with Ganga CERN, 2.
David Adams ATLAS ATLAS-ARDA strategy and priorities David Adams BNL October 21, 2004 ARDA Workshop.
ATLAS-specific functionality in Ganga - Requirements for distributed analysis - ATLAS considerations - DIAL submission from Ganga - Graphical interfaces.
INFSO-RI Enabling Grids for E-sciencE Using of GANGA interface for Athena applications A. Zalite / PNPI.
© Prepared By: Razif Razali 1 TMK 265: UNIX SYSTEM CHAPTER FOUR – QUOTING IN DETAIL.
Distributed Analysis Tutorial Dietrich Liko. Overview  Three grid flavors in ATLAS EGEE OSG Nordugrid  Distributed Analysis Activities GANGA/LCG PANDA/OSG.
Finding Data in ATLAS. May 22, 2009Jack Cranshaw (ANL)2 Starting Point Questions What is the latest reprocessing of cosmics? Are there are any AOD produced.
David Adams ATLAS ATLAS Distributed Analysis and proposal for ATLAS-LHCb system David Adams BNL March 22, 2004 ATLAS-LHCb-GANGA Meeting.
INFSO-RI Enabling Grids for E-sciencE Ganga 4 Technical Overview Jakub T. Moscicki, CERN.
A GANGA tutorial Professor Roger W.L. Jones Lancaster University.
1 Lecture 7 Introduction to Shell Scripts COP 3353 Introduction to UNIX.
DQ2 ‘User’ Experience Borut Kersevan Jozef Stefan Inst. Univ. of Ljubljana.
Linux Administration Working with the BASH Shell.
Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.
INTRODUCTION TO SHELL SCRIPTING By Byamukama Frank
Starting Analysis with Athena (Esteban Fullana Torregrosa) Rik Yoshida High Energy Physics Division Argonne National Laboratory.
ANALYSIS TRAIN ON THE GRID Mihaela Gheata. AOD production train ◦ AOD production will be organized in a ‘train’ of tasks ◦ To maximize efficiency of full.
First look at MB stream FDR files and using GANGA Craig Buttar 11 th March 2008.
Introduction to Unix for FreeSurfer Users
UNIX To do work for the class, you will be using the Unix operating system. Once connected to the system, you will be presented with a login screen. Once.
Basic aliensh S. Bagnasco, INFN Torino CNAF Nov 27-28, 2007.
MCproduction on the grid
LHCb Computing Model and Data Handling Angelo Carbone 5° workshop italiano sulla fisica p-p ad LHC 31st January 2008.
The Linux Operating System
Shell Environments.
John Carelli, Instructor Kutztown University
Chapter 3 The UNIX Shells
Essential Shell Programming
Presentation transcript:

GETTING STARTED ON THE GRID: W. G. SCOTT (RAL/PPD) RAL PHYSICS MEETING TUES 15 MAY GENERATED 10K SAMPLES IN EACH CHANNEL ON LXPLUS (IN 2006) SIMULATED/DIGITISDED 1K SUB-SAMPLES ON LXPLUS BUT LXPLUS TOO SLOW FOR LARGE SAMPLES – MOVE TO GRID? MC STUDY OF RADIATIVE B-DECAYS FOR B-PHYSICS/HLT STUDIES

csc B_Kstar_Kplus_PiMinus_gamma4.evgen._00001.pool.root csc B_kstar_Kplus_PiMinus_gamma4.evgen._00002.pool.root csc B_kstar_Kplus_PiMinus_gamma4.evgen._00003.pool.root csc B_kstar_Kplus_PiMinus_gamma4.evgen._00004.pool.root csc B_kstar_Kplus_PiMinus_gamma4.evgen._00005.pool.root csc B_Kstar_Kplus_PiMinus_gamma4.evgen._00006.pool.root csc B_Ksatr_Kplus_PiMinus_gamma4.evgen._00007.pool.root csc B_Ksatr_Kplus_PiMinus_gamma4.evgen._00008.pool.root csc B_kstar_Kplus_PiMinus_gamma4.evgen._00009.pool.root csc B_Ksatr_Kplus_piMinus_gamma4.evgen._00010.pool.root /castor/cern.ch/user/s/scottw/gjab02/ events per file = 10,000 B -> K* gamma events total Generated Bd -> K* gamma Events stored in Castor: Generation is easy: 1000 events per 8 hr job ~ 30Mb per file RELEASE csc no. is not present for these Bd -> K* gamma data sets

csc Bs_phi_Kplus_Kminus_gamma4.evgen._00001.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00002.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00003.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00004.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00005.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00006.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00007.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00008.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00009.pool.root csc Bs_phi_Kplus_Kminus_gamma4.evgen._00010.pool.root /castor/cern.ch/user/s/scottw/gjab02/ events per file = 10,000 Bs -> Phi gamma events total Bs -> phi gamma Events stored in Castor: RELEASE Generation is easy: 1000 events per 8 hr job ~ 30Mb per file

Simulated and digistsed Bd -> K* gamma Samples : B_Kstar_Kplus_PiMinus_gamma4.digi._00001.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00002.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00003.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00004.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00005.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00006.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00007.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00008.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00009.pool.root B_Kstar_Kplus_PiMinus_gamma4.digi._00010.pool.root castor/cern.ch/user/s/scottw/gjab02/ events per file = 1000 Bd -> K* gamma events total 100 EVENTS PER 1 DAY JOB ON LXPLUS = TOO SLOW !! ~200Mb PER 100 EVENT FILE!! RELEASE USING STANDARD SIMUL+DIGIT TRANSFORMATIONS

Simulated and digitised Bs -> phi gamma sample : csc Bs_phi_Kplus_Kminus_gamma4.digit._00001.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00002.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00003.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00004.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00005.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00006.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00007.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00008.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00009.pool.root csc Bs_phi_Kplus_Kminus_gamma4.digit._00010.pool.root castor/cern.ch/user/s/scottw/gjab02/ events per file = 1000 Bs -> Phi gamma events total 100 EVENTS PER 1 DAY JOB ON LXPLUS = TOO SLOW !! ~200Mb PER 100 EVENT FILE!! RELEASE USING STANDARD SIMUL+DIGIT TRANSFORMATIONS

LVL-! TRIGGER EFFIENCY PLOT (THANKS JULIE!) LVL-1 THR: 3 GeV 4 GeV 5 GeV

MORE JULIE PLOTS!!

MOVING TO THE GRID: /afs/cern.ch/user/s/scottw/mycertificate.pfx You need a copy of your Grid Certificate in your home directory: Go to your work directory and use z-shell: cd public/b1206 zsh Execute the following commands to setup Ganga (and Don-Quijote): source /afs/cern.ch/project/gd/LCG-share/sl3/etc/profile.d/grid_env.sh source /afs/usatlas.bnl.gov/Grid/Don-Quijote/dq2_user_client/setup.zsh.CERN export PATH=${PATH}:/afs/cern.ch/sw/ganga/install/slc3_gcc323/4.2.12/bin/ export GANGA_CONFIG_PATH=GangaAtlas/Atlas.ini voms-proxy-init Enter your 18-character pass phrase in response to this last command: ****************** ALL QUESTIONS TO S. DALLISON PLEASE!!

PREPARE A SIMPLE JOB TO RUN ON THE GRID: emacs mygangajob.py Prepare the “job” as a simple python script (in your work area)t: j=Job() j.application=Executable() j.application.exe=File('$HOME/public/b1206/myscript.sh') j.backend=LCG() j.submit() Prepare the “exe” referenced above as a simple shell script: emacs myscript.sh #!/bin/sh echo 'myscript.sh running...' echo " " /bin/hostname echo " " env This script does essentially nothing except to announce that it is running and list all the environment variables

RUN THE JOB USING GANGA: Start Ganga and submit the job: ganga *** Welcome to Ganga *** Version: Ganga etc. In[1]: execfile(`mygangajob.py’) submitting job 58 to LCG backend …. etc. In[2]:jobs (to see status of all jobs) In[3]:jobs[58] (to see detailed status of job 58) job status changes: submitted /new? running completed /failed When the job is completed, vew the output using !less: In[4]!less /afs/cern.ch/user/s/scottw/gangadir/workspace/Local/58/output/stdout CNTRL D (to quit ganga!) myscript.sh running wn59.itep.ru ….. etc. q (to quit less command) e.g. HOSTNAME=wn59.itep.ru HOSTNAME=r4b-s16.local HOSTNAME=wn60.itep.ru HOSTNAME=wn072.athena.hellasgrid.gr …. etc. It seems that you run Ganga for the first time …creates /afs/cern.ch/user/s/scottw/.gangarc ganga --gui to use the gui

dq2_ls -g user.billscott.testregister011 dq2_ls user.billscott.testregister011 NEED TO REGISTER FILES IN DON-QUIJOTE: Now register the file(s) in Don Quijote: nsls -l /castor/cern.ch/user/s/scottw/gjab02/ csc B_Kstar_Kplus_PiMinus_gamma4.evgen._00011.pool.root List my input file in castor: pool_insertFileToCatalog rfio:/castor/cern.ch/user/s/scottw/gjab02/ csc B_Kstar_Kplus_PiMinus_gamma4.evgen._00011.pool.root Insert the file into the PoolFile Catalog: dq2_put -p PoolFileCatalog.xml user.billscott.testregister011 List the dataset in Don Quijote: List the dataset and the files comprising: (dq2_ls –h for help on dq2_ls) (dataset name) (file name)

j=Job() j.application=AthenaMC() j.application.random_seed=' ' j.application.production_name='csc' j.application.process_name='B_Kstar_Kplus_PiMinus_gamma4' j.application.run_number='018200' j.application.firstevent='1' j.application.number_events_job='10' j.application.atlas_release='12.0.6' j.application.transform_archive='AtlasProduction_12_0_6_1_noarch.tar.gz' j.application.mode='simul' j.application.geometryTag = 'ATLAS-CSC ' j.application.datasets.indirectory="user.billscott.testregister001" j.application.datasets.inputfile="csc B_Kstar…gamma4.evgen._00011.pool.root" j.backend=LCG() #j.backend.CE='heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-atlas' j.backend.CE='ce102.cern.ch:2119/jobmanager-lcglsf-grid_2nh_atlas' j.backend.middleware='EDG' j.outputdata=DQ2OutputDataset() j.application.se_name='RALDISK' j.splitter=AthenaMCSplitterJob() j.splitter.numsubjobs = 2 j.splitter.nsubjobs_inputfile = j.splitter.numsubjobs # j.submit() MY CURRENT GANGA SIMUL/DIGIT JOB: emacs gsim006.py

cd /tmp dq2_get -r user.billscott.testregister001 csc B_Kstar…gamma4.evgen._00011.ntup.root.61 Use Don-Quijote to “Get” the Output file: WHEN THE JOB HAS RUN: Remove the job number appended the end of the output file before using root: mv csc B_Kstar…gamma4.evgen._00011.ntup.root.61 csc B_Kstar…gamma4.evgen._00011.ntup.root root csc B_Kstar…gamma4.evgen._00011.ntup.root How do I clean-up/remove all the old jobs/files in Ganga and start fresh?? BIG BROTHER IS WATCHING YOU/ME!! !less /afs/cern.ch/user/s/scottw/gangadir/workspace/Local/27/0/output/stdout Use the “less” command to view the Output file: Subjob #Job # unix command prefixed by shriek !