Your university or experiment logo here Performance Monitoring Gidon Moont e-Science, HEP, Imperial College London Talk to JRA1.

Slides:

Advertisements

Similar presentations

Workload management Owen Maroney, Imperial College London (with a little help from David Colling)

Advertisements

VersaCall’s Performance Intelligence Software Shop Floor Reporting Software Real Time Status Visibility Management Summary Reports Trend & Analysis Reports.

Test Case Management and Results Tracking System October 2008 D E L I V E R I N G Q U A L I T Y (Short Version)

RETRIEVING DATA FROM FCC LICENSE DATABASE Steps for obtaining query results, and importing it into MS Excel Spreadsheet.

MCTS GUIDE TO MICROSOFT WINDOWS 7 Chapter 10 Performance Tuning.

Development of test suites for the certification of EGEE-II Grid middleware Task 2: The development of testing procedures focused on special details of.

INFSO-RI Enabling Grids for E-sciencE EGEE Middleware The Resource Broker EGEE project members.

David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL March 25, 2003 CHEP 2003 Data Analysis Environment and Visualization.

1 Searching the Web Junghoo Cho UCLA Computer Science.

Using ADO.NET Chapter Microsoft Visual Basic.NET: Reloaded 1.

Basic Grid Job Submission Alessandra Forti 28 March 2006.

DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.

Oxford Jan 2005 RAL Computing 1 RAL Computing Implementing the computing model: SAM and the Grid Nick West.

IMPORT WIZARD 491a Summer 2005 Roudabeh Moraghebi.

Tutorial 11: Connecting to External Data

Stuart K. PatersonCHEP 2006 (13 th –17 th February 2006) Mumbai, India 1 from DIRAC.Client.Dirac import * dirac = Dirac() job = Job() job.setApplication('DaVinci',

1 Archive-It Training University of Maryland July 12, 2007.

LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.

Tuesday, December 4, 2012 Getting Started with Citrix Marketing Concierge Sheralyn Felix, Sr. WW Online Channel Marketing Coordinator.

1 Network Statistic and Monitoring System Wayne State University Division of Computing and Information Technology Information Technology.

MCTS Guide to Microsoft Windows 7

The GAVO Cross-Matcher Application Hans-Martin Adorf, Gerard Lemson, Wolfgang Voges GAVO, Max-Planck-Institut für extraterrestrische Physik, Garching b.

Copyright © 2007, Oracle. All rights reserved. Managing Concurrent Requests.

M1G Introduction to Database Development 6. Building Applications.

Real Time Monitor of Grid Job Executions Janusz Martyniak Imperial College London.

Organisation Management and Policy Group (MPG): Responsible for setting and policy decisions and resolving any issues concerning fractional usage, acceptable.

Learningcomputer.com SQL Server 2008 – Profiling and Monitoring Tools.

Grid infrastructure analysis with a simple flow model Andrey Demichev, Alexander Kryukov, Lev Shamardin, Grigory Shpiz Scobeltsyn Institute of Nuclear.

Stuart Wakefield Imperial College London Evolution of BOSS, a tool for job submission and tracking W. Bacchi, G. Codispoti, C. Grandi, INFN Bologna D.

Send all X-Ray’s to All X-Ray’s received by App Man will be scrubbed of any Customer Names or Identity using.

GridPP Deployment & Operations GridPP has built a Computing Grid of more than 5,000 CPUs, with equipment based at many of the particle physics centres.

Query – One of the objects in Microsoft Access – It can help users extract data, which meets the criteria defined by them, from a database file. – It must.

CERN IT Department CH-1211 Genève 23 Switzerland t Monitoring: Tracking your tasks with Task Monitoring PAT eLearning – Module 11 Edward.

Introduction to Enterprise Guide Jennifer Schmidt Rhonda Ellis Cassandra Hall.

INFSO-RI Enabling Grids for E-sciencE SA1 and gLite: Test, Certification and Pre-production Nick Thackray SA1, CERN.

TERENA 2003, May 21, Zagreb TERENA Networking Conference, 2003 MOBILE WORK ENVIRONMENT FOR GRID USERS. TESTBED Miroslaw Kupczyk Rafal.

DDM Monitoring David Cameron Pedro Salgado Ricardo Rocha.

David Adams ATLAS DIAL/ADA JDL and catalogs David Adams BNL December 4, 2003 ATLAS software workshop Production session CERN.

1 Andrea Sciabà CERN Critical Services and Monitoring - CMS Andrea Sciabà WLCG Service Reliability Workshop 26 – 30 November, 2007.

Website: Answering Continuous Queries Using Views Over Data Streams Alasdair J G Gray Werner.

LCG workshop on Operational Issues CERN November, EGEE CIC activities (SA1) Accounting: current status

Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,

INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.

 CMS data challenges. The nature of the problem.  What is GMA ?  And what is R-GMA ?  Performance test description  Performance test results  Conclusions.

PPDG February 2002 Iosif Legrand Monitoring systems requirements, Prototype tools and integration with other services Iosif Legrand California Institute.

Characterization of a Computational Grid as a Complex System Lovro Ilijasic ( Lorenza Saitta

CERN IT Department CH-1211 Geneva 23 Switzerland t A proposal for improving Job Reliability Monitoring GDB 2 nd April 2008.

FTS monitoring work WLCG service reliability workshop November 2007 Alexander Uzhinskiy Andrey Nechaevskiy.

CERN - IT Department CH-1211 Genève 23 Switzerland CCRC Tape Metrics Tier-0 Tim Bell January 2008.

Accounting in LCG/EGEE Can We Gauge Grid Usage via RBs? Dave Kant CCLRC, e-Science Centre.

Status of MICE on the GRID  MICE VO u CEs  G4MICE Installation  Example test job  Station QA Analysis  Analysis jobs  File Storage  Documentation.

The National Grid Service User Accounting System Katie Weeks Science and Technology Facilities Council.

Analysis of job submissions through the EGEE Grid Overview The Grid as an environment for large scale job execution is now moving beyond the prototyping.

The DataGrid Project NIKHEF, Wetenschappelijke Jaarvergadering, 19 December 2002

INFSO-RI Enabling Grids for E-sciencE gLite Test and Certification Effort Nick Thackray CERN.

D.Spiga, L.Servoli, L.Faina INFN & University of Perugia CRAB WorkFlow : CRAB: CMS Remote Analysis Builder A CMS specific tool written in python and developed.

CERN - IT Department CH-1211 Genève 23 Switzerland t Grid Reliability Pablo Saiz On behalf of the Dashboard team: J. Andreeva, C. Cirstoiu,

DBS Monitor and DAN CD Projects Report July 9, 2003.

INFSO-RI Enabling Grids for E-sciencE File Transfer Software and Service SC3 Gavin McCance – JRA1 Data Management Cluster Service.

TIFR, Mumbai, India, Feb 13-17, GridView - A Grid Monitoring and Visualization Tool Rajesh Kalmady, Digamber Sonvane, Kislay Bhatt, Phool Chand,

Advanced Taverna Aleksandra Pawlik University of Manchester materials by Katy Wolstencroft, Aleksandra Pawlik, Alan Williams

Enabling Grids for E-sciencE Work Load Management & Simple Job Submission Practical Shu-Ting Liao APROC, ASGC EGEE Tutorial.

A Statistical Analysis of Job Performance on LCG Grid David Colling, Olivier van der Aa, Mona Aggarwal, Gidon Moont (Imperial College, London)

On behalf of D. Colling, G. Moont, M. Aggarwal

Accounting at the T1/T2 Sites of the Italian Grid

Introduction to Grid Technology

JRA2 Pisa, Tuesday, 25 October 2005

gLite Information System

EGEE Middleware: gLite Information Systems (IS)

Title Goes Here Subtitle goes here if needed Introduction Methods

Presentation transcript:

Your university or experiment logo here Performance Monitoring Gidon Moont e-Science, HEP, Imperial College London Talk to JRA1 All-Hands CERN

Your university or experiment logo here 24 March 2006Performance Monitoring Introduction How we gather data. How we release the information. –Real Time Monitor –LCG Load Monitor –Daily Reports –XML files and ROOT analysis Interesting metrics

Your university or experiment logo here 24 March 2006Performance Monitoring How we gather data The data comes from direct queries of the mySQL databases of Resource Brokers. Around 30 Resource Brokers currently monitored. Queries once a minute. –find all jobs that had an event in the last minute –retrieve status and CE/WN information –write a complete (XML) description of all jobs –remove jobs that have finished status after 2 hours (or if Cleared) –As a job is removed, query all events and write a summary file Multithreaded (one thread per RB) Java program.

Your university or experiment logo here 24 March 2006Performance Monitoring Current RB List gdrb01.cern.ch lcgrb01.gridpp.rl.ac.ukrb01.pic.es gdrb02.cern.ch gfe01.hep.ph.ic.ac.ukrb-egee.bifi.unizar.es gdrb03.cern.ch egee-rb-01.cnaf.infn.itgrid09.lal.in2p3.fr gdrb04.cern.ch egee-rb-02.cnaf.infn.itnode04.datagrid.cea.fr gdrb06.cern.ch egee-rb-03.cnaf.infn.itmu3.matrix.sara.nl gdrb07.cern.ch gridit-rb-01.cnaf.infn.itrb.isabella.grnet.gr gdrb08.cern.ch a gridka.derb101.grid.ucy.ac.cy gdrb09.cern.ch grid-rb0.desy.degrid151.kfki.hu gdrb10.cern.ch grid-rb2.desy.delcg16.sinp.msu.ru gdrb11.cern.ch lcg00124.grid.sinica.edu.tw rb.phy.bg.ac.yu ui.ulakbim.gov.tr

Your university or experiment logo here 24 March 2006Performance Monitoring Real Time Monitor The Real Time Monitor has developed from a demo to show real time usage of the LCG Further development will include sortable tables of RB/CE info Java applet - does not require extra libraries

Your university or experiment logo here 24 March 2006Performance Monitoring LCG Load Monitor Requested as a tool to monitor London Tier 2 Java Application Can monitor RBs, CEs, and groups of CEs (eg a T2) Jobs colour coded by VO (stacked) Sortable table of all current jobs

Your university or experiment logo here 24 March 2006Performance Monitoring Daily Reports PDF documents created automatically at 3am Provides counts and metrics for all jobs that left the RTM in a 24 hour period Analysis split by –Resource Brokers –Virtual Organisation –Computing Element Metrics can identify problems Data used to generate reports is available as a tab delimited plain text file on request

Your university or experiment logo here 24 March 2006Performance Monitoring XML Files and ROOT Information from each RB is presented as an XML file For efficiency reasons the RTM and LCG Load programs use a single plain text file To see long term trends, the data is imported into ROOT. Graphs can then be made with larger data sets, and time dependent trends can be shown. We currently have data for half a year (from September now) ROOT file available on request

Your university or experiment logo here 24 March 2006Performance Monitoring Interesting Metrics We can identify RB problems by looking at the match time for jobs. We have established that all RBs slow down with more than 10 jobs/second being submitted. We can show VO behaviour by average job lengths and success rates, as well as the usage of LCG components (RBs/CEs used) and the number of users (unique DNs). We can measure CE/VO efficiency by both the fraction of successful jobs AND by the amount of computational WN time that resulted in a Done (Success) state against the total time of all jobs (including those that failed) - labeled as “Useful Time”.

Your university or experiment logo here 24 March 2006Performance Monitoring RB Match Times Job scheduling (Match Time) versus load (mean number of jobs/sec during the matching)

Your university or experiment logo here 24 March 2006Performance Monitoring DNs over time / VO We can see weekends, as well as relative users per VO

Your university or experiment logo here 24 March 2006Performance Monitoring Useful Time Useful time for those CEs that had more than jobs submitted from September February 2006 inclusive.

Your university or experiment logo here 24 March 2006Performance Monitoring URLS etc.