Athena I/O Component Refactorization - Overview Random notes about todays agenda. Peter Van Gemmeren (Argonne National Laboratory (US))

Slides:



Advertisements
Similar presentations
Workshop goals Promote learning: –exchange info; stimulate ideas for cooperation; add to collective knowledge base Help NDIIPP/JISC plan the future: –Bring.
Advertisements

Interoperability Aspects in Europeana Antoine Isaac Workshop on Research Metadata in Context 7./8. September 2010, Nijmegen.
Athena/POOL integration
Peter Chochula, January 31, 2006  Motivation for this meeting: Get together experts from different fields See what do we know See what is missing See.
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL March 25, 2003 CHEP 2003 Data Analysis Environment and Visualization.
CS 300 – Lecture 22 Intro to Computer Architecture / Assembly Language Virtual Memory.
Ext3 Journaling File System “absolute consistency of the filesystem in every respect after a reboot, with no loss of existing functionality” chadd williams.
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL June 23, 2003 GAE workshop Caltech.
From 3 weeks to 30 minutes – a journey through the ups and downs of test automation.
Persistence Technology and I/O Framework Evolution Planning David Malon Argonne National Laboratory 18 July 2011.
GIFT Authoring Tool 8 Aug Version 4.0 Goals Replace Course Authoring Tool (CAT) & Domain Knowledge File Authoring Tool (DAT) with a user-friendly,
Amanda L. Perkins, CPA Senior Auditor University of Arizona.
Wahid Bhimji University of Edinburgh J. Cranshaw, P. van Gemmeren, D. Malon, R. D. Schaffer, and I. Vukotic On behalf of the ATLAS collaboration CHEP 2012.
Event Metadata Records as a Testbed for Scalable Data Mining David Malon, Peter van Gemmeren (Argonne National Laboratory) At a data rate of 200 hertz,
13 September 2012 The Libraries’ Role in Research Data Management: A Case Study from the University of Minnesota Meghan Lafferty, Chemistry, Chemical Engineering,
HPC USER FORUM I/O PANEL April 2009 Roanoke, VA Panel questions: 1 response per question Limit length to 1 slide.
Developer workshop on I/O and persistence evolution LAL,Orsay, Feb 2012 Marcin Nowak (PAS BNL) Extended T/P Converters.
30 Jan 2009Elizabeth Gallas1 Introduction to TAGs Elizabeth Gallas Oxford ATLAS-UK Distributed Computing Tutorial January 2009.
2nd September Richard Hawkings / Paul Laycock Conditions data handling in FDR2c  Tag hierarchies set up (largely by Paul) and communicated in advance.
Event Data History David Adams BNL Atlas Software Week December 2001.
A Framework-oriented Bridge to Metadata Discussions David Malon, Jack Cranshaw, Peter van Gemmeren, Marcin Nowak, Alexandre Vaniachine US ATLAS Technical.
11 Sep 2009Paul Dauncey1 TPAC test beam analysis tasks Paul Dauncey.
Toward a Next-generation I/O Framework David Malon, Jack Cranshaw, Peter van Gemmeren, Marcin Nowak, Alexandre Vaniachine US ATLAS Technical Planning Meeting.
Mtivity Client Support System Quick start guide. Mtivity Client Support System We are very pleased to announce the launch of a new Client Support System.
Portable Infrastructure for the Metafor Metadata System Charlotte Pascoe 1, Gerry Devine 2 1 NCAS-BADC, 2 NCAS-CMS University of Reading PIMMS provides.
I/O Infrastructure Support and Development David Malon ATLAS Software Technical Interchange Meeting 9 November 2015.
Transformational Government Workshop 9 th December 2010 Washington DC.
CHEP 2004, Core Software Integration of POOL into three Experiment Software Frameworks Giacomo Govi CERN IT-DB & LCG-POOL K. Karr, D. Malon, A. Vaniachine.
A Flexible Distributed Event-level Metadata System for ATLAS David Malon*, Jack Cranshaw, Kristo Karr (Argonne), Julius Hrivnac, Arthur Schaffer (LAL Orsay)
AMI, Metadata, and Software Infrastructure David Malon 30 August 2010 ATLAS AMI and Metadata Workshop.
A python-based testing infrastructure Colin Bernet (IPNL)
Valeri Fine Athena/POOL integration.
I/O Strategies for Multicore Processing in ATLAS P van Gemmeren 1, S Binet 2, P Calafiura 3, W Lavrijsen 3, D Malon 1 and V Tsulaia 3 on behalf of the.
TAGS in the Analysis Model Jack Cranshaw, Argonne National Lab September 10, 2009.
Summary of persistence discussions with LHCb and LCG/IT POOL team David Malon Argonne National Laboratory Joint ATLAS, LHCb, LCG/IT meeting.
Alexei Klimentov. December 2, 2010 SW&C workshop. Database Session. December, 2010 ATLAS Metadata Handling and AMI Wokshop Highlights Alexei Klimentov.
ATLAS Distributed Analysis Dietrich Liko IT/GD. Overview  Some problems trying to analyze Rome data on the grid Basics Metadata Data  Activities AMI.
11/01/2012B.Couturier - Core Software Workshop 1 Software Development Infrastructure Main Topics Development tools Build and Release tools Tracking/Management.
ELSSISuite Services QIZHI ZHANG Argonne National Laboratory on behalf of the TAG developers group ATLAS Software and Computing Week, 4~8 April, 2011.
Distributed Analysis Tutorial Dietrich Liko. Overview  Three grid flavors in ATLAS EGEE OSG Nordugrid  Distributed Analysis Activities GANGA/LCG PANDA/OSG.
Analysis Performance and I/O Optimization Jack Cranshaw, Argonne National Lab October 11, 2011.
Next-Generation Navigational Infrastructure and the ATLAS Event Store Abstract: The ATLAS event store employs a persistence framework with extensive navigational.
W4118 Operating Systems Instructor: Junfeng Yang.
I/O aspects for parallel event processing frameworks Workshop on Concurrency in the many-Cores Era Peter van Gemmeren (Argonne/ATLAS)
I/O and Metadata Jack Cranshaw Argonne National Laboratory November 9, ATLAS Core Software TIM.
Multiplication Find the missing value x __ = 32.
POOL Based CMS Framework Bill Tanenbaum US-CMS/Fermilab 04/June/2003.
Kaltura Player Platform 2.0 August Key Advantages of Player Platform High performance full featured lead with HTML5 Most players display in 1 second.
Mini-Workshop on multi-core joint project Peter van Gemmeren (ANL) I/O challenges for HEP applications on multi-core processors An ATLAS Perspective.
Multi Process I/O Peter Van Gemmeren (Argonne National Laboratory (US))
Metadata and Supporting Tools on Day One David Malon Argonne National Laboratory Argonne ATLAS Analysis Jamboree Chicago, Illinois 22 May 2009.
ATLAS Distributed Computing Tutorial Tags: What, Why, When, Where and How? Mike Kenyon University of Glasgow.
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL May 19, 2003 BNL Technology Meeting.
Peter van Gemmeren (ANL) Persistent Layout Studies Updates.
Steal These Tips! Audit Techniques for Accountants
(on behalf of the POOL team)
CSE 374 Programming Concepts & Tools
CS703 - Advanced Operating Systems
Multicore Computing in ATLAS
Dirk Düllmann CERN Openlab storage workshop 17th March 2003
Web Engineering.
Online Software Status
GAPPS Thought Leadership Forum #38
Metadata Status and future
UNCONFERENCE Welcome to the 2012 PLA
Index tuning Hash Index.
THE BEST IS YET TO COME.
XSL API for Tag Management
Payment in Adult Day and Employment Services Work Group
New I/O navigation scheme
Presentation transcript:

Athena I/O Component Refactorization - Overview Random notes about todays agenda. Peter Van Gemmeren (Argonne National Laboratory (US))

Agenda AM 09:00 Local Welcome: Dr. Sebastien Binet 09:10 Workshop Welcome: Dr. David Malon 09:20 Athena I/O Component Refactorization - Overview: Peter Van Gemmeren 09:30 EventSelector: Paolo Calafiura –Common nucleus in Event Selectors 10:00 Discussion / Planning 10:30 Coffee 11:00 MetaDataSvc: Dr. David Malon 11:30 OutputStream: R D Schaffer –On Athena side, think a bit about how data, metadata, (and TAGs) can be written into a single file right now, multiple streams sharing a name, hoping they don't step on each other's toes –Single event data spread over multiple files (tracking and calorimentry, say, or ESD+ExtendedESD, or...) 12:30 Lunch

Agenda PM 14:00 ByteStream Components: Dr. Jack Cranshaw 14:30 Discussion / Planning 15:30 Coffee 16:00 Forum: I/O infrastructure outside Athena –Can we provide infrastructure that can be reused in part outside of Athena? in downstream non-Athena analysis, for example, but maybe even more broadly

EventSelector: Paolo Calafiura  What’s wrong with the current EventSelector: –Too many responsibilities: Iterate over collections, Throw collection incidents, Apply event selection, Read collection metadata, Iterate over files, Throw file incidents, Iterate over events, Seek events, Select events, (Provide addresses), Back navigation –Asymmetry with OutputStream No AthenaEventSelector base class –Dissimilarity between AthenaPOOL and other technologies (ByteStream) –Special treatment by Control framework Failure is really ‘no more events’

MetaDataSvc: Dr. David Malon  MetaDataSvc to handle in-file metadata was developed as an add-on component to the Athena POOL I/O infrastructure.  Should it become a base component similar to OutputStream or EventSelector (we don’t yet have the latter (yet)).  Writing is done via ‘shadow’ stream with same PFN. –Works but fragile  Merging metadata needs full athena frame work.

OutputStream: R D Schaffer  AthenaOutputStream uses a Tool to do most of the work.  Tool is technology specific and has a ConversionSvc (even though that could eliminate technology dependence). –We are using two different tools with AthenaPOOL AthenaOutputStreamTool depends on AthenaPoolCnvSvc, but not DataHeader. –Only used for PixelDCSTool (and PixelDCSTestWrite.py), TRT_StrawStatusSummarySvc, SCT_MonitorConditionsSvc, SCTCalibWriteSvc (and SCTCalib_topOptions.py) and LArG4GenShowerLib –Main difference is the use of DataHeader and infrastructure.

ByteStream Components: Dr. Jack Cranshaw  Was developed separately from AthenaPOOL infrastructure.  Some functionality had to be duplicated, because of missing common base classes: –E.g. Firing of FileIncidents  Some concepts are borrowed from AthenaPOOL: –E.g. DataHeader, Guid