LAT Data Server Serve what?

Slides:



Advertisements
Similar presentations
Database Management Systems, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 11.
Advertisements

GLAST Collaboration Meeting, March 2008 T.Johnson1/22 GLAST Large Area Telescope Data Access Tony Johnson Stanford Linear Accelerator Center
Software for Science Support Systems EVLA Advisory Committee Meeting, March 19-20, 2009 David M. Harland & Bryan Butler.
GLAST LAT Project January 2005 Data Handling Workshop, SLAC 1 Data Server Needs from the Science Tools Perspective [Needs from the Perspective of.
23/04/2008VLVnT08, Toulon, FR, April 2008, M. Stavrianakou, NESTOR-NOA 1 First thoughts for KM3Net on-shore data storage and distribution Facilities VLV.
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL March 25, 2003 CHEP 2003 Data Analysis Environment and Visualization.
Page 1JSOC Review – 17 March 2005 Database Servers Challenges A very large database on the order of a few TB -- We can't copy the whole database in real.
GLAST LAT Project ISOC Peer Review - March 2, 2004 Document: LAT-PR Section 4.3 Pipeline, Data Storage and Networking Issues1 Gamma-ray Large.
GLAST LAT ProjectGround Software Workshop, July 15-18, Wrap-up: Science Tools Seth Digel HEPL/Stanford Univ. 18 July 2003 EGRET >300 MeV What we.
GLAST LAT Project 1S. Ritz Discussion: To Where From Here? DC1 Closeout Meeting February 13, 2004 S. Ritz Gamma-ray Large Area Space Telescope.
1 External Sorting Chapter Why Sort?  A classic problem in computer science!  Data requested in sorted order  e.g., find students in increasing.
External Sorting 198:541. Why Sort?  A classic problem in computer science!  Data requested in sorted order e.g., find students in increasing gpa order.
Secondary Storage Management Hank Levy. 8/7/20152 Secondary Storage • Secondary Storage is usually: –anything outside of “primary memory” –storage that.
GLAST LAT ProjectDOE/NASA Baseline-Preliminary Design Review, January 8, 2002 K.Young 1 LAT Data Processing Facility Automatically process Level 0 data.
Ch. 6 – Switch Configuration
3rd June 2004 CDF Grid SAM:Metadata and Middleware Components Mòrag Burgon-Lyon University of Glasgow.
06/02/2006 M.Razzano - DC II Closeout Meeting Pulsars in DC2 preliminary results from an “optimized” analysis Gamma-ray Large Area Space Telescope Massimiliano.
8th November 2002Tim Adye1 BaBar Grid Tim Adye Particle Physics Department Rutherford Appleton Laboratory PP Grid Team Coseners House 8 th November 2002.
Bookkeeping Tutorial. Bookkeeping & Monitoring Tutorial2 Bookkeeping content  Contains records of all “jobs” and all “files” that are created by production.
U NIVERSITY OF M ASSACHUSETTS A MHERST Department of Computer Science Computer Systems Principles Concurrency Patterns Emery Berger and Mark Corner University.
Introduction Advantages/ disadvantages Code examples Speed Summary Running on the AOD Analysis Platforms 1/11/2007 Andrew Mehta.
CPSC 404, Laks V.S. Lakshmanan1 External Sorting Chapter 13: Ramakrishnan & Gherke and Chapter 2.3: Garcia-Molina et al.
Online Reconstruction used in the Antares-Tarot alert system J ü rgen Brunner The online reconstruction concept Performance Neutrino doublets and high.
GLAST Science Support CenterJuly, 2003 LAT Ground Software Workshop Status of the D1 (Event) and D2 (Spacecraft Data) Database Prototypes for DC1 Robert.
Source catalog generation Aim: Build the LAT source catalog (1, 3, 5 years) Jean Ballet, CEA SaclayGSFC, 29 June 2005 Four main functions: Find unknown.
GLAST LAT Project LAT Instrument Analysis Workshop – Feb 27, 2006 Hiro Tajima, TKR Data Processing Overview 1 GLAST Large Area Telescope: TKR Data Processing.
Bookkeeping Tutorial. 2 Bookkeeping content  Contains records of all “jobs” and all “files” that are produced by production jobs  Job:  In fact technically.
BaBar and the GRID Tim Adye CLRC PP GRID Team Meeting 3rd May 2000.
AliRoot survey: Analysis P.Hristov 11/06/2013. Are you involved in analysis activities?(85.1% Yes, 14.9% No) 2 Involved since 4.5±2.4 years Dedicated.
Database Management Systems, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 11.
June 27-29, DC2 Software Workshop - 1 Tom Stephens GSSC Database Programmer GSSC Data Servers for DC2.
Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.
GLAST Beamtest 2006 Pisa R.Dubois1/3 Offline Possibilities for Ancillary Data Handling.
BaBar & Grid Eleonora Luppi for the BaBarGrid Group TB GRID Bologna 15 febbraio 2005.
ANALYSIS TRAIN ON THE GRID Mihaela Gheata. AOD production train ◦ AOD production will be organized in a ‘train’ of tasks ◦ To maximize efficiency of full.
Source catalog generation Aim: Build the LAT source catalog (1, 3, 5 years) Jean Ballet, CEA SaclaySLAC, 23 May 2005 Four main functions: Find unknown.
Overview of software tools for gLite installation & configuration
Internet/Web Databases
Storage Area Networks The Basics.
Global Data Access – View from the Tier 2
Credits: 3 CIE: 50 Marks SEE:100 Marks Lab: Embedded and IOT Lab
Glast Collaboration Data Server and Data Catalog
Operating System.
IT-DB Physics Services Planning for LHC start-up
Towards a CTA high-level science analysis framework
Designing For Testability
External Sorting Chapter 13
Steve Ko Computer Sciences and Engineering University at Buffalo
Existing Perl/Oracle Pipeline
“Running Monte Carlo for the Fermi Telescope using the SLAC farm”
Essentials of UrbanCode Deploy v6.1
Artem Trunov and EKP team EPK – Uni Karlsruhe
Cloud based Open Source Backup/Restore Tool
Universita’ di Torino and INFN – Torino
Lessons Learned: The Organizers
Steve Ko Computer Sciences and Engineering University at Buffalo
GLAST Large Area Telescope
External Sorting Chapter 13
PROOF - Parallel ROOT Facility
Secondary Storage Management Brian Bershad
Data Challenge 2: Aims and Status
Chapter 2: Operating-System Structures
Secondary Storage Management Hank Levy
Data Challenge 1 Closeout Lessons Learned Already
GLAST Large Area Telescope Instrument Science Operations Center
Status and plans for bookkeeping system and production tools
External Sorting Chapter 13
Lecture 4: File-System Interface
Users and Administrators
Presentation transcript:

LAT Data Server Serve what? Glibly stated – all the ‘science’ output files produced by the pipeline From I&T, generic MCs and Level 1 operations Real data – instrument analysis (eg runs taken with different hardware configurations, etc) – I&T, ISOC MC & Level 1 – astronomy analyses Include pointing, livetime histories? Event selection by location on sky, time, energy mainly Perhaps connected to FRED We envisage a pool of servers running Gleam with remote internet access. Data server technology presumably needed for finding the desired events. To whom? LATers anywhere Not just at SLAC Though data transfer latencies for SLAC users will be minimal, since data will just be moving around local nfs disks. Prototype Server for DC1 Written by Navid (see his talk) Allowed for event selection on Merit ntuples by user supplied TCut on all allowed variables; or via web form for standard cut variables Found all the events that pass and created output file for user to fetch Could find “Tree” events by (run, event) ID Was told by hand what directories to search

Issues Pipeline database is processing-centric and only contains file information Events are known to be in particular files by run ID, but nothing else is know about them Do we need to recast or mirror the db to enhance access for the server? Do we need to index the events in some fashion to get good performance? And only for ‘astronomical’ analysis events? What is good performance? As fast as reasonable? As we can afford? Spec’ed out by SSC for L1 server; what about instrument analysis? Are the promises of random access to Root files + PROOF just hot air? Can we do parallel processing? C++ API? For local to SLAC nfs access to files eg pipelinedataSets.cxx class for access to Root files Any possibility of remote access (à la xrootd)? Output targeted for ScienceTools in FITS Access to files generated not-at-SLAC Quite related to pipeline issues of how to maintain the bookkeeping for MC (at least) generated on LAT CPU farms What about Level 2 products??