Code Management James N. Bellinger University of Wisconsin at Madison 19 January 2011 1.

Slides:



Advertisements
Similar presentations
Grid Wizard Enterprise Basic Tutorial Using Web Control Panel.
Advertisements

Sorting Really Big Files Sorting Part 3. Using K Temporary Files Given  N records in file F  M records will fit into internal memory  Use K temp files,
CSCI 4717/5717 Computer Architecture
22nd January 2003Tim Adye1 Summary of Bookkeeping discussions at RAL Workshop Tim Adye Rutherford Appleton Laboratory Kanga Phone Meeting 22 nd January.
11 Getting Started with ASP.NET Beginning ASP.NET 4.0 in C# 2010 Chapters 5 and 6.
Programming Logic and Design Fourth Edition, Introductory
Database Management Systems, R. Ramakrishnan and J. Gehrke1 Query Evaluation Chapter 11 External Sorting.
David Adams ATLAS DIAL Distributed Interactive Analysis of Large datasets David Adams BNL March 25, 2003 CHEP 2003 Data Analysis Environment and Visualization.
Molly, Gwyn, Sam, and Eric.  Configure DACs to have their digital input set to zero (2.5V) when they receive power  Ramp up to higher voltage at a “user.
Understanding the Mainline Logical Flow Through a Program (continued)
Operating Systems Concepts 1. A Computer Model An operating system has to deal with the fact that a computer is made up of a CPU, random access memory.
Trains status&tests M. Gheata. Train types run centrally FILTERING – Default trains for p-p and Pb-Pb, data and MC (4) Special configuration need to be.
QA train tests M. Gheata. Known problems QA tasks create too many histograms – Pushing resident memory limit above 3GB – Train gets kicked out by some.
CFT Offline Monitoring Michael Friedman. Contents Procedure  About the executable  Notes on how to run Results  What output there is and how to access.
3rd June 2004 CDF Grid SAM:Metadata and Middleware Components Mòrag Burgon-Lyon University of Glasgow.
© Chinese University, CSE Dept. Distributed Systems / Simple Example Open Microsoft Visual Studio 2005:
Code Management James N. Bellinger University of Wisconsin at Madison 16 March
David Adams ATLAS DIAL status David Adams BNL November 21, 2002 ATLAS software meeting GRID session.
Code Management James N. Bellinger University of Wisconsin at Madison 11 May
October 8, 2002P. Nilsson, SPD General Meeting1 Paul Nilsson, SPD General Meeting, Oct. 8, 2002 New tools and software updates Test beam analysis Software.
Remote Data Access with OPeNDAP Dr. Dennis Heimbigner Unidata netCDF Workshop October 25, 2012.
Legion - A Grid OS. Object Model Everything is object Core objects - processing resource– host object - stable storage - vault object - definition of.
Analysis trains – Status & experience from operation Mihaela Gheata.
Nurcan Ozturk University of Texas at Arlington US ATLAS Transparent Distributed Facility Workshop University of North Carolina - March 4, 2008 A Distributed.
David Adams ATLAS DIAL: Distributed Interactive Analysis of Large datasets David Adams BNL August 5, 2002 BNL OMEGA talk.
Package & Deploy. OBJECTIVES Package Deploy Way to package.
Efi.uchicago.edu ci.uchicago.edu FAX status report Ilija Vukotic on behalf of the atlas-adc-federated-xrootd working group S&C week Jun 2, 2014.
Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,
AliRoot survey: Analysis P.Hristov 11/06/2013. Are you involved in analysis activities?(85.1% Yes, 14.9% No) 2 Involved since 4.5±2.4 years Dedicated.
1111 Creating HTML Programatically Objectives You will be able to Invoke C# code on the server from an ASP.NET page. Write C# code to create HTML.
M. Gheata ALICE offline week, October Current train wagons GroupAOD producersWork on ESD input Work on AOD input PWG PWG31 (vertexing)2 (+
Code Management James N. Bellinger University of Wisconsin at Madison 8 December
Migration to SL5 James Bellinger University of Wisconsin at Madison 7-Apr
D0 Farms 1 D0 Run II Farms M. Diesburg, B.Alcorn, J.Bakken, R. Brock,T.Dawson, D.Fagan, J.Fromm, K.Genser, L.Giacchetti, D.Holmgren, T.Jones, T.Levshina,
Analysis train M.Gheata ALICE offline week, 17 March '09.
Migration to SL5 James Bellinger University of Wisconsin at Madison 21-Apr
External Sorting Jianlin Feng School of Software SUN YAT-SEN UNIVERSITY courtesy of Joe Hellerstein for some slides.
DZero Monte Carlo Production Ideas for CMS Greg Graham Fermilab CD/CMS 1/16/01 CMS Production Meeting.
McFarm Improvements and Re-processing Integration D. Meyer for The UTA Team DØ SAR Workshop Oklahoma University 9/26 - 9/27/2003
Joe Foster 1 Two questions about datasets: –How do you find datasets with the processes, cuts, conditions you need for your analysis? –How do.
ANALYSIS TRAIN ON THE GRID Mihaela Gheata. AOD production train ◦ AOD production will be organized in a ‘train’ of tasks ◦ To maximize efficiency of full.
September 26, 2003K User's Meeting1 CCJ Usage for Belle Monte Carlo production and analysis –CPU time: 170K hours (Aug.1, 02 ~ Aug.22, 03)
University of Wisconsin at Madison
Analysis trains – Status & experience from operation
Work report Xianghu Zhao Nov 11, 2014.
Analysis Operations Monitoring Requirements Stefano Belforte
Analysis framework - status
University of Wisconsin at Madison
University of Wisconsin at Madison
University of Wisconsin at Madison
James N. Bellinger University of Wisconsin at Madison 10 November 2010
University of Wisconsin at Madison
James N. Bellinger University of Wisconsin at Madison 13 October 2010
ADA aodhisto transformation
University of Wisconsin at Madison
University of Wisconsin at Madison
James N. Bellinger University of Wisconsin at Madison 3 November 2010
James N. Bellinger University of Wisconsin at Madison 21 July 2010
James N. Bellinger University of Wisconsin at Madison 6 October 2010
University of Wisconsin at Madison
University of Wisconsin at Madison
Accessing Remote Datasets through the netCDF interface.
University of Wisconsin at Madison
James N. Bellinger University of Wisconsin at Madison 11-January-2012
Data Challenge 1 Closeout Lessons Learned Already
CHAPTER 4 I/O PORT PROGRAMMING.
James N. Bellinger University of Wisconsin at Madison 28 July 2010
Status and plans for bookkeeping system and production tools
James N. Bellinger University of Wisconsin at Madison 4 August 2010
University of Wisconsin at Madison
Presentation transcript:

Code Management James N. Bellinger University of Wisconsin at Madison 19 January

Current Work Need to test restoration of “cdfsoft” areas Need to decide what sam product to provide offsite and how to help remote sites configure it Donatella is looking at glitches in code browser: at least one library is missing src: files disagree with descriptions, non-standard characters, other things; killing the browser Large file support investigations Ready to sign off? Create Wiki page w/ instructions User has to request large files, not default Production/setup_output.tcl 19 January 20112

Large File Checklist 19 January WhatStatus AC++ Unpatched AC++Dump reads large filesOK Large file Production vs input files histogram comparison Run 2M test to shake down I/O Apply usual Stntuple/histogramming, compare w/ oldBEWARE TH1F AC++Dump with late event in file: emulate PickEventOK jump/skip in DHInput with large fileOK? Event displayOK Compare job summary lists! Look for memory leaks in ProductionExeYes, not bad Look for memory leaks in AC++DumpYes, not bad Look for memory leaks in stnmakerYes, not bad Look for memory leaks in MC Ntuple (if TopNtuple ok, will this be OK too?) Patchlist to Ray/Hyun Su for new version Scan code for problemsOK Run over a large production file using some analysisOK Concatenate small files to largeOK Write large file from many inputs to analysisOK Read and rewrite a large fileOK Skim and thin large filesOK Check that output file content agrees with inputOK Make test dataset of several large files,upload and read back via cdfgrid/SAM/dcacheOK TBrowser works on large file? (Does it work on Stntuple?)OK Chaining large files works?OK General QuestionsStattus Does the patch work on SL4? Create simple AC++ job, 8GB file, read it back rcp works on SL5 with large filesOK Does rcp work on SL4 w/ large files? Does rootd work on SL4/4 4/5 5/4 5/5 w/ large files? Merge 0-event files w/ patch: readable? Needn't be big? TopNtuple Scan code for problemsOK Run over a large production file using TopFindsee below Concatenate small tnt files to large using TopMergeOK Write large file from many inputs to TopFind:OK ttop25 files from PythiaMC: TopFind makes 3GB fileOK Concatenate larger (>2GB) files to largeOK, not to 8GB Read and rewrite a large fileOK Skim and thin large filesOK Check that getTntFileInfo worksOK Check that output file content agrees with inputOK Content check between input and large file is identicalOK top analysis on both sets is identicalOK Make test dataset of several large files,upload and read back via cdfgrid/SAM/dcachesee above TBrowser works on large file?see above Chaining large files works?see above Large file test w/SAMMonday?