David Adams Brookhaven National Laboratory September 28, 2006 Data validation DDM Workshop BNL David Adams Brookhaven National Laboratory September 28, 2006 Updated September 27, 2006

D. Adams BNL data validation BNL DDM Workshop Contents Goals Publication Issues Status Conclusions D. Adams BNL data validation BNL DDM Workshop September 28, 2006

D. Adams BNL data validation BNL DDM Workshop Goals Goal of the BNL data validation effort Determine which data is available at BNL Which datasets Which files in each of these datasets Validate each dataset Validity of GUID and LFN’s LFN corresponds to dataset name Duplicate file numbers within datasets Consistency of BNL replica catalog Publish results Create “BNL” datasets Include only files at BNL Remove duplicate and invalid files Registered as DSNAME_bnl in DQ2 D. Adams BNL data validation BNL DDM Workshop September 28, 2006

D. Adams BNL data validation BNL DDM Workshop Publication Validation is published on a series of web pages Starting point: http://www.usatlas.bnl.gov/~dial/atprod/validation/html BNL summary: http://www.usatlas.bnl.gov/~dial/atprod/validation/html/bnl_datasets.html Tables are updated twice a day Update time at the top of each page Automatic and fairly robust procedure Tables provide field that can be used to restrict listing Simple pattern matching with * for wildcard E.g. *Zmumu*AOD* Tables for tasks, task names, datasets and BNL resident datasets D. Adams BNL data validation BNL DDM Workshop September 28, 2006

Issues: Which datasets Which datasets should be validated? I start from the task table (BNL replica) Select tasks that begin with “csc” (task table) Combine tasks with the same name (task name table) Follow conventions to guess the datasets produced by each task Check if dataset name is registered in DQ2 (DQ2 tasks) Check if BNL is a DQ2 location for the dataset (BNL datasets) This has potential problems Conventions change and my code has to keep up Datasets become obsolete and should be dropped from validation Restricted to production datasets Preferable to have an external source listing datasets of interest Perhaps the metadata catalog D. Adams BNL data validation BNL DDM Workshop September 28, 2006

Issues: Additional validation What additional validation is desired? Check existence of physical files at BNL BNL dcache sometimes loses files and the replica catalog is not updated to reflect this Not too difficult if check is done with ls command Data inside file Right type (AOD, ESD, RDO, …) Event numbers consistent with file name Difficult because these checks require sophisticated code and reading each file Accessibility: can files be read? Again difficult because it is time consuming to open and read files Staging: report how many files in each dataset are staged Expensive to check each file with dc_check Status often changes faster than my twice daily validation checks D. Adams BNL data validation BNL DDM Workshop September 28, 2006

D. Adams BNL data validation BNL DDM Workshop Issues: Remediation Remediation When there are problems (and there are some), who should resolve them? E.g. duplicate files, files missing in dcache Problems such as duplicate files are a feature of the dataset definition and are not BNL-specific Need production expert to sort out which file should be kept Need authorization to change the dataset definition Other problems such as files disappearing from dcache and not from the replica catalog need to resolved locally D. Adams BNL data validation BNL DDM Workshop September 28, 2006

D. Adams BNL data validation BNL DDM Workshop Issues: Data movement Replicating data at BNL Which data? Long term model is all AOD and ESD And 2/N of raw? Should support reasonable user requests Can we do this now? Are we trying? 245/402 AOD datasets are at BNL Those at BNL are mostly complete Big improvement since the spring Validation At present the BNL validation table lists datasets registered at BNL Replace this selection with a policy or an external list? Or just register BNL as an (incomplete) location for desired datasets? Can/should table let users know what data is coming? Or why data is missing? D. Adams BNL data validation BNL DDM Workshop September 28, 2006

Issues: Historical information Current validation pages only provide a snapshot view Difficult to know if the situation is improving or deteriorating Historical data is available Results from each scan are saved Last week on disk, web accessible All data since June stored in dcache Interesting to track the number of datasets and files of each type as a function of time Both in DQ2 and at BNL Volunteers? D. Adams BNL data validation BNL DDM Workshop September 28, 2006

D. Adams BNL data validation BNL DDM Workshop Status Validation has been running at BNL since Spring Automated: I only need to update my proxy every week or two Fairly robust Down for a couple weeks when I was on vacation because database passwords changed Major DB failures will occasionally leave it in a state that require me to clean up BNL table provides a nice answer to the question “What data is available at BNL?” Easy to select by physics process, data type and release Up to date without requiring DB query for each request D. Adams BNL data validation BNL DDM Workshop September 28, 2006

D. Adams BNL data validation BNL DDM Workshop Conclusions Validation pages provide useful summaries For users and production experts Easy to use and understand Can be improved External list or list of datasets Additional validation Active reporting of problems Information about data movement (or lack thereof) Historical information What else? Volunteers welcome To address any of the above or whatever other features you would like to see D. Adams BNL data validation BNL DDM Workshop September 28, 2006