July 10, 2006ElizabethGallas1 Luminosity Database Elizabeth Gallas Fermilab Computing Division / D0 Computing and Analysis Group D0 Database ‘Taking Stock’ meeting July 10, 2006

ElizabethGallas2 Outline Luminosity Database Status Client/Server Status Short Term activities Long Term activities Storage Requirements Conclusions Appendix –Purpose: Luminosity Offline Software –Lum Database is central to Lum Offline Software –These systems are extensively documented

July 10, 2006ElizabethGallas3 Luminosity Database Status Schema is stable Legacy Data is loaded Online Data is loaded automatically (lumGrabber) –evolving with other short term work (next slide) Data Quality (DQ by LBN is waiting…) –Test system online 30-March-2006 No problems found –waiting for feedback from DQ group  Cannot go to production without specifications

July 10, 2006ElizabethGallas4 Luminosity Client/Server Status Operations Reports –Need to: understand operation and discrepancies address performance lmAccess –Need to: understand operation and discrepancies address performance –Short term help: luminosity db server –Long term help: replace db server (?) The corresponding legacy systems still in place. Goal for coming year: Need to phase out legacy systems.

July 10, 2006ElizabethGallas5 LumDB: Short Term Activities Current work on the database: –Luminosity Constant – many years of studies changing all luminosity measurements back to start of Run 2A –Other improvements: Eliminating duplicate records, impose unique keys and constraints Understanding Special Values and Status Flags Fix Transition Times Get Begin/End LBN for every Store in the database loading of fuzzy or other times (transition records) Development database fully loaded Makes testing/implementation of these changes more robust Thanks to DBAs for making this possible.

July 10, 2006ElizabethGallas6 LumDB: Long Term Issues Procedures for updating the database –Examples: luminosity constant change discrepancies found –In future: establish on case by case basis –Examples exist: provide templates for future changes Program: check database vs online Scalar files –at regular intervals and especially after updates (above) Expect to start this project when ‘constant change’ is complete Online data loading –will require some maintenance/supervision –Thanks Vladimir for help so far ! In the absence of change, these systems have become more and more stable, not requiring much supervision.

July 10, 2006ElizabethGallas7 1. LumDB Table Counts (Corrections and DataQuality) TABLECURRENTSTART ROWS END ROWS LBNFactors 0 no use case RangedFactors 0 no use case QualFlags 5 No growth expected QualGroups 9 No growth expected QualUsers 8 No growth expected LBNQual 0 *684 k1,823 k LBNQualParams 0 *219 k583 k * Waiting for additional requirements from Data Quality Group.

July 10, 2006ElizabethGallas8 2. LumDB Table Counts (PK’s, Transitions) TABLECURRENTSTART ROWS END ROWS LBNS2.7 M2.0 M5.2 M ExpGroups8 No growth expected Lmtypes9 No growth expected TICKS160 No growth expected Transitions705 k *1 K8 k * Transitions – decided to include all transitions except 60 second timer, some duplicates will be eliminated.

July 10, 2006ElizabethGallas9 3. LumDB Table Counts (the big ones) TABLECURRENTSTART ROWS END ROWS STREAMS2.3 M1.4 M3.8 M PrdFileChks43 K4 M11 M LmRates385 M217 M579 M Delivered95 M72 M192 M ExpFraction755 M513 M1,370 M ExpLums355 M436 M1,162 M Triggers294 M239 M639 M

July 10, 2006ElizabethGallas10 4. LumDB Table Counts (Status Conditions) STATUS TABLE DISTINCT CONDITIONS CURRENTSTART ROWS END ROWS ELTS8213 K LBNS373.0 M0.6 M1.6 M LFS LSS51.3 M140 K374 K LTS42550 M24 M64 M PFCS00420 K1.1 M RFS TKS246 K618 K1.6 M TRS

July 10, 2006ElizabethGallas11 Conclusions Luminosity Database –Stable for some time Activity in short term will make it robust for the foreseen future –Storage: within expectations Luminosity Offline Software –client(s)/db server systems using LumDB in operation complementary to legacy systems –corresponding legacy systems still in place. –D0 & D0 Luminosity group empowered will need advice from DBA’s (as always) need specific help from FNAL CD REX on Lum DB Server issue –conversations underway, need more specifications Goal for coming year: Phase out legacy systems: –Operations Reports, –lmAccess stage files

July 10, 2006ElizabethGallas12 Appendix Backup slides

July 10, 2006ElizabethGallas13 Purpose – Luminosity Offline Software Calculate/Deliver a Luminosity Normalization –for any physics data set Verify and certify data integrity –for normalization purposes Improve error tracking and characterization Produce Operations Reports and Graphs Produce LBN Quality Reports

July 10, 2006ElizabethGallas14 Purpose – Luminosity Database The Luminosity Offline Software is centered on the Luminosity Database The Luminosity Database stores information from: –Luminosity Monitor Detector –Trigger Framework –Level 3 –Fermilab Tevatron Accelerator –Offline Data Processing (SAM) –"Group" assigned quality quality flags –"Consistency" determined quality Plays administrative role cross checking information from all online and some offline subsystems

July 10, 2006ElizabethGallas15 Documentation Luminosity Group – Home page –www-d0.fnal.gov/phys_id/luminosity/ links to documentation on all related Online and Offline systems Luminosity Offline Software Documentation –www-d0.fnal.gov/phys_id/luminosity/software/offline.html Loading online and offline Data Luminosity database description (below) clients and servers Luminosity Database Documentation (D0 note 4915) –/d0dist/dist/packages/lm_db/devel/www/ LumDB.html/d0dist/dist/packages/lm_db/devel/www/ LumDB.html the luminosity database schema database definition language in cvs descriptions of database tables and columns, relationships, data types and links to database access methods in production and development

July 10, 2006ElizabethGallas16 The Luminosity Database “Schema”