On the same page with Streaming Adam Lyon Analysis Tools - 6/04/2002

2 A. Lyon (FNAL/D0) – The Paper Trail New Phenomena Comments on the D0 Run II data access D0 Note 3313 (9/97, Bantly, Boehnlein, Hobbs for NP)  Run I inclusive streaming: ~50% replication  Disjoint classified streams based on Level 3 Filter bit, grouped by L3 physics object. 2 n -1 exclusive streams for n classifications  Stream min/zero-bias events separately (inclusive) for easy access to entire sample Study of Possible Run II Streaming Scenarios D0 Note 3326 (10/97, Schellman, Bertram)  Studied exclusive, partially exclusive, and inclusive streaming scenarios with 100,000 Run I events. Exclusive streaming with L3 classification bits. Weighs pros and cons of each scenario

3 A. Lyon (FNAL/D0) – The Paper Trail Run II Streaming - How to Stream Events D0 Note 3662 (6/99, Boehnlein, Fan, Schellman, de Souza)  Recipe for streaming in Run II LogicalStream to Physical Stream Mapping [~schellma/upgrade_computing] (2/2000 [10/2000], Schellman, Yu)  Tree algorithm scheme for combining the “primitive” streams into physical streams (2 n -1  m; m << 2 n -1) based on event probability to “load balance” the physical streams Plain English Explanation of Streaming [~schellma/upgrade_computing] (6/2001, Schellman)  Short summary of previous work

4 A. Lyon (FNAL/D0) – The Paper Trail Streaming and Level 3 [ (8/2001, Hays)  Combining Primitive Streams to Physical Streams -- two example tree algorithms Conceptual Design for Offline Data Analysis and D0 Streaming (8/2001, White) [Greg’s web page]  Offline streaming issues (SAM, Luminosity) Stream Definition Table (Strawman Proposal) (8/2001, Landsberg) [Greg’s web page]  Streaming proposal we’d like to implement as a test case

5 A. Lyon (FNAL/D0) – Streaming (as documented) at a glance From L2 L2 trigger (MU+JET) L2 trigger (ELE+JET_HI) L3 Filter Script MU_HI (0001) JT_HI (0100) L3 Filter Script ELE_LO (0010)    Primitives BitMap (0101) Physical Stream MU_STRM JT_BIG (0100)  L3 Filter assigned a “primitive” bit for the physics object it tests Primitives bit map is OR of bits from filters of passing L3 filter scripts (2 n -1) possibilities Event is mapped (via some algorithm) to physical stream using BitMap m << 2 n -1 physical streams

6 A. Lyon (FNAL/D0) – How does L3 form the Primitive BitMap?  Each L3 filter can set its bit when it runs and passes  Gives people heebee-jeebees (what if filter writer forgets to code this step)  Must OR BitMaps from passing filter scripts at the end  Use a lookup table (L3 filter script  Primitives BitMap)  Since all filters of a filter script must pass, if a filter script passes, you know the resulting bit map a priori  Download this information to L3 (in trigger database) [ Can this be made even better? See next slide…]  L3 takes no time setting bits and it always works  Can override settings easily  Still must OR BitMaps from passing filter scripts at the end to form final word

7 A. Lyon (FNAL/D0) – How to go from BitMap to Physical Stream  Run mapping algorithm in Level 3 (slide)  Jon Hays already has this  Must download algorithm configuration via XML file (Where is this configuration stored? Trigger database? “Streaming database”?)

8 A. Lyon (FNAL/D0) – BitMaps  Physical Streams [ from Hays’ document]  Simple one-to-one:  2 n -1 physical streams  Always n decisions  No ambiguity  Priority:  Flexible  Can sometimes decide in < n decisions  Can produce few streams, though must be careful that streams are not big  You have to think! What if Muon & Electron?

9 A. Lyon (FNAL/D0) – Exceptions (For now, the rule!)  Special L1 triggers (with normal physics L3 filters)  Cosmic gap trigger  Dean’s non-zero suppressed Cal trigger  Hard-wire their streams in the XML  Triggers with L3 Prescale 1 (no other L3 filters)  50% of current triggers  Hard wire stream  L3 Mark & Pass (no L3 Filter satisfied)  Go to monitor stream (inclusive)  L2 Mark & Pass [not implemented yet]

10 A. Lyon (FNAL/D0) – Conclusions  Exclusive Streaming  Some (most) triggers span more than one stream  Minimize tapes  A few groups win big, most groups still look at almost all the data  Hard to do inclusive studies ( e.g. commissioning)  Hard to determine luminosity, trigger rates  Hard to re-reco a certain class of events  Lost data in low rate streams may be devastating TOOLS WILL BE CRITICAL TO KEEP USERS HAPPY!

11 A. Lyon (FNAL/D0) – Tools  User wants to run over data from certain triggers  Determine the streams to obtain Stream definitions can change over time Needs Run database, trigger database Must be able to work through mapping table  Determine files for those streams SAM does this  Determine good runs (run over only these)  Determine bad luminosity blocks (skip these events)  User wants luminosity for data  Monitoring tools (trigger rates, verfication)

12 A. Lyon (FNAL/D0) – Tools