Peta-Cache: Electronics Discussion I Gunther Haller Presentation Ryan Herbst, Mike Huffer, Leonid Saphoznikov Gunther Haller haller@slac.stanford.edu (650) 926-4257

Content Collection of information so base-line is common Couple of slides how it seems to be done presently Part 1: Option 3: (Tuesday) Alternative architecture with skim builder and event server Part 2: Option 1 and 2 (Thursday) Using Flash with minimum changes to present architecture Using Flash in more changes to present architecture

Collection of Info (to be checked) Total number of BBR events: ~ 400 Million Original dataflow events: ¼ to ½ Mbytes -> reconstruction results in 16-kbytes raw events From raw events (16 kbytes) generate Mini’s Micros (~2.5 kbytes) (MC’s ~ 5kbytes) Tags (512 bytes) Raw (reconstructed) events: 400 Million x 16 kbytes = 6 Terrabytes Fraction of them are needed most often Estimate: 200 skims total Large skims use 10-20% of total # of events Smallest use ~<1% of total Question of how many of those are needed how frequently Presently about 6-10 ms disk-access time What is event analysis time Very preliminary benchmark for analysis with little processing including decompression 35 ms if data in memory 70 ms if data on disk What is the number as a function of users

More info Now: each type of *new* skim involves linking new code into the babar (to zeroth order, single image!) application, which entails a significant amount of time for creating a new build, testing and worrying about the effect of its robustness on the rest of operation etc. What does Redirector do? (check): effectively maps an event *number* requested by the user to a file some where in the storage system (the combination of all the disks over all the tape system). Basically, if the user doesn't have an event in its xdroot server, it finds out which one has the file on the disk it serves and then asks the original server to make *copy* of it onto its *own* disks. If no server has the file then the user's xrootd server goes to the tape (HPSS) system and fetches it Xrootd server: ~ $6k Disk: ~$2/Gbyte (including cooling, etc)

Raw Events - Tags – Skims Raw-Data (12-kbytes) Micros (2.5 KBytes) Skim Tag data (512 Bytes) Tag data (512 Bytes) Tag data (512 Bytes) Collection of tags (with pointers to raw-data or can contain raw-data Alls tags, or all micros, etc Contains interesting characteristics and pointer to raw data Tag data and sometimes micro data is used for skims Run over tag data with defined algorithm and produce a skim Pointer skim: collection of tags with pointers to raw-data Deep skim: contains also raw-data (e.g. for transfer to remote institution) About 200 different skims. Time to get a new skim can be about 9 months due process (formalism to make sure skims are of interest to many) One goal is to be able to produce easily and fast new skim

Present Architecture (needs check) Skims: go thru all tags (sometimes micro’s, but never raw-events) on disk to make collection of tags (which is a skim) Client broadcasts to xrootd servers the first event to be fetched in a specific skim If no xrootd servers responds (i.e. time-out) then go to tape and put event into xrootd server So populate the disks with events which have been requested If other xrootd has it, it copies it over Still read ~2-6 Tbytes/day from tape? Does xrootd server really know what it has in its disks? is redirector involved for every event to find where event is from event number client is supplying? What if disk is full? How recover new space? Seems inefficient use of space since many copies 4.8 TB Mass Storage (RAID-5) 3 TB usable 4.8 TB Mass Storage (RAID-5) 3 TB usable 40 sets, to be extended to 80 sets Fibre-Channel xrootd server xrootd server Tape 1-G Ethernet Switch Redirector Client (1, 2, or 4 core) Client (1, 2, or 4 core) Up to 1,500 cores in ~ 800 units?

Present Architecture (needs check) con’t At the end, all tags and events which client desires on a single xrootd server Why are all of those copied into one place, because the client or other clients reuses them many times later? What is xrootd server really doing? Xrootd server fetches event from its disk and delivers to client So select clients for a server which uses same skims? Client decompresses event for analysis 4.8 TB Mass Storage (RAID-5) 3 TB usable 4.8 TB Mass Storage (RAID-5) 3 TB usable 40 sets, to be extended to 80 sets FiberChannel xrootd server xrootd server Tape 1-G Ethernet Redirector Client (1, 2, or 4 core) Client (1, 2, or 4 core) Up to 1,500 cores in ~ 800 units?

Goals (really need requirements) Reduce time/overhead to make new skim Goal that each user can make own “skim” quickly Reduce time to get data to client Presently analysis time and time to get next event is not concurrent but sequential. To get next event it queries all xrootd servers If one has it, it is copied into its (for that analysis) xrootd server If not, its xrootd server gets it from tape Issue is latency to get data from disk Minimize duplicate storage

Option with Skim Builder and Event Server (1) Skim builder and event server have access to all disks Client gives skim builder code to build skim Results are given to client Client broadcast/multicast to event servers the request of events in the list (name of list and name of client) Event server is selected which fits the criteria best Maybe already has the list, may not doing anything, maybe whoever has most in the list is selected, that is flexible (or optionally a specific event server can be locked to a client (directly IO) If no event in cache then after initial latency going to disk, get all the data Event server may multicast its events to be delivered to all which requested any events in the list Clients don’t have to get events in list in order Any client can get events at 100 Mbytes/s independent of number of clients Event server is cache box (e.g. 16 Gbytes of mem) Pipe between event servers and storage is slower but averaged due to cache Client could consume ~< 50 events/sec. Number of clients, e.g. 4,000. each events is 16kbytes. So average thruput 3.6Gbyte/sec Tape Disk Storage Disk Storage To file system Skim builder (s) Event Server Event Server Ethernet/PCI-E/etc Optionally direct IO Client (1, 2, or 4 core) Client (1, 2, or 4 core) Client (1, 2, or 4 core) Up to 1,500 cores in ~ 800 units?

Event Processing Center file system fabric disks pizza box as skim builder switch HPSS switch(s) switch (s) sea of cores fabric out protocol conversion pizza box in protocol conversion Event processing node

Pizza box block diagram (In) PCI Express x16 PLX8532 x4 PPC 405 PLX8508 RLDRAM II IGbyte x4 x16 PLX8508 XILNIX XC4VFX40 (Out) PCI Express PLX8532