Computing at Fermilab David J. Ritchie Computing Division May 2, 2006

May 2006David J. Ritchie:CD-doc-800-v1 2 Computing At Fermilab is Essential To: Control the Accelerator. Record the Detector information from the Collisions. Reconstruct the Particles’ Parameters. Globally Distribute the Data and Calibrations. Physics results from the Particles. Estimate the Errors. Publish our Discoveries. Simulate everything to check the answers. Discuss and Analyze across Physicists Worldwide

May 2006David J. Ritchie:CD-doc-800-v1 3 Computing At Fermilab is At the Edge Respond to Accelerator time scales of Pico- Seconds (10** -12 ) –Special-purpose Custom Built “Processors” with DSPs etc Collect data at Gigahertz (10**9 words/second). –Use specially built parallel optic fibers from DSPs to 100MByte buffers. Filter information in real time –~150 PCs operating in parallel to read and compute on the data in real time before. Record data to tape at 20MBytes/sec all the time the accelerator is working. … and …

May 2006David J. Ritchie:CD-doc-800-v1 4 Computer Security The Dark Side of the Force The Internet is now ~100 Million computers/users: “one in a million” events happen every day. With the net ‘somewhere on the net’ is always just next door. We are learning how to live in a world where not everyone is friendly -- but most are. A recent demonstration of an unpatched Mac showed it was hacked in 2 minutes. We use automated scanning software to detect problems and always follow up on any inkling of a problem.

May 2006David J. Ritchie:CD-doc-800-v1 5 Accelerator Controls: Measuring where the Beam is

May 2006David J. Ritchie:CD-doc-800-v1 6 How Do You Do Science? (elementary school version) 1.Observe phenomena. 2.Develop a hypothesis. 3.Use hypothesis to make predictions. 4.Devise experiment to look for predictions. 5.Obtain results demonstrating (or not) predictions. 6.Draw conclusions about correctness of hypothesis. 7.If correct, add to accumulated hypotheses which are the theory. If not, revise hypothesis and … Repeat

May 2006David J. Ritchie:CD-doc-800-v1 7 How Do You Do Science? (Large Scale Science Version) Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan and Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat

May 2006David J. Ritchie:CD-doc-800-v1 8 Computing Needs Phenomena and Hypothesis Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: access previous results PC’s (Windows, Linux), UNIX, Macs Printers – color and B/W World-wide networking MSWord, TeX, Acrobat Web preprint repositories –Arxiv.org at Cornell U.Arxiv.org –Spires at Stanford U.Spires

May 2006David J. Ritchie:CD-doc-800-v1 9 Computing Needs Organize Collaboration Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: PC’s (Windows, Linux), UNIX, Macs World-wide networking Web Pages –DZero “Top Group”DZero “Top Group” –MINOSMINOS

May 2006David J. Ritchie:CD-doc-800-v1 10 Computing Needs Propose Experiment Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: with colleagues show the feasibility of a proposal. PC’s (Windows, Linux), UNIX, Macs World-wide networking Web pages –Dark Energy Camera ProjectDark Energy Camera Project –Nova ProposalNova Proposal Simulations…

May 2006David J. Ritchie:CD-doc-800-v1 11 Simulations are done throughout the experiment lifetime -- before, during, after data taking. e.g. CDF Silicon Vertex Detector A 3D modeling program simulating particles in the detector.

May 2006David J. Ritchie:CD-doc-800-v1 12 Computing Needs Get Approved and Funded Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: Project Management Funds come from DOE, NSF, NASA, Congress, Taxpayer—YOU. PC’s (Windows, Linux, Macs) Printers – color and B/W World-wide networking MSProject, Power Point… Proposal review… –NUMI Proposal ReviewNUMI Proposal Review

May 2006David J. Ritchie:CD-doc-800-v1 13 Computing Needs Plan / Design, Build / Install Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: simulation, sensor design, engineering software…, Unusual sensors Custom electronics Computing at its core: microprocessors, silicon logic

May 2006David J. Ritchie:CD-doc-800-v1 14 Unusual Sensors

May 2006David J. Ritchie:CD-doc-800-v1 15 Computing Needs Acquire / Record / Store Data Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: Acquire, record, store Use sensors and custom electronics

May 2006David J. Ritchie:CD-doc-800-v1 16 Acquire, Record, Store Data

May 2006David J. Ritchie:CD-doc-800-v1 17 Recording Data Record data (“events”) –The sensors and custom electronics convert the voltages and currents into 300,000 bytes of 1’s and 0’s: Now, we are in the computer realm. –Get an event every few hundred microseconds. –Raw data rate — hundred’s of GB/second. Either trigger on interesting events or filter out the uninteresting ones – ~10 MB/s. What gets through, one records to disk and eventually to tape.

May 2006David J. Ritchie:CD-doc-800-v1 18 Computing Needs Analyze Data Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: Reconstruct 1’s and 0’s into tracks and particle identification Do physics analysis on resulting collections of tracks Means: PC farms, lots and lots Disk backed up by tape with robotic tape mounts Enstore Usage.Enstore Usage.

May 2006David J. Ritchie:CD-doc-800-v Programmers to make 1,000,000 lines of code to get to the results (ack. L Sexton-Kennedy)

May 2006David J. Ritchie:CD-doc-800-v1 20 Computing Needs Obtain Results Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: Do additional physics analysis. Means: Use analysis farms with their disk caching capabilities to speed data analysis. CDF Usage

ROOT An object oriented HEP analysis framework.

May 2006David J. Ritchie:CD-doc-800-v1 22 A Framework Are often histograms Results CDF Tops the Top World Average provides utilities and services.

May 2006David J. Ritchie:CD-doc-800-v1 23 And in AstroPhysics

May 2006David J. Ritchie:CD-doc-800-v1 24 To Public Data Release

May 2006David J. Ritchie:CD-doc-800-v1 25 Computing Needs Publish Conclusions Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Each stage of scientific process has different computing needs. Repeat Need: Show evidence of progress in scientific work. Means: Publish papers and conferences.Publish papers and conferences. Top Quark Mass

May 2006David J. Ritchie:CD-doc-800-v1 26 Mass of the Top Quark

May 2006David J. Ritchie:CD-doc-800-v1 27 Vital Statistics FNAL has >10,000 network connected devices They are connected by >1,000 miles of cabling There is more than 200TB of disk spinning now We have over 4 PB of data on tape Half of it is less than 2 years old

May 2006David J. Ritchie:CD-doc-800-v1 28 Moore’s Law Data Volume doubles every 2.4 years Lots of Data!

May 2006David J. Ritchie:CD-doc-800-v Petabytes of Data stored in the Fermilab Robots 4 Petabytes = 4,000 TB = 4,000,000 Gigabytes or about 100,000 Disks on your PC. Equivalent to a stack of CDs nearly 10 times as high as the Eiffel Tower Data In Fermilab Robots Eiffel Tower

May 2006David J. Ritchie:CD-doc-800-v1 30 Computing Facilities The activities map into facilities at the laboratory: –Networks –Mass Storage robotics / tape drives –Large computing farms –Databases –Operations –Support

May 2006David J. Ritchie:CD-doc-800-v1 31 D0 computing systems MBps 100+ MBps 400+ MBps “data-logger” “farm” “central-analysis” Enstore Mass Storage System “linux-analysis -clusters” “linux-build-cluster” “clueD0” ~100 desktops “d0-test” and “sam-cluster”

May 2006David J. Ritchie:CD-doc-800-v1 32 No Data Center is an Island Run II -- D0 planned for Regional Analysis Centers (RAC’s) since 1998: –Distributed analysis around the globe (arbitrary imagined distribution) UO UA CINVESTAV Rice FSU LTU UTA

May 2006David J. Ritchie:CD-doc-800-v1 33 And for the LHC in 2007 the CMS Experiment - example of a global community grid Germany Taiwan UK Italy Data & jobs moving locally, regionally & globally within CMS grid. Transparently across grid boundaries from campus to the world. Florida CERN Caltech Wisconsin UCSD France Purdue MIT UNL

May 2006David J. Ritchie:CD-doc-800-v1 34 US CMS Computing Fermilab is the host lab of U.S. CMS experiment which will begin taking data at CERN in Geneva, Switzerland in ~2007 Fermilab hosts the project management for the U.S. CMS Software and Computing Program in DOE U.S. Physicists will participate in this research.

May 2006David J. Ritchie:CD-doc-800-v1 35 The US CMS center at FNAL transfers data to 39 sites worldwide in CMS global Xfer challenge. Peak Xfer rates of ~5Gbps are reached. CMS Data Xfer FNAL to World

May 2006David J. Ritchie:CD-doc-800-v1 36 ~45 Sites Working together as a Coherent Computing Facility ~20 Research Groups Sharing the Sites to Run Compute Intensive Jobes 2500 jobs

May 2006David J. Ritchie:CD-doc-800-v1 37 Collaborations across Continents and across Disciplines coarse-grained particle ion channel simulator based on the Boltzmann Transport Monte Carlo methodology.

May 2006David J. Ritchie:CD-doc-800-v1 38 The Grid is for Everyone. How will it be used?

May 2006David J. Ritchie:CD-doc-800-v1 39 Acknowledgements As always in a laboratory, one gains from the work of one’s colleagues. This talk is no exception. I would especially like to complain about Ruth Pordes who was meant to give this talk!.