Cooperative Computing for Data Intensive Science Douglas Thain University of Notre Dame NSF Bridges to Engineering 2020 Conference 12 March 2008

What is Cooperative Computing? By combining our computing and storage resources together, we can attack problems larger than we could alone. I can use your computer when it is idle, and vice versa. (Most computers are idle about 90 percent of the day.) Also known as… –Grid computing, distributed computing, metacomputing, volunteer computing, etc…

Who Needs Coop Computing? Many fields of study rely on simulation and data processing to conduct science. –Physics, chemistry, biology, engineering, finance, sociology, computer science. More Computing == Better Results –NOT High Performance: Speed up one program. –High Throughput: Produce as many results as possible over the next day / week / year.

Cooperative Computing Lab We design and build distributed systems that helps people to attack BIG problems. Work directly with end users to make sure that our solutions affect the real world. Operate a modest computing system as both a production service and a research testbed. –Currently about 500 cpus and 300 disks. CS Research challenges: scalability, robustness, usability, debugging, and performance.

What Makes this Challenging? The Programming Model –I want to process 10 TB of data on 100 machines, then distribute it across 20 disks, then view the best results on my workstation. Fault Tolerance –Something is always broken! Performance Robustness –There is always one slowpoke. Debugging –My job runs correctly here but not there...!?

An Example Collaboration: Biometrics Research and Distributed Systems

A Common Pattern in Biometrics F Sample Workload: 4000 images 256KB each 1s per F 185 CPU-days Future Workload: images 1MB each 0.1s per F 4166 CPU-days

Non-Expert User Using 500 CPUs Try 1: Each F is a batch job. Failure: Dispatch latency >> F runtime. HN CPU FFFF F Try 2: Each row is a batch job. Failure: Too many small ops on FS. HN CPU FFFF F F F F F F F F F F F F F F F F Try 3: Bundle all files into one package. Failure: Everyone loads 1GB at once. HN CPU FFFF F F F F F F F F F F F F F F F F Try 4: User gives up and attempts to solve an easier or smaller problem.

All Pairs Production System Web Portal 300 active storage units 500 CPUs, 40TB disk FGH S T All-Pairs Engine 2 - AllPairs(F,S) FFF FFF 3 - O(log n) distribution by spanning tree. 6 - Return result matrix to user. 1 - Upload F and S into web portal. 5 - Collect and assemble results. 4 – Choose optimal partitioning and submit batch jobs.

Some Results on Real Workload

Collaboration is Where the Interesting Problems Are! (Cooperative Computing Provides the Resources)

What Makes a Collaboration Work? Like a marriage? (old joke.) First, a show of commitment: go after some low hanging fruit, and publish it. A proposal for funding only succeeds if you have already started working together. Need very concrete goals: your partner may not share your idea of an interesting tangent. Students sometimes need a big push to leave their comfort zone and work together.

For more information… Douglas Thain Cooperative Computing Lab – Apply for Summer 2008 REU: – Supported by NSF Grants CCF and CNS