Do’s and Don’ts of Building Grand Challenge Application Teams Ed Seidel Max-Planck-Institut für Gravitationsphysik (Albert Einstein Institute) NCSA, U of Illinois Co-Chair, GGF Applications Working Group Things I wish I could do with (or to) my Grand Challenge Projects…

My experiences: what can we learn? Six large scale projects NSF BH Grand Challenge NASA NS Grand Challenge NSF KDI Astrophysical Simulation Collaboratory Project EU Astrophysics Network (5th Framework Program) EU GridLab Project German DFN-Verein TiKSL/GriKSL Projects They are largely about (MultiDisciplinary) Community Building Somewhat overlooked, even by the PIs Examples of Future of Science & Engineering Require Large Scale Simulations, beyond reach of any machine Require Large Geo-distributed Cross-Disciplinary Collaborations Require Grid Technologies, but not yet using them!

Black Hole Grand Challenge Alliance ($4.5M, Original NSF GC Program) Background/Goals: 8 US Institutions, Solve problem of colliding black holes (try…) Bring together computer and physical scientists to solve problem on HPC hardware Develop a community Problems: Difficult community: money brought them together No pre-existing Infrastructure for Computational Collaborations CS and Physicists had trouble together Not enough cycles: where was the TFlop? Successes: Community came closer together (though somewhat scarred) Learned what we needed: Computational tools like Cactus, GrACE, came out of lessons learned Bandwidth needs: very low , remote login, web pages (very new!)

Neutron Star Grand Challenge ($1.4M, NASA Round 2) Background/Goals 5 Institutions (Develop infrastructure to) solve problem of colliding NSs Issues: Personality clashes Infrastructure (Cactus, GrACE under development) Computer Scientist in Charge of Science Project Project not seen as very successful in astro community: “Where’s the physics?” But: Project excessively performance milestone based –Q: “How can you cut our postdoc funding?? Must do Physics!” –A: “If you achieve 100GF we’re pretty sure you’ll find a way to do some physics…” Successful, but mixed, perhaps even did some damage… Bandwidth needs: minimal, but: could have been much more (remote Viz, etc, too hard for people, but should use!)

Astrophysics Simulation Collaboratory ($2.2M NSF KDI Program) Background/Goals 4 US Institutions + German Projects Basically a Technology program with application Driver –Portal, AMR, NS collapse problem Issues: Technologist or Scientist in charge? Deployment of Technologies Difficult Community Acceptance –Scientists need this, but don’t get it –Criticized for using word “Collaboratory” in NRAC proposal! Bandwidth Needs Should be much higher than they are! Catch 22 again…

German TiKSL/GriKSL Projects (2.5MDM DFN-Verein) Background/Goals Develop remote Viz/steering/collaborative simulation, distributed computing capabilities Successes Wonderful technology; all works! All research/dev steered by application needs Incredible Matching Effort: –Embedded in physics research group. Have a dozen physics postdocs/students in Potsdam, forced to use the stuff! –Leverage: Tightly coupled to ASC Project –Visitor Program supplements effort considerably Problems Far too little travel money! I have to supplement to make it work! Technologies never quite mature enough for easy adoption by community Even in my group, people very reluctant to waste time Bandwidth Needs Aimed to drive high speed networking, Gbit networks easily pushed (Shalf Gigabit Challenge Award…)

EU Astrophysics Network (€1.5M, EU 5th Framework Programme) Background/Goals 10 EU Institutions, 3 years Solve same problems, build on previous works Build/train community Problems No EU Computing centers, policies, etc Level of computational expertise in apps groups very low compared to US (OK: train them right from beginning!) Cultural differences much bigger Successes/Advantages Draw on/integrate individual strengths: no forced march People see scientific advantage of working together Existing Collaborative Infrastructure! Leveraging all the above: Cactus, ASC, etc… Bandwidth Needs: growing, but people make due… Want conferencing for collabs, training Could use grid technologies for science: Catch 22: bandwidth not there, so don’t push…

GridLab Project and Others like it (€5M, EU IST Programme) Background/Goals Co-develop innovative Grid infrastructure and applications/experiments: –Cactus, Triana, Grav. Wave Astro, others –Bring others in later Use other Apps projects for testing Grass roots effort: Egrid testbed came first Success (Not started yet): Created excited community Brussels agreed to send money to US!!! Problems: Excessive regulation, control by Brussels Hard to find experienced people Lack of applications No money for conferencing facilities, coordinating with other projects… Bandwidth needs: can be very large!!

Summary of Issues The Obvious: Basic “Application-Driving-Technology” Model is Correct Need, must encourage application teams for high bandwidth grid apps to drive program Chair of Apps group of GGF: “Small, small fraction of groups using the grid!!!” Need programs like this to force centers to provide capabilities How to Achieve Real Collaboration/Communication in such projects? Basic Principle: People do NOT naturally communicate, projects always confused Push, encourage, fund collaborative technologies (makes a huge difference) –VTCs, Full scale AG nodes, Smaller scale way to connect –Better if embedded in real groups: not just developed in void! How to do this? –Don’t forget obvious: time difference can be significant hindrance, sometimes advantage Adequate, and GENEROUS travel allowances: many projects strangled Explicitly ask proposers to explain how they will work coherently, how they will use/make used the technology

How to achieve real leverage within projects? Real Progress requires real effort, real people People typically too busy to do their jobs Don’t hire 10 people at 10% each! Visitor money very important (single most important in my experience) Exchanges between project members at different sites Significant Matching/Embedding can be good sign: (e.g. my TiKSL project…) Need to encourage strong PIs Apps teams headed by apps people CS teams headed by CS people Need good standing in community, good social skills

Achieving Leverage Between Projects Encourage, provide specific mechanism for clustering/linking App/Infrastructure balance in single project is good, BUT Explicit linkage in between mostly tech and mostly apps good, too –People may work better, focus better this way –Technology projects without apps groups sometime have no rudder, go awry –Must couple them, either by PI design or Agency “encouragement” –Provide money for exchanges, travel, joint meetings, etc Cross Grand Challenge/eScience/Infrastructure Workshops –Generally people don’t know what each other do, or how to use it Encourage collaboration with schools of sociology, psychiatry… Interagency links should be encouraged Pair up with EU, Asian agencies: this is a global world! Get Centers closely involved in such projects Somehow encourage projects to force centers to provide needed services Dedicate person at centers to consult/aid/watch over, as well as resources: bandwidth, disk, CPU, etc…

More General Educational Mission Major emphasis of EU Network, good idea Appalling amount of ignorance/lack of imagination out there in apps! 2 Groups to educate: –Apps community –New Generation of Apps people –Struggling to find their place: it is here. Even within projects, people do not try to use the technology!! Must provide adequate support for prototype ---> production Testing Documentation, support Old NCSA Problem: lots of hardware, not as many people to develop/use/support it (and it is better than other places!)

Final Suggestions/Thoughts Reasonable milestones for focus, but not smothering requirements Good Balance between engineering approach (large coordinated machine) and individual research freedom: encourage people to make sure they fit together Allow adequate administrative support Encourage people to be ambitious Allow risky proposals through! Get participants to think Big, understand their responsibilities to puch communities forward