1. University at Buffalo The State University of New York Russ Miller, Director Center for Computational Research Supercomputing and Visualization “Top 10 Worldwide Supercomputing Center” - www.gapcon.com

2. University at BuffaloThe State University of New York CCR Center for Computational Research Outline Pervasive Computing Computer Trends Definition of Supercomputer Overview of Center for Computational Research Sample CCR Projects Center of Excellence in Bioinformatics H.S. Workshop

3. University at BuffaloThe State University of New York CCR Center for Computational Research Take-Home Message Computers play an important role in your life Currently ~10 processors per person Working with computers can be fun and exciting

4. University at BuffaloThe State University of New York CCR Center for Computational Research Introduction Computers play an important role in your life Working with computers can be fun There are many careers involving computers even if you are not a science or engineering major

5. University at BuffaloThe State University of New York CCR Center for Computational Research Computers Touch Every Aspect of Our Life….

6. University at BuffaloThe State University of New York CCR Center for Computational Research …. including entertainment

7. University at BuffaloThe State University of New York CCR Center for Computational Research Computers are used in Many Professions Science and Engineering  Physics, Chemistry, Biology  Aerospace, Mechanical, Civil, Environmental Architecture  Building and Bridge Design Computer Animation  Cartoons, Movies, Advertising  Games (Playstation, Nintendo, PC games, etc) Graphic Arts/Design Computer programmers

8. University at BuffaloThe State University of New York CCR Center for Computational Research It’s the computer’s brain - it’s the main processor CPU stands for Central Processing Unit What is a CPU?

9. University at BuffaloThe State University of New York CCR Center for Computational Research Gordon E. Moore Co-Founder of Intel Predicted (1965/75) that transistor density would double every 12/18 months Processing speed doubling every 18 mos. Disk storage doubling every 12 mos. Aggregate bandwidth doubling every 9 mos. Gordon E. Moore A computation that took 1 year to run on a PC in 1985 would only take 5 mins to run on a PC today! A computation that runs in 2 hours on a PC today would have taken 24 years to run on a PC in 1985!

10. University at BuffaloThe State University of New York CCR Center for Computational Research A computer that contains more than 1 processor (CPU) Why are they used? To solve problems faster than they could be solved using only 1 processor What is a Parallel Computer?

11. University at BuffaloThe State University of New York CCR Center for Computational Research Parallel Computing Architectures P P PPP P PP M Shared Memory Distributed Memory Distributed-Shared Memory

12. University at BuffaloThe State University of New York CCR Center for Computational Research What is a (Beowulf) Cluster? Industry Standard Hardware and Software  PC-Based Components (Intel or AMD)  Ethernet or Myrinet  Linux, PBS, MPI  “Commodity Off-The-Shelf” (COTS) Operates as a Single System Rivals Performance of Traditional Supercomputer at a Fraction of the Price Thomas Sterling Caltech

13. University at BuffaloThe State University of New York CCR Center for Computational Research What is a Supercomputer? Fastest computers at any point in time Used to solve large and complex problems Machines 1000 times faster than a PC Machines 10 times slower than what you need to solve the most challenging problems Seymour Cray 1925-1996 “Seymour Cray is the Thomas Edison of the supercomputing industry” - Larry L. Smarr Cray1 - 1976

14. University at BuffaloThe State University of New York CCR Center for Computational Research If you wanted to know what the weather will be like tomorrow, you could... Solve the problem at home on your PC and wait one month to get the answer or Solve the problem on a supercomputer and have the answer in one hour! Example

15. University at BuffaloThe State University of New York CCR Center for Computational Research Fastest Computers YearMachProcsGFlopsYearMachProcsGFlops 1976Cray 110.11993Cray T3D 1024152 1982Cray X-MP 40.91994Fujitsu VPP 140236 1986Cray 2421996Hitachi SR2 2048368 1989Cray Y- MP 82.71997Intel ASCI-R 91521830 1989TMC CM-2 8192281999SGI ASCI-BM 61443072 1992TMC CM-5 10241312000IBM ASCI-W 819212,288 2002NEC E.S. 512040,960 A 1-year calc in 1980 = 5.4 sec today A 1990 HPC = a laptop today

16. University at BuffaloThe State University of New York CCR Center for Computational Research Earth Simulator in Japan (NEC Vector Supercomputer)

17. University at BuffaloThe State University of New York CCR Center for Computational Research Earth Simulator 40TFlops Peak Homogeneous, Centralized, Proprietary, Vector Expensive! CFD-Weather, Climate, Earthquake 640 NEC SX/6 Nodes (5120 CPUs) Footprint = 4 tennis courts $6M/year in power

18. University at BuffaloThe State University of New York CCR Center for Computational Research High-Performance Computing and High-End Visualization  70 (40+ active) Research Groups in 27 Depts  13 Local Companies  10 Local Institutions  External Funds: $108M  Vendor Contributions: $41M Deliverables  350 Publications and Presentations  Hardware, Software, Algorithms, etc Training  Workshops  Courses  Degree Programs

19. University at BuffaloThe State University of New York CCR Center for Computational Research SGI Origin3800  64 Processors (400 MHz)  32 GB RAM; 400 GB Disk IBM RS/6000 SP  78 Processors  26 GB RAM; 640 GB Disk Sun Microsystems Cluster  48 Sun Ultra 5s (333MHz)  16 Dual Sunblades (750MHz)  30 GB RAM, Myrinet SGI Intel Linux Cluster  150 PIII Processors (1 GHz)  75 GB RAM, 2.5 TB Disk Storage Apex Bioinformatics System  Sun V880 (3), 6800, 280R (2), PIIIs  Sun 3960: 7 TB Disk Storage HP/Compaq SAN (3/2003)  25 TB Disk; 250 TB Tape Computational Resources Dell Linux Cluster - #22 in World  600 P4 Processors (2.4 GHz)  600 GB RAM; 40 TB Disk Dell Linux Cluster - #187 in World  4036 Processors (PIII 1.2 GHz)  2TB RAM; 160TB Disk; 16TB RD  Private Use

20. University at BuffaloThe State University of New York CCR Center for Computational Research Sample Computational Research Computational Chemistry (King, Kofke, Coppens, Furlani, Tilson, Lund, Swihart, Ruckenstein, Garvey)  Algorithm development & simulations Groundwater Flow Modeling (Rabideau, Jankovic, Becker, Flewelling)  Predict contaminant flow in groundwater & possible migration into streams and lakes Geophysical Mass Flows (Patra, Sheridan, Pitman, Bursik, Jones, Winer)  Study of geophysical mass flows for risk assessment of lava flows and mudslides Bioinformatics (Zhou, Miller, Hu, Szyperski – NIH Consortium, HWI)  Protein Folding: computer simulations to understand the 3D structure of proteins  Structural Biology; Pharmacology Computational Fluid Dynamics (Madnia, DesJardin, Lordi, Taulbee)  Modeling turbulent flows and combustion to improve design of chemical reactors, turbine engines, and airplanes Physics (Jones, Sen)  Many-body phenomena in condensed matter physics Chemical Reactions (Mountziaris) Molecular Simulation (Errington)

21. University at BuffaloThe State University of New York CCR Center for Computational Research Visualization Resources Fakespace ImmersaDesk R2  Portable 3D Device Tiled-Display Wall  20 NEC projectors: 15.7M pixels  Screen is 11’  7’  Dell PCs with Myrinet2000 Access Grid Node  Group-to-Group Communication  Commodity components SGI Reality Center 3300W  Dual Barco’s on 8’  4’ screen VREX VR-4200 Stereo Imaging Projector  Portable projector works with PC

22. University at BuffaloThe State University of New York CCR Center for Computational Research Computational Science (Patra, Sheridan, Becker, Flewelling, Baker, Miller, Pitman)  Simulation and modeling Urban Visualization and Simulation (CCR)  Public projects involving urban planning Medical Imaging (Hoffmann, Bakshi, Glick, Miletich, Baker)  Tools for pre-operative planning; predictive disease analysis Geographic Information Systems (CCR, Bisantz, Llinas, Kesavadas, Green)  Parallel data sourcing software Historical Reenactments (Paley, Kesavadas, More)  Faithful representations of previously existing scenarios Multimedia Presentations (Anstey, Pape)  Networked, interactive, 3D activities Sample Visualization Areas

23. University at BuffaloThe State University of New York CCR Center for Computational Research Groundwater Flow Modeling Regional-scale modeling of groundwater flow and contaminant transport (Great Lakes Region) Ability to include all hydrogeologic features as independent objects

24. University at BuffaloThe State University of New York CCR Center for Computational Research Groundwater Flow Modeling Regional-scale modeling of groundwater flow and contaminant transport (Great Lakes Region) Ability to include all hydrogeologic features as independent objects Current work is based on Analytic Element Method Key features:  High precision  Highly parallel  Object-oriented programming  Intelligent user interface  GIS facilitates large-scale regional applications Utilized 10,661 CPU days (32 CPU years) of computing in past year on CCR’s commodity clusters

25. University at BuffaloThe State University of New York CCR Center for Computational Research Computational Geophysical Mass Flows

26. University at BuffaloThe State University of New York CCR Center for Computational Research Risk Mitigation Integrate information from several sources  Simulation results  Remote sensing  GIS data Develop realistic 3D models of geophysical mass flows Present information at user appropriate resolutions

27. University at BuffaloThe State University of New York CCR Center for Computational Research Protein Dynamics Dynamics of Hemoglobin (Example) 50 Days of Processing on 16 Processors (800 CPU Days) Key  White – Heme Groups  Red – Phe97  Red – Oxygen (in the subunit at bottom)  Green – His 69 and 101  Blue – Tyr 72  Cyan (Ball) – Water Molecules  Yellow – Helix E/F Interest  Flip of the Phe97 ring at top  Water movement around Phe97  Heme-heme relative movement

28. University at BuffaloThe State University of New York CCR Center for Computational Research Protein Folding Ability of proteins to perform biological function is attributed to their 3-D structure. Protein folding problem refers to the challenge of predicting 3-D structure from amino-acid sequence. Solving the protein folding problem will impact drug design.

29. University at BuffaloThe State University of New York CCR Center for Computational Research 3D Medical Visualization App Collaboration with Children’s Hospital  Leading miniature access surgery center Application reads data output from a CT Scan Visualize multiple surfaces and volumes Export images, movies or CAD representation of model

30. University at BuffaloThe State University of New York CCR Center for Computational Research Multiple Sclerosis Project Collaboration with Buffalo Neuroimaging Analysis Center (BNAC)  Developers of Avonex, drug of choice for treatment of MS MS Project examines patients and compares scans to healthy volunteers

31. University at BuffaloThe State University of New York CCR Center for Computational Research Multiple Sclerosis Project Compare caudate nuclei between MS patients and healthy controls Looking for size as well as structure changes  Localized deformities  Spacing between halves Able to see correlation between disease progression and physical structure changes

32. University at BuffaloThe State University of New York CCR Center for Computational Research StreetScenes ® StreetScenes ® Demo StreetScenes ® StreetScenes ® is a Virtual Reality (VR) software solution for 3D visualization of surface traffic 3D model of proposed soccer stadium in Rochester StreetScenes ® Used StreetScenes ® to import output file from Synchro traffic simulation

33. University at BuffaloThe State University of New York CCR Center for Computational Research Peace Bridge Visualization International Crossing The Problem  75 year old bridge  3 lanes – poor capacity  Existing US plaza: small and poor design Proposed Options  Relocate US plaza  Build a 3-lane companion span, rehab existing bridge  Build a six lane signature span

34. University at BuffaloThe State University of New York CCR Center for Computational Research Buffalo Niagara Medical Campus Eight (8) Block radius in Downtown Buffalo Urban planning  Rapidly explore changes in building design/location  Public involvement - Meetings, CDs, Videos, etc

35. University at BuffaloThe State University of New York CCR Center for Computational Research Sample UB Synergies Media Study (Anstey, More)  Donation of PCs  Courses, Students, UB Grant  “Alive on the Grid”  NSF ITR Grant  AVID Software MAE (Kesavadas)  Gov. Pataki Visit  Peace Bridge (Early)  Commodity Projection Classics/MAE (Paley, Kesavadas)  Virtual Site Museum, BBC2 H.S. Bioinformatics Program (Pitman)  Verizon/Compaq Buffalo Neuroimaging Analysis Center (Bakshi)  MS Visualization Children’s Hospital (Glick)  CT 3D Viz Dent Neurologic Institute (Miletich)  PET Imaging Computer Science & Engineering  Crash Lab Library (Dilandro, Bertholf)  Darwin Martin House  James Joyce Novel On-Line Anthropology (Zubrow)  Solar Powered Cluster Management (Jain) & Math (Pitman)  Tops Friendly Markets  M&T Bank Access Grid Node (too numerous)  Campus-Wide Outreach for Conf.  Center for Americas  Human Rights (China/Sweden)

36. University at BuffaloThe State University of New York CCR Center for Computational Research Select WNY Synergies IBC Digital  Gov. Pataki Visit  Peace Bridge (Early & Current)  Buffalo-Niagara Medical Campus  Compute Cycles for Animation Bergmann Associates  Peace Bridge (Current)  NYS Thruway Toll Plaza Azar & More  Reenactment of 1901 Pan Am Exhibition  PHSCologram & Courses  Avid Digital Editing Niagara College  Start up  Peace Bridge (Current) Hauptman-Woodward Medical Research Institute  Computing  Collaboratory The Children’s Hospital of Buffalo  Medical Visualization Veridian  Battlespace Management

37. University at BuffaloThe State University of New York CCR Center for Computational Research Bioinformatics in Buffalo “This Center [of Excellence in Bioinformatics] will, through the University of Buffalo’s Center for Computational Research, create academic and industrial partnerships …” - NYS Gov. George S. Pataki, January 2001 Congressman ReynoldsSenator Clinton Gov. Pataki

38. University at BuffaloThe State University of New York CCR Center for Computational Research WNY Biomedical Advances PSA Test (screen for Prostate Cancer) Avonex: Interferon Treatment for Multiple Sclerosis Artificial Blood Nicorette Gum Fetal Viability Test Implantable Pacemaker Edible Vaccine for Hepatitis C Timed-Release Insulin Therapy Anti-Arrythmia Therapy  Tarantula venom Direct Methods Structure Determination  Listed on “Top Ten Algorithms of the 20 th Century”  Vancomycin  Gramacidin A High Throughput Crystallization Method: Patented NIH National Genomics Center: Northeast Consortium Howard Hughes Medical Institute: Center for Genomics & Proteomics

39. University at BuffaloThe State University of New York CCR Center for Computational Research Bioinformatics in Buffalo UB Center for Advanced Bioengineering & Biomedical Technologies  $1M/yr NYS  Med Tech for Product Dev & Commer. Center Disease Modeling & Therapy Discovery  UB, HWI, RPCI, Kaleida  $15.3M NYS  Software, device development, and drug therapies Buffalo Center of Excellence in Bioinformatics  UB, HWI, RPCI  $61M NYS  $10.6M Federal Government  $151 Corporate Funding  Significant Local Foundation Support

40. University at BuffaloThe State University of New York CCR Center for Computational Research Buffalo Center of Excellence in Bioinformatics Act as a research, development, education, and economic resource for industries based on bioinformatics, including information technology, biotech, and pharmaceuticals. Combine state-of-the-art computational facilities with high-throughput experimental facilities to enable the development of new medical treatments. Develop and exploit new algorithms for data acquisition, storage, management, and transmission.

41. University at BuffaloThe State University of New York CCR Center for Computational Research Academic Programs Bachelor’s & Master’s Program in Bioinformatics Related Disciplines  Chemical Biology  Computational Chemistry  Environmental Analysis (Sloan Support)  Medical Informatics (Sloan Support) Advanced Degrees under Development  Pharmacometrics, Biophotonics UB-HWI Department of Structural Biology Complementary Degrees  Canisius College  Niagara University

42. University at BuffaloThe State University of New York CCR Center for Computational Research Outreach

43. University at BuffaloThe State University of New York CCR Center for Computational Research 2003 H.S. Summer Workshop Bioinformatics June 30 – July 11 Perl Scripts Public Databases Filtering Results Graphics & Visualization Contact  Dr. Bruce Pitman (pitman@buffalo.edu)

44. University at BuffaloThe State University of New York CCR Center for Computational Research Lunch & Exhibition miller@buffalo.edu www.ccr.buffalo.edu H.S. Program pitman@buffalo.edu

45. University at BuffaloThe State University of New York CCR Center for Computational Research Contact Information www.ccr.buffalo.edu miller@buffalo.edu Photo Computer Generated

46. University at BuffaloThe State University of New York CCR Center for Computational Research Antibiotics & Supercomputers Result: New, better drugs in shorter time Vancomycin solved with SnB (UB/HWI)  SnB: “Top Algorithms of the Century”  “Antibiotic of Last Resort”  Original molecular structure required 5 months  (Re)solved in a single day on CCR’s supercomputers  Current Efforts: Grid, Collaboratory, Intelligent Learning

47. University at BuffaloThe State University of New York CCR Center for Computational Research Contact Information miller@buffalo.edu www.ccr.buffalo.edu

48. University at BuffaloThe State University of New York CCR Center for Computational Research Computing at UB Center for Computational Research (CCR)  Top Ten Supercomputing Center in USA  Parallel Supercomputers  High-End Visualization

49. University at BuffaloThe State University of New York CCR Center for Computational Research Power of UB’s Supercomputers Medicine Vancomycin  Important, powerful antibiotic  Original molecular structure required 5 months of computing to solve  We solved the structure at UB in a single day using our supercomputers! Result: New, better medicines in shorter time People Live Longer!

50. University at BuffaloThe State University of New York CCR Center for Computational Research CCR’s “Big Iron” SGI Origin3800 - 64 processors Pentium Cluster - 150 processorsIBM SP - 78 processors

51. University at BuffaloThe State University of New York CCR Center for Computational Research Shared Memory Analogy Data (paint) is shared Painter Paint Supply

52. University at BuffaloThe State University of New York CCR Center for Computational Research Federal IT R&D Investment is Inadequate Federal IT R&D should focus on long-term Research Priorities  Scalable Information Infrastructure  Software and Communications technology for scaling infor. infrastr.  Expand Next Generation Internet testbeds to foster development and deployment of enabling technologies  High-End Computing  Innovative computing technologies and architectures  Software for improving high-end computing  Acquisition of high-end computing to support science and engineering research  Expand Fed. High End Computing and Computation (HECC) program  Software  Socio-Economic and Workforce Impact PITAC Report (President’s Information Technology Advisory Committee)

53. University at BuffaloThe State University of New York CCR Center for Computational Research Parallelization GOAL: Reduce total execution time Why use multiple processors  Physical limits on single processor speed  Speed of Light  Cost  very fast single processors are too expensive  commodity processors are fast and inexpensive

54. University at BuffaloThe State University of New York CCR Center for Computational Research Parallel Architectures Distributed Memory Shared Memory Distributed Shared Memory

55. University at BuffaloThe State University of New York CCR Center for Computational Research Distributed Memory Memory is physically distributed Local memory directly accessible only by its processor memory CPU 1 CPU 2 CPU 3 Network

56. University at BuffaloThe State University of New York CCR Center for Computational Research Distributed Memory Analogy Each painter has “own” paint (data) supply Painter Paint Supply

57. University at BuffaloThe State University of New York CCR Center for Computational Research Message Passing Painter Paint Supply Each painter needs colors (data) “owned” by other 3 painters

58. University at BuffaloThe State University of New York CCR Center for Computational Research Shared Memory Memory is shared by all processors MEMORY CPU 1 CPU 2 CPU 3

59. University at BuffaloThe State University of New York CCR Center for Computational Research “Memory” Contention processors access the same memory location at the same time Limits scalability of shared memory computers

60. University at BuffaloThe State University of New York CCR Center for Computational Research Distributed - Shared Memory Multiple shared memory systems connected via a high speed link CPU 1 CPU 2 CPU 3 CPU 1 CPU 2 CPU 3 MEMORY Interconnection Network

61. University at BuffaloThe State University of New York CCR Center for Computational Research Shared vs Distributed Programming Distributed Memory  must use message passing  MPI, PVM, TCGMSG  scalable to a large number of processors  generally more difficult to program Shared Memory  can use compiler directives (OpenMP)  can also use message passing  easier to program  limited scalability Distributed - Shared Memory

62. University at BuffaloThe State University of New York CCR Center for Computational Research Sequential to Parallel Conversion Automatic parallelization  Compilers can do limited parallelization  Significant speedups require manual coding Manual parallelization  Actions  Analyze code - determine time consuming steps  Remove data dependencies  Restructure algorithm

63. University at BuffaloThe State University of New York CCR Center for Computational Research Parallel Algorithm Dev. Try to achieve load balance Be aware of Amdahl’s Law  Speedup is limited by sequential portion of code tBtB N t parallel = t A + t serial t parallel Speedup = tBtB N t serial t A + Speedup =

64. University at BuffaloThe State University of New York CCR Center for Computational Research Amdahl’s Law Example Serial Program  t A = 2 minutes, t B = 98 minutes  t serial = t A + t B = 100 minutes Parallel Implementation  Parallelize step B  t parallel = t A + t B /N Speedup  S = 100 min/ (2 min + 98 min/N) Observed Speedup N S ideal S obs 1 11 2 22 10 108 50 5025 100 10034 1000100048

65. University at BuffaloThe State University of New York CCR Center for Computational Research Maximum Speedup %P248163264 50%1.31.61.81.91.92.0 75%1.62.32.93.53.73.8 90%1.83.14.76.47.88.8 95%1.93.55.99.112.515.4 99%2.3.97.513.924.439.3 Number of Processors %P = Percent parallelism

66. University at BuffaloThe State University of New York CCR Center for Computational Research Load Balance Time to complete parallel region limited by last processor to finish

67. University at BuffaloThe State University of New York CCR Center for Computational Research Fine-Grained vs Coarse- Grained Parallism Granularity refers to the size of the parallel region

68. University at BuffaloThe State University of New York CCR Center for Computational Research Fine-Grained / Coarse-Grained Fine-grained parallelism  Can be implemented incrementally, one loop at a time  Doesn’t require deep knowledge of program structure  compiler directives (OpenMP) Coarse-grained parallelism  Parallelize more code - higher level  Requires deeper knowledge of program structure  May be easier to implement with message passing

69. University at BuffaloThe State University of New York CCR Center for Computational Research Performance Falloff Program efficiency falls off as number of processors increases  Amdahl’s Law  Too many processors/not enough work  Poor algorithm design  load balance  data dependencies

70. University at BuffaloThe State University of New York CCR Center for Computational Research Personnel Leadership  Director & Associate Director Clerical  Office Manager; Budget; Receptionist; Computational Scientists  Comptutational Chemistry, Comptutational Physics  Bioinformatics, Scientific Visualization Programmers  Bioinformatics, Database, MultiMedia System Administrators  Sysadmins (5), SAN admin, Web/Help Desk Soft Money  Post-Docs (4)