csinparallel.org Surviving in the Wild: Teaching and Training for the Parallel Future Dick Brown St. Olaf College SPLASH Educators’ and Trainers Symposium October 24, 2011

csinparallel.org Overview Review of the need for parallelism Strategies we need – What to teach – How to teach it – How to get it taught Surviving in the wild of Parallelism With interludes, to be announced…

csinparallel.org The need for parallelism Question: Why do we need parallelism at the programming level? – Hint: It’s not “because it’s there” (… although desktop applications do tend to find ways to use ever greater power per dollar) Note: I will use parallel as a generic for concurrent, parallel, distributed, cloud, accelerator (e.g., GPGPU), etc.

csinparallel.org The need for parallelism Answer: Scale

csinparallel.org The need for parallelism Answer: Scale Cloud applications + = 30,000,000,000,000,000

csinparallel.org The need for parallelism Answer: Scale Cloud applications Scientific applications: Particle-level simulation of turbulance is exascale Can’t achieve exascale performance without many cores (Berkeley “walls”), accelerators

csinparallel.org Challenges for industry Technology: – Heterogeneous computing (CPU + accelerators) – Sophisticated “on the fly” runtime systems – “Wall” of memory hierarchy vs. on-chip access

csinparallel.org Challenges for industry Technology: – Heterogeneous computing (CPU + accelerators) – Sophisticated “on the fly” runtime systems – “Wall” of memory hierarchy vs. on-chip access Examples – AMD Fusion System Architecture: CPU+GPU – Intel MIC (Many Integrated Cores): 50+ CPUs on a chip, as a cluster-like accelerator

csinparallel.org Challenges for industry Technology: – Heterogeneous computing (CPU + accelerators) – Sophisticated “on the fly” runtime systems – “Wall” of memory hierarchy vs. on-chip access Programming models: – Higher level; more “human-centric” – Scalable – Versatile

csinparallel.org Challenges for Education/Training We want to prepare our students for what they’ll need, before the demand explodes, but What are the enduring principles? Technologies, (hence) tools change rapidly! (Educators:) Change the curriculum???

csinparallel.org A wild ecosystem Industry/Academia

csinparallel.org A wild ecosystem Industry/Academia Learning curve/Rapid change

csinparallel.org A wild ecosystem Industry/Academia Learning curve/Rapid change Principles/Practices

csinparallel.org A wild ecosystem Industry/Academia Learning curve/Rapid change Principles/Practices Teaching/Research – New research discoveries in technology and programming models need to get into the curriculum yesterday

csinparallel.org A wild ecosystem Industry/Academia Learning curve/Rapid change Principles/Practices Teaching/Research We need strategy! And, it’s coming fast! – Took OOPSLA 20 years to become SPLASH… We can’t wait that long

csinparallel.org Strategies to be found What to teach How to teach it How to get it taught ITiCSE 2010 working group, Strategies for Preparing Computer Science Students for the Multicore World

csinparallel.org What to teach Parallel computing has a head start: ACM/IEEE Curriculum ’91 – 3 required hours on parallel algorithms – 3 required hours on distributed and parallel programming language constructs, with hands- on practice Ada, Concurrent Pascal, Occam, or Parlog (Was not universally embraced…)

csinparallel.org What to teach Parallel computing has a head start: ACM/IEEE Curriculum ’91 – 3 required hours on parallel algorithms – 3 required hours on distributed and parallel programming language constructs, with hands-on practice But, ten years later… ACM/IEEE Curriculum ’01 – 0 required hours of parallel algorithms – No mention of programming language constructs – Replaced by: “net-centric computing,” etc.

NSF/TCPP Curriculum Standards Initiative in Parallel and Distributed Computing – Core Topics for Undergraduates Sushil K. Prasad, IEEE TCPP Chair, Georgia State University Richard LeBlanc, Seattle University, ACM Education Council Charles Weems, University of Massachusetts, Amherst Alan Sussman, University of Maryland Arnold Rosenberg, Northeastern and Colorado State University Andrew Lumsdaine, Indiana University Curriculum Initiative Website: linked through tcpp.computer.orgtcpp.computer.org

Who are we? Chtchelkanova, Almadena - NSF Dehne, Frank - University of Carleton, Canada Gouda, Mohamed - University of Texas, Austin, NSF Gupta, Anshul - lBM T.J. Watson Research Center JaJa, Joseph - University of Maryland Kant, Krishna - NSF, Intel La Salle, Anita - NSF LeBlanc, Richard, University of Seattle Lumsdaine, Andrew - Indiana University Padua, David- University of Illinois at Urbana-Champaign Parashar, Manish- Rutgers, NSF Prasad, Sushil- Georgia State University Prasanna, Viktor- University of Southern California Robert, Yves- INRIA, France Rosenberg, Arnold- Colorado State University Sahni, Sartaj- University of Florida Shirazi, Behrooz- Washington State University Sussman, Alan - University of Maryland Weems, Chip, University of Massachussets Wu, Jie - Temple University

Specifying Curriculum Recommendations – NSF/TCPP Approach Identify topics in four existing areas: architecture, algorithms, programming, and cross-cutting topics For each topic, recommend – Bloom level – “Hours” of coverage – Suggested learning outcome – Possible core course for coverage Focus: First two years

Bloom Levels Use first three levels for recommended core topics K= Know the term/recall definition (basic literacy) C = Comprehend so as to paraphrase/illustrate A = Apply it in some way (requires operational command) N = Not in core (but may be useful in elective or advanced courses)

Example Parallel and Distributed Models and Complexity – Costs of computation Algorithms Topics Bloom #CourseLearning Outcome Algorithmic problems The important thing here is to emphasize the parallel/distributed aspects of the topic Communication broadcast C/A Data Struc/Algo represents method of exchanging information - one-to-all broadcast (by recursive doubling) multicast K/C Data Struc/Algo Illustrate macro-communications on rings, 2D- grids and trees scatter/gather C/AData Structures/Algorithms gossip N Not in core Asynchrony KCS2 asynchrony as exhibited on a distributed platform, existence of race conditions Synchronization K CS2, Data Struc/Algo aware of methods of controlling race condition, Sorting C CS2, Data Struc/Algoparallel merge sort, Selection K CS2, Data Struc/Algo min/max, know that selection can be accomplished by sorting K: know term C: paraphrase/illustrate A: apply

Programming Assume some conventional (sequential) programming experience Key is to introduce parallel programming early to students Four overall areas – Paradigms – By target machine model and by control statements – Notations – language/library constructs – Correctness – concurrency control – Performance – for different machine classes

Parallel Programming Paradigms (Selections) By target machine model – Shared memory (Bloom classification A) – Distributed memory (C) – Client/server (C) – Hybrid (K) – e.g., CUDA for CPU/GPU By control statements – Task/thread spawning (A) – Parallel Loop (C)

How to Read the Proposal Oh no! Not another class to squeeze into our curriculum!

Oh yes! Not another class to squeeze into your curriculum! How to Read the Proposal

Oh yes! Not another class to squeeze into your curriculum! Draft curriculum released Dec 2010 (tcpp.computer.org) How to Read the Proposal

csinparallel.org Enduring skills? Since the tool set is subject to change at any time, how much investment in those skills? – Many parallel languages and features have come and gone Need hands-on experience for effective learning. Anything may suddenly emerge as important – Python as a prototyping language for HPC

csinparallel.org A candidate for addition Patterns of parallel programming

csinparallel.org Patterns, a candidate for addition Background – “Gang of Four” book

csinparallel.org Patterns, a candidate for addition Background – “Gang of Four” book, 1994 – Doug Lea, Concurrent programming in Java: Design principles and patterns, 1999 – Tim Mattson, et al, Patterns for Parallel Programming, 2005 – Kurt Kreutzer and Berkeley Parlab, the Dwarves Motifs – Kreutzer and Mattson, OPL (parlab.eecs.berkeley.edu/wiki/patterns)

csinparallel.org Patterns, a candidate for addition Why patterns? They capture reusable units of expert problem-solving strategy Thus, they provide novices with a way to acquire expertise Many are supported by tools – Loop parallel, Message passing, Map-reduce, …

csinparallel.org How to teach it Agree with NSF/TCPP Initiative, that parallelism should be taught early and often – Scratch team kept concurrent scripts, because users “not surprised that a sprite can do several things at once” – Lessons of Vishkin’s “Peanut Butter Sandwich” exercise

csinparallel.org CSinParallel project Add parallelism early and often at all levels Incremental, flexible approach via modules Sharing within our community

csinparallel.org CSinParallel project Modular Approach – Short units (1-3 days) – Identified learning objectives – Self-contained – Flexible for use in various courses and curricula Make software/libraries more accessible – Parallel Platform Packages, Resources Share, discuss and help as a community –

csinparallel.org CSinParallel project Some selected module topics Introductory: – Map-Reduce computing for CS1 using WebMapReduce – Concurrent access to data structures in Java or C++ – Multicore programming with Intel’s Manycore Testing Lab Intermediate: – Introduction to parallel computing concepts – Concurrency strategies in programming languages – Parallel sorting algorithms

csinparallel.org Module: WebMapReduce

csinparallel.org Module: MTL with OpenMP Intel’s Manycore Testing Lab Module – #pragma omp parallel for num_threads(threadct) \ shared (a, n, h, integral) private(i) – reduction(+: integral)

csinparallel.org CSinParallel We seek collaborators and contributors

csinparallel.org Patterns Methodology Kreutzer and Mattson OPL not only provides a catalog of patterns, but also a software problem-solving methodology

csinparallel.org Patterns Methodology Kreutzer and Mattson OPL not only provides a catalog of patterns, but also a software problem-solving methodology Purposes: – Education – Communication – Design

csinparallel.org How to get it taught Pressures on the professor – “Oh no! Not another course to squeeze…” So, take an incremental spiral approach (agreeing with NSF/TCPP) – Small changes in curriculum in many places – Revisit challenging issues – Students come to think of parallelism as natural part of computation – Spiral approach is pedagogically effective

csinparallel.org How to get it taught Incentives Microgrants: small (e.g., $1500) amounts for contributing first steps in teaching parallelism – Intel Academic Community (intel.com/AcademicCommunity) – Educational Alliance for a Parallel Future (eapf.org)

NSF/TCPP Initiative Early Adopter Program

How to obtain Early Adopter Status? 16 Early adopters chosen for Spring term Early adopters chosen for Fall term 2011 Next round of competition: Fall 2012; Deadline November 5, 2011 – NSF/Intel funded Stipend/Honorarium – Which course(s), topics, evaluation plan?

How to obtain Early Adopter Status? Instructors for – core CS/CS courses such as CS1/2, Systems, Data Structures and Algorithms – department-wide adoption preferred – elective courses such as Algorithms, Architecture, Programming Languages, Software Engg, etc. – introductory/advanced PDC course – dept chairs, dept curriculum committee members responsible

csinparallel.org How to get it taught Other supports needed Platform availability Support community Educational elements – Learning objectives, assessment tools, etc.

csinparallel.org Surviving in the wild ecosystem – Industry/Academia – Learning curve/Rapid change – Principles/Practices – Teaching/Research

csinparallel.org Surviving in the wild ecosystem – Industry/Academia – Learning curve/Rapid change – Principles/Practices – Teaching/Research Mine from the heritage of the past Incremental approach Spiral exposition Pattern-based methods

csinparallel.org An example (go-lang.org) Mine from the heritage of the past – Hoare’s CSP; CCS  Pi Calculus [teach/research] Incremental approach – Not far from C [academic/industry] Spiral exposition – Midway steps towards explicit threads, message passing [Learning curve/rapid change] Pattern-based methods – Message passing, Fork-join, channel as Parallel Queue¨ [Principles/Practice]

csinparallel.org Questions?