Panel Abstractions for Large-Scale Distributed Systems Henri Bal Vrije Universiteit Amsterdam

Background - Projects ● Virtual Laboratory for e-Science (VL-e) ● 40 M€ Dutch project ● Generic application support for e-Science ● Scientific workflow applications ● Ibis: grid programming environment for distributed supercomputing applications

Background – Infrastructure: DAS-3 Computer Science grid WAN: Gb/s Optical Private Network

● Performance & scalability ● Heterogeneous ● Low-level & changing programming interfaces Why are distributed applications difficult to program & deploy? ● Connectivity issues ● Fault tolerance ● Malleability Wide-Area Grid SystemsUser !

Common (scalable) patterns Programming –Master-worker => divide-and-conquer Image analysis, N-body, SAT, gene seq, grammar learning, …. –Asynchronous high-throughput communication Model checking, search applications –Mixed task- and data-parallelism Multimedia content analysis Deployment (OS-like functionality) –File I/O, resource allocation, job submission, …

Abstractions we use ● Programming ● Divide-and-conquer (Satin) ● Fault-tolerant, malleable ● IPL: Java-centric grid communication layer ● Run-anywhere, connectivity ● Deployment ● JavaGAT, SAGA (hide low-level APIs) Ibis

Abstractions we need ● Transparent grid libraries ● Domain-specific patterns ● E.g. imaging operations ● More scheduling functionality ● Give me N clusters with P processors at time T ● Map application onto heterogeneous grid (containing clusters, Cells, GPUs)

● Large-scale e-Science infrastructures ● Computing + storage + networking ● High-bandwidth, flexible optical WANs ● Supporting large-scale systems ● Exploit locality in hierarchical systems ● Latency-tolerant algorithms & systems ● Bandwidth allocation (light paths) ● Adapt network topology to application: alloc(80-Gb/s, A, B) ● Co-allocate CPU, data, network Emerging distributed infrastructures (1)

● Very large scale: mobile devices ● Smart phones: ubiquitous, location-aware ● Distributed smart phone applications ● Social networking, crisis management ● Integrated with grid backbone ● Supporting such applications ● Location-aware communication (geo-cast) ● Mobile ad hoc networks ● Dealing with resource constraints Emerging distributed infrastructures (2)