Conductor A Framework for Distributed, Type-checked Computing Matthew Kehrt

Distributed Computing Today, people are dealing with larger and larger computational problems Today, people are dealing with larger and larger computational problems

Distributed Computing Today, people are dealing with larger and larger computational problems Today, people are dealing with larger and larger computational problems Scientific computations often analyze enormous amounts of data Scientific computations often analyze enormous amounts of data Radio telescope data Radio telescope data

Distributed Computing Today, people are dealing with larger and larger computational problems Today, people are dealing with larger and larger computational problems Scientific computations often analyze enormous amounts of data Scientific computations often analyze enormous amounts of data Radio telescope data Radio telescope data Some problems have large search spaces Some problems have large search spaces Factoring Factoring Password cracking Password cracking

Distributed Computation A solution – split large computations across multiple computers A solution – split large computations across multiple computers

Distributed Computation A solution – split large computations across multiple computers A solution – split large computations across multiple computers Some projects already do this Some projects already do this (protein folding) (protein folding) (genetics) (genetics)

Distributed Computation A solution – split large computations across multiple computers A solution – split large computations across multiple computers Some projects already do this Some projects already do this (protein folding) (protein folding) (genetics) (genetics) However, these examples are all for very specific problems However, these examples are all for very specific problems

Grid Computing We want something more generic We want something more generic

Grid Computing We want something more generic We want something more generic A general framework for distributing computation A general framework for distributing computation A way of submitting work to a collection of machines A way of submitting work to a collection of machines

Grid Computing We want something more generic We want something more generic A general framework for distributing computation A general framework for distributing computation A way of submitting work to a collection of machines A way of submitting work to a collection of machines We want a peer-to-peer network to distribute code We want a peer-to-peer network to distribute code

Safety? We want to run code on remote hosts We want to run code on remote hosts

Safety? How do we guarantee code is benign? How do we guarantee code is benign? Running code in a safe VM is expensive Running code in a safe VM is expensive

Safety? We want to run code on remote hosts We want to run code on remote hosts How do we guarantee code is benign? How do we guarantee code is benign? Running code in a safe VM is expensive Running code in a safe VM is expensive Native code is faster Native code is faster

Safety? We want to run code on remote hosts We want to run code on remote hosts How do we guarantee our code is benign? How do we guarantee our code is benign? Running code in a safe VM is expensive Running code in a safe VM is expensive Native code is faster Native code is faster Can we do static checks for safety on code to run? Can we do static checks for safety on code to run?

TAL Type systems can make static guarantees about program behavior Type systems can make static guarantees about program behavior

TAL However, we want to send object code over the network However, we want to send object code over the network

TAL Type systems can make static guarantees about program behavior Type systems can make static guarantees about program behavior However, we want to send object code over the network However, we want to send object code over the network Reduces work that needs to be done at runtime Reduces work that needs to be done at runtime How about a typed assembly language which need be only linked? How about a typed assembly language which need be only linked?

TAL A good type system allows us to distribute guaranteed safe code A good type system allows us to distribute guaranteed safe code

TAL Assembly language means little work needs to be done to prepare distributed code for running. Assembly language means little work needs to be done to prepare distributed code for running.

The Conductor Each machine on the grid runs a Conductor Each machine on the grid runs a Conductor The Conductor The Conductor Maintains a queue of work to be done Maintains a queue of work to be done Locates peers through a gossip protocol Locates peers through a gossip protocol Queries known peers for others on the network Queries known peers for others on the network Downloads, checks and runs work Downloads, checks and runs work Returns results Returns results

The Conductor Allows for multiple safety policies (different type systems) selected by the user Allows for multiple safety policies (different type systems) selected by the user Publishes code and results in a distributed hashtable Publishes code and results in a distributed hashtable Caches results for quick recomputation Caches results for quick recomputation Is tolerant to machines joining and leaving the network Is tolerant to machines joining and leaving the network

Summer Goals A network running on machines at CMU A network running on machines at CMU

Summer Goals A network running on machines at CMU A network running on machines at CMU A larger network running on machines owned by the Pittsburgh public school system A larger network running on machines owned by the Pittsburgh public school system

Summer Goals A network running on machines at CMU A network running on machines at CMU A larger network running on machines owned by the Pittsburgh public school system A larger network running on machines owned by the Pittsburgh public school system Adding a new type system to the conductor (TAL Two, or TALT) Adding a new type system to the conductor (TAL Two, or TALT)

Summer Goals A network running on machines at CMU A network running on machines at CMU A larger network running on machines owned by the Pittsburgh public school system A larger network running on machines owned by the Pittsburgh public school system Adding a new type system to the conductor (TAL Two, or TALT) Adding a new type system to the conductor (TAL Two, or TALT) Ability to merge two grids Ability to merge two grids