PAGIS: An Architecture for Programming on the Grid Andrew Wendelborn Distributed & High Performance Computing Group Department of Computer Science, University of Adelaide Adelaide SA 5005, Australia December 2002

Background PAGIS –DHPC propject –Processing of satellite imagery –Started as a compositional process-oriented model for GIS applications –Now an architecture for building grid applications Outline –Base-level API and GUI –Reflective (metalevel) programming for grid applications Framework for defining grid behaviours –Adaptive software: safe reconfiguration –An adaptive resource broker

Principles High-level computational model –Programmers shouldn’t need to be grid experts –Reason about behaviour at an abstract level Simple, intuitive interface –Following compositional component-based paradigm Access to system –Programmer abstracts over low-level details … BUT –Wants to tune aspects of system critical to grid application –We provide this via a separate interface

Process Networks General, clearly defined semantics Separates meaning from implementation –Different strategies produce equivalent results Compositional Reconfigurable Highly applicable to grid programming Process1 Process3 Process2 Process4

Evaluator Client PAGIS overview GUI PN structure

Evaluator Scheduler Client

Evaluator GRAM - GRIS GIIS Scheduler Grid enabled Globus toolkit Distributed JVMs

Tuning and Behaviour We have seen how to build base-level application –How to fine-tune its implementation and behaviour Types of tuning –Replacing code for PN process –Allocation and re-allocation –Monitoring, and other non-application (meta) concerns “Actors” observe and influence behaviour –Human observation and intervention –Other software systems –Change how the computation occurs, not the result of the computation

A Reflective Approach Separate concerns into two programs –Functional concerns: baselevel program –Non-functional concerns: metalevel program (meta-behaviour) –Reflects separation in process network model Reify certain aspects of baselevel Metalevel programmer reflects upon selected aspects to control or customize aspects critical to application Software engineering for the grid

Method Reification Constructs an explicit representation of a method invocation at the metalevel Provides hooks to control and customize method invocation process We have designed a customization framework, Enigma ObjectAObjectB MetaAMetaB MI

Grid Behaviours Enigma gives us a model for describing and implementing interesting grid behaviours Locality –Migrating the computation and its parts Allocation and Adaptation –Computation meta-attributes to drive allocator –Periodically reassess allocation Performance Monitoring –Intercept invocations to monitor progress

PAGIS as a Grid Programming Model API provides baselevel interface to system PN provides semantics and safety of adaptation Enigma: hooks to observe / influence behaviour –Software engineering of adaptive applications PN System Implementation Enigma Metalevel Interface Process Network API

The System Allocation Manager Channel Service Process Service Distribution Tools Globus Plugin BrokerCOG MDS GRAM Gatekeeper Job Manager Worker JVM fork GRAM data RSL Client Worker LDAP query Client Plugin Java implementation Globus toolkit –Java CoG Coglets for peer-to-peer Java / Globus

Current Implementation Building a PN –Java; CoG; Globus toolkit –Operations for simple image processing Inspection of meta-level attributes Changing meta-level attributes –Influences allocation –May lead to migration Underway: integration with DARC

DARC provides data abstractions PAGIS provides process abstractions Host A Host B Host C Node DR TCP PAGIS and DARC

Client Evaluator GRAM - GRIS DARC GIIS Scheduler DARC provides data sources Powerful abstraction over Globus toolkit

Adaptive Resource Broker Project Testbed for different RB strategies Emulator (NISTNet) and simulators NWS (Network Weather Service) –Performance database –Forecasts Detection of anomalous conditions –Generate threshold events –RB adapts its behavior

Possible emulation framework

Grid Simulator Simple discrete event simulation package –Builds on GridSim and SimGrid –Java, multi-threaded Can specify a full network topology Heterogeneous resources of any kind modelled easily Arbitrary information can be published about resources Allows study of RB techniques on an arbitrary grid –monitored by NWS –threshold events feedback to RB

Next Steps Java peer-to-peer environment –Coglets for peer-to-peer Java / Globus DARC integration Adaptive middleware for mobile computing Experimental testbed for Enigma-based behaviours Adaptive RB framework XML intermediate form for PN structure … applications …