WebFlow High-Level Programming Environment and Visual Authoring Toolkit for HPDC (desktop access to remote resources) Tomasz Haupt Northeast Parallel Architectures Center Syracuse University

T. Haupt: WebFlow - Desktop Access to Remote Resources 2 Outline 4 WebFlow 4 WebFlow over Globus 4 WebFlow Applications –LMS –Quantum Simulations 4 Comments on DATRR

T. Haupt: WebFlow - Desktop Access to Remote Resources 3 WebFlow Mission 4 The visual HPDC framework that offers an intuitive Web browser based interface and a uniform point of interactive control for a variety of computational modules and applications, running at various labs on different platforms. 4 Our technology goal is to build a high-level user friendly commodity software based visual programming and runtime environment for HPDC.

T. Haupt: WebFlow - Desktop Access to Remote Resources 4 WebFlow Applications 4 WebFlow applications are composed of independent reusable modules 4 Modules are written by module developers –people who have only limited knowledge of the system on which the modules will run. 4 The WebFlow system hides module management and coordination functions –allocating and running the modules on various machines, creating connections among the modules, sending and receiving data across these connections, or running several modules concurrently on one machine.

T. Haupt: WebFlow - Desktop Access to Remote Resources 5 WebFlow Implementation 4 WebFlow is implemented as a modern three-tier system –Tier 1 is a high-level front-end for visual programming built on top of the Web and OO commodity standards. –Distributed object-based, scalable, and reusable Web server and Object broker Middleware forms Tier 2. – Back-end services comprise Tier 3. In particular, high performance services are implemented using the metacomputing toolkit of Globus.

T. Haupt: WebFlow - Desktop Access to Remote Resources 6 WebFlow Architecture

T. Haupt: WebFlow - Desktop Access to Remote Resources 7 Middle-Tier Mesh of Web Servers running servlets that manage and coordinate distributed computation. –Session Manager, –Module Manager, –Connection Manager. Front End

T. Haupt: WebFlow - Desktop Access to Remote Resources 8 Backend Modules 4 are Java objects that implement WebFlow specific module interface: –initialize register itself to the module manager, and registers its input and output ports to the connection manager –run implements the actual functionality of the module –destroy release resources

T. Haupt: WebFlow - Desktop Access to Remote Resources 9 Run Method 4 The functionality of the module (run method) can be implemented entirely in Java 4 Existing applications written in languages different than Java can be easily encapsulated 4 Execution of the module can be delegated to an external system capable of resource allocation such as Globus, Condor, etc.

T. Haupt: WebFlow - Desktop Access to Remote Resources 10 WebFlow over Globus 4 In order to run WebFlow over Globus there must be at least one WebFlow node capable of executing Globus commands, such as globusrun 4 Jobs that require computational power of massively parallel computers are directed to the Globus domain, while others can be launched on much more modest platforms, such as the user’s desktop or even a laptop running Windows NT. Bridge between WebFlow and Globus

T. Haupt: WebFlow - Desktop Access to Remote Resources 11 WebFlow Applications 4 Landscape Management System –CEWES: MSRC/PET 4 Quantum Simulations –Alliance/NCSA

T. Haupt: WebFlow - Desktop Access to Remote Resources 12 LMS Objectives To develop a web based system that implements a “navigate-and-choose” paradigm and allows the end user to: –Select (a set of) computational modules that provide answers to the problem at hand –Retrieve input data sets from remote sources –Use adequate (remote) computational resources –Visualize and analyze output data on the local host Anytime, anywhere, using any platform (e.g., a connected to the internet laptop PC)

T. Haupt: WebFlow - Desktop Access to Remote Resources 13 LMS: Changes in Vegetation l A decision maker (the end user of the system) wants to evaluate changes in vegetation in some geographical region over a long time period caused by some short term disturbances such as a fire or human’s activities. l One of the critical parameters of the vegetation model is soil condition at the time of the disturbance. l This in turn is dominated by rainfalls that possibly occur at that time

T. Haupt: WebFlow - Desktop Access to Remote Resources 14 LMS: Changes in Vegetation l Data retrieval l Data preprocessing l Simulation: two interacting codes ¶ EDYS · CASC2D l Visualization WMS EDYSCASC2D DEM Land Use Soil Texture Vegetation

T. Haupt: WebFlow - Desktop Access to Remote Resources 15 LMS Front End

T. Haupt: WebFlow - Desktop Access to Remote Resources 16 LMS Front End (3)

T. Haupt: WebFlow - Desktop Access to Remote Resources 17

T. Haupt: WebFlow - Desktop Access to Remote Resources 18

T. Haupt: WebFlow - Desktop Access to Remote Resources 19 LMS Front End (4)

T. Haupt: WebFlow - Desktop Access to Remote Resources 20 Visualizations

T. Haupt: WebFlow - Desktop Access to Remote Resources 21 Watershed Simulation

Quantum Simulations using WebFlow - a High Level Visual Interface for Globus (Alliance Team B)

T. Haupt: WebFlow - Desktop Access to Remote Resources 23 Quantum Simulations

T. Haupt: WebFlow - Desktop Access to Remote Resources 24 QS: WebFlow implementation

T. Haupt: WebFlow - Desktop Access to Remote Resources 25 QS Front End

T. Haupt: WebFlow - Desktop Access to Remote Resources 26

T. Haupt: WebFlow - Desktop Access to Remote Resources 27 Lessons learned 4 Access to data from many different sources –data sources identification –support for standard protocols (fire walls!) –support for high performance data transfer –authentication and authorization –security (encryption) –filtering and preprocessing

T. Haupt: WebFlow - Desktop Access to Remote Resources 28 Lessons learned (2) 4 Resource identification & allocation –small tasks –access to commodity components –HPCC (usually batch systems) –coallocation of resources –resource brokers –resource allocation language –authentication and authorization

T. Haupt: WebFlow - Desktop Access to Remote Resources 29 Lessons learned (3) 4 Monitoring (generic and custom) 4 Fault tolerance 4 Quality of service 4 Security of Web access and Web transactions 4 Front-End API