Tool Integration with Data and Computation Grid GWE - “Grid Wizard Enterprise”

AGENDA 1.Computing Needs Overview. 2.Grid Architecture. 3.GWE - “Grid Wizard Enterprise” 4.Demo + Q&A

COMPUTING NEEDS OVERVIEW 1.Requirements for computing increasing fast. Main reasons: More data to process. More compute intensive algorithms available. 2.Approaches to supply demand: Qualitative: Optimized algorithms, faster processors, more memory. Quantitative: Grid Computing (parallel, distributed, etc).

GRID ARCHITECTURE Clusters & Resource Managers 1.Cluster Operating System: Operating system managing all the hardware components in a cluster. Example: Rocks. 2.Network: Communication backbone among all hardware components in a cluster. 3.Head Node: Dedicated system hosting cluster level services. 4.Compute Nodes: Computers and processors. 5.File System: Networked file system accessible by any node in the cluster (independent from other file system nodes may have access to). 6.Resource Manager: Optional software component. It serves as a front end to queue and manage cluster “client”s jobs. Example: Condor, SGE, PBS, Torque, LSF, etc.

GRID ARCHITECTURE The Grid Collection of Clusters. To use a it, the Grid “Client” is responsible to provide the logic to integrate the clusters within the logic of their specific applications. Most of the times this effort is non-trivial, non-reusable, non-extensible, lacks robustness and is far from giving the end user all the desired functionality.

GRID ARCHITECTURE “Meta Scheduler” enabled Grid - 1 An approach to overcome the daunting task of integrating clusters under a single view. Meta schedulers provide a “Resource Manager” translation layer to access clusters. Provides a unified language for accessing the clusters to the Grid “Client”.

GRID ARCHITECTURE “Meta Scheduler” enabled Grid - 2 Advantages: Simple and powerful solution. Disadv.: No granular control, depends on resource manager to execute anything on the cluster. Samples: Globus “GridWay”, “Community Scheduler Framework”, “Grid Wizard”

GWE - “GRID WIZARD ENTERPRISE” Introduction Why? Because you do not care about the grid and you just want to run your jobs faster! Complete rework of “Grid Wizard” meta scheduler. It is more a “Grid Virtualization Platform”. Java based distributed system. Collection of autonomous back end applications running in clusters head and compute nodes. Extremely easy integration, installation and usage. Provides single view to clusters and/or grid. Provides lots of granular control and features to grid “client”s. Provides auto configuration with most sensible and auto discovered values to minimize user input. Requirements: SSH enabled clusters, Java 1.5.

GWE - “GRID WIZARD ENTERPRISE” Internal Components Interfaces Two types: a)To other GWE components. b)To cluster specific components. These are built on top of a pluggable framework which allows third party providers to easily add support to other systems through customized drivers.

GWE - “GRID WIZARD ENTERPRISE” “Grid Wizard Enterprise” enabled Cluster Tight integration with cluster internals and transparent access to them. Granular level of execution control: submission, pause, resume, abort. Real time monitoring and alerting capabilities. Granular reporting of historic, diagnostics and statistics data. Transparent environment translation for requests. Programmatic control with rich and simple API. Workflow capabilities (future).

GWE - “GRID WIZARD ENTERPRISE” Distributed System - 1 “Grid Wizard Enterprise” Cluster Components are designed to interface with other “Grid Wizard Enterprise” Cluster Components. This feature allows them to form a distributed network of one and/or many clusters “on the fly”. Grid ‘clients’ can form virtual views of these distributed network “on the fly” as they wish and have access to. Provides the features of a single GWE enabled cluster to “N” GWE enabled clusters by chaining them in a ring configuration which is: Dynamically updated in real time. Transparently self-managed. Customizable per user and per request.

GWE - “GRID WIZARD ENTERPRISE” Distributed System - 2

GWE - “GRID WIZARD ENTERPRISE” “Grid Wizard Enterprise” enabled Grid Grid “Client” has to establish a link to any “Grid Wizard Enterprise Components” sitting on a cluster to get a unified view of a grid and gain access to all the services “Grid Wizard Enterprise” provides over such grid.

GWE - “GRID WIZARD ENTERPRISE” Architecture Overview

GWE - “GRID WIZARD ENTERPRISE” Architecture Overview

GWE - “GRID WIZARD ENTERPRISE” Tool Integration Out of the box, generic, command line integration using xml configuration files and VTL (to describe jobs): VTL Job Descriptor Sample: #set($path = "srbfile:/home/mruiz.ucsd-bcc/2d") #foreach( $count in ["01", "02", "03", "04", "05", "06", "07"]) /opt/BIRN/lddmm/1.0.1/lddmm -A ${path}/Atlas.img -T ${path}/Patient${count}.img -d 2 #end Implementation of highly specialized integrations through rich java APIs: Client API: Provides interface for user to control requests execution. Monitor API: Provides interface to register interest in events and gather them in real time from distributed system.

GWE - “GRID WIZARD ENTERPRISE” Drivers Integration Plug-in bundles contain: Java binaries. Plug-in xml descriptors. Drivers categories (so far): Network Protocols. Samples: Local, SSH File Systems. Samples: Local, SFTP, SRB Resource Managers. Samples: Condor, SGE, PBS, Torque, LSF

GWE - “GRID WIZARD ENTERPRISE” Schedule December 2007 Pre-release. Alpha version of complete infrastructure including automatic deployment, limited drivers (including limited SRB drivers) and limited monitoring. Tool Integration Migration: BIRN Portal LDDMM. February 2008 Pre-release. Beta version. Tool Integration: Slicer 3 (embedded and generic). April 2008 Production Release. Stable version with drivers for most popular cluster components (including full transparent SRB support), full monitoring, alert capabilities and API bundles. Tool Integration: BIRN Portal (embedded and generic). June 2008 Tool Integration: FreeSurfer and FIPS (specialized and full featured).

GWE - “GRID WIZARD ENTERPRISE” Proof of concept demo Q&A