Tool Integration with Data and Computation Grid “Grid Wizard 2”

AGENDA 1.Computing Needs Overview. 2.Grid Architecture. 3.“Grid Wizard 2” 4.Demo + Q&A

COMPUTING NEEDS OVERVIEW 1.Requirements for computing increasing fast. Main reasons: More data to process. More compute intensive algorithms available. 2.Approaches to supply demand: Qualitative: Optimized algorithms, faster processors, more memory. Quantitative: Grid Computing (parallel, distributed, etc).

GRID ARCHITECTURE Clusters & Resource Managers 1.Cluster Operating System: Operating system managing all the hardware components in a cluster. Example: Rocks. 2.Network: Communication backbone among all hardware components in a cluster. 3.Head Node: Dedicated system hosting cluster level services. 4.Compute Nodes: Computers and processors. 5.File System: Networked file system accessible by any node in the cluster (independent from other file system nodes may have access to). 6.Resource Manager: Optional software component. It serves as a front end to queue and manage cluster “client”s jobs. Example: Condor, SGE, PBS, Torque, LSF, etc.

GRID ARCHITECTURE The Grid Collection of Clusters. To use a it, the Grid “Client” is responsible to provide the logic to integrate the clusters within the logic of their specific applications. Most of the times this effort is non-trivial, non-reusable, non-extensible, lacks robustness and is far from giving the end user all the desired functionality.

GRID ARCHITECTURE “Meta Scheduler” enabled Grid - 1 An approach to overcome the taunting task of integrating clusters under a single view. Meta schedulers provide a “Resource Manager” translation layer to access clusters. Provides a unified language for accessing the clusters to the Grid “Client”.

GRID ARCHITECTURE “Meta Scheduler” enabled Grid - 2 Advantages: Simple and powerful solution. Disadv.: No granular control, depends on resource manager to execute anything on the cluster. Samples: Globus “GridWay”, “Community Scheduler Framework”, “Grid Wizard 1”

“GRID WIZARD 2” Introduction Why? Because you do not care about the grid and you just want to run your jobs faster! Complete rework of “Grid Wizard 1” meta scheduler. It is more some sort of “Grid Virtualization System”. Java based distributed system. Collection of autonomous back end applications running in clusters head and compute nodes. Extremely easy integration, installation and usage. Provides single view to clusters and/or grid. Provides lots of granular control and features to grid “client”s. Provides auto configuration with most sensible and auto discovered values to minimize user input. Provides workflow capabilities to control to a more granular level execution conditions and dependencies within requests (mid 2008). Requirements: SSH enabled clusters, Java 1.5.

“GRID WIZARD 2” Internal Components Interfaces Two types: a)To other “Grid Wizard 2” components. b)To cluster specific components. These are built on top of a pluggable framework which allows third party providers to easily add support to other systems through customized drivers.

“GRID WIZARD 2” “Grid Wizard 2” enabled Cluster Tight integration with cluster internals and transparent access to them. Granular level of execution control: submission, pause, resume, abort. Real time monitoring and alerting capabilities. Granular reporting of historic, diagnostics and statistics data. Transparent environment translation for requests. Programmatic control with rich and simple API. Workflow capabilities (future).

“GRID WIZARD 2” Distributed System - 1 “Grid Wizard 2” Cluster Components are designed to interface with other “Grid Wizard 2” Cluster Components. This feature allows them to form a distributed network of one and/or many clusters “on the fly”. Grid “Client”s can form virtual views of these distributed network “on the fly” as they wish and have access to. Provides the features of a single “Grid Wizard 2” enabled cluster to “N” “Grid Wizard 2” enabled clusters by chaining them in a ring configuration which is: Dynamically updated in real time. Transparently self-managed. Customizable per user and per request.

“GRID WIZARD 2” Distributed System - 2

“GRID WIZARD 2” “Grid Wizard 2” enabled Grid Grid “Client” has to establish a link to any “Grid Wizard 2 Components” sitting on a cluster to get a unified view of a grid and gain access to all the services “Grid Wizard 2” provides over such grid.

“GRID WIZARD 2” Architecture Overview

“GRID WIZARD 2” Architecture Overview

“GRID WIZARD 2” Tool Integration Out of the box, generic, command line integration using xml configuration files and VTL (to describe jobs): VTL Job Descriptor Sample: #set($path = "srbfile:/home/mruiz.ucsd-bcc/2d") #foreach( $count in ["01", "02", "03", "04", "05", "06", "07"]) /opt/BIRN/lddmm/1.0.1/lddmm -A ${path}/Atlas.img -T ${path}/Patient${count}.img -d 2 #end Implementation of highly specialized integrations through rich java APIs: Client API: Provides interface for user to control requests execution. Monitor API: Provides interface to register interest in events and gather them in real time from distributed system.

“GRID WIZARD 2” Drivers Integration Plug-in bundles contain: Java binaries. Plug-in xml descriptors. Drivers categories (so far): Network Protocols. Samples: Local, SSH File Systems. Samples: Local, SFTP, SRB Resource Managers. Samples: Condor, SGE, PBS, Torque, LSF

“GRID WIZARD 2” Schedule December 2007 Pre-release. Beta version of complete infrastructure including automatic deployment, limited drivers (including limited SRB drivers) and limited monitoring. Tool Integration Migration: BIRN Portal LDDMM. February 2008 Tool Integration: Slicer 3 (embedded and generic). April 2008 Production Release. Stable version with drivers for most popular cluster components (including full transparent SRB support), full monitoring, alert capabilities, API bundles, Tool Integration: FreeSurfer and FIPS (specialized and full featured). Tool Integration: BIRN Portal (embedded and generic). June 2008 Production Release. Workflow definition and execution capabilities.

“GRID WIZARD 2” Proof of concept demo Q&A