Grid-Powered Scientific & Engineering Applications Ho Quoc Thuan INSTITUTE OF HIGH PERFORMANCE COMPUTING

Agenda Introduction Real-life Applications Future Work

Introduction Grid computing: –Flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resource. (From “The Anatomy of the Grid: Enabling Scalable Virtual Organizations”) Benefits: resource sharing Issues: –Connection –Dynamic environment –Heterogeneity –Security –Management

To Grid-enable scientific computing focusing on end-to-end solutions –Developing Grid computing middleware: Information service Execution management Data management User interfaces … –Grid-enabling applications: Parallelize applications Integrate applications with Grid middleware Deploy applications Supply resources and execute applications …

GUI Portal Command ComponentsIntegrationInterface Data Grid Multimedia Grid Access Grid Infrastructure – IT Services Compute Grid Info. Grid Information Service Execution Management Data Management Multimedia Management Deployment Service APIs Workflow

VO Information Hub Information Service Allow users to discover & select resources that meet their needs. –Capturing/publishing/supporting queries –Large-scale information collection –Able to publish any types of information: Hardware Software Workload... –Hot plugable sensors Allow users to discover & select resources that meet their needs. –Capturing/publishing/supporting queries –Large-scale information collection –Able to publish any types of information: Hardware Software Workload... –Hot plugable sensors … End users … Site Information Service Info Agents & Info Sensors Site X Resources … Grid VO Info Agents & Info Sensors Site Y Resources Site Information Service

Execution Management Framework Execution Service LRM Execution Service LRM Metascheduler Execution Service Client Grid VO Information Hub Site IS Sensor dynamic discovery and matching reservation LRM integration Distribute job across VO resources Integrated with IS for dynamic resource discovery & matching Integrated with DS for file movement Assist clients to reserve resources using access token Allow workload optimization plug-in Job Execution Service component at resource layer for queue management & multiple job submission Integration with LRM (SGE, Torque, LSF) GT4 and DRMAA compliant Sequential, simultaneous and MPI jobs Allow hybrid execution modes Distribute job across VO resources Integrated with IS for dynamic resource discovery & matching Integrated with DS for file movement Assist clients to reserve resources using access token Allow workload optimization plug-in Job Execution Service component at resource layer for queue management & multiple job submission Integration with LRM (SGE, Torque, LSF) GT4 and DRMAA compliant Sequential, simultaneous and MPI jobs Allow hybrid execution modes Handle-driven Forwarding Hybrid

Data/Software Management Automatic indexing, file change notification Supporting search and application mapping (search-found-process) Data Component Data Manager Metadata Store Distributed Storage (1) Client Replica Catalog (RLI) File Sensors Monitored Directory Replica Catalog (LRC) Content Store File To Application Mapping OGSA-DAI Files XQuery Update Metadata Store LFN of files with matching keywords Content Indexing Query by Keyword Metadata Information (XML) XQuery Metadata & Appln mapping info Query with LFN List of PFN Insert LFN Distributed Storage (N)......

Multi-modal GUI XML Form AutomaticFormGenerator Man Page User editable User Customizations Command Line Application Rendered in Runtime DynamicFormRenderer

Grid Portal Remote access to Grid resource via browsers Submit batch job Monitor job status Download data Secure access

Workflow Editor Grid-based PSE for Engineering of Material Target complex scientific workflows Integrate with Grid middleware

Virtualization & Resource Management Apply virtualization technologies (Xen, VMWare) Package scientific tools in virtual machine images for different applications (template-based) Quick deployment Easy backup/recovery/migrate OSs as resources Apply virtualization technologies (Xen, VMWare) Package scientific tools in virtual machine images for different applications (template-based) Quick deployment Easy backup/recovery/migrate OSs as resources Grid Middleware SUN GRID ENGINE MPI VM 1 VM 2 VM 3 VM n … … Hardware Resources vmvm vmvm vmvm vmvm vmvm Software/VM Repository Configurations User Workspace provisioning customization use and accounting Xen/VMWare discoveryselectiondeploymentadaptation

End-to-End Solution Interfaces Upload, download & search data Submit, monitor jobs View/download results Middleware Capture, monitor, publish information Extract information, transfer data Schedule jobs Resources Capture information Store data Execute jobs Construct & run applications

Real-life Applications Fire Dynamic Simulation (FDS) Multiphysics (Femlab and Comsol) Computational Electromagnetics (GECEM) Motif Finder (GIS) Digital Media Grid (Adaptive

Grid-enabled Fire Dynamic Simulation Fire & smoke simulation Compute intensive applications Data intensive (output) Fire & smoke simulation Compute intensive applications Data intensive (output) Portal Execution Service LRM Execution Service Sensor Metascheduler

high bandwidth and low latency network Grid-enabled Multiphysics Grid-enabled commercial software (Femlab, Comsol) Resource discovery & execution Different models of sharing: 1.Sharing license 2.Sharing licence & computational power 3.Sharing desktop Grid-enabled commercial software (Femlab, Comsol) Resource discovery & execution Different models of sharing: 1.Sharing license 2.Sharing licence & computational power 3.Sharing desktop License Server User Computing Servers jobs (2) License Server User license (1) licenses License Server User Desktop Servers desktop (3) licenses (2)

Computational Electromagnetics Grid-enabled Computational Electromagnetics (GECEM) Grid infrastructure Grid middleware Portal Remote visualization Security Grid-enabled Computational Electromagnetics (GECEM) Grid infrastructure Grid middleware Portal Remote visualization Security Other locations Geometry data UWS IHPC BAE SYSTEMS Create geometry Generate mesh CEM simulation MeshOutput Partners: BAE SYSTEMS HP UK University of Swansea Cardiff University Partners: BAE SYSTEMS HP UK University of Swansea Cardiff University

Grid-enabled Motif Finder Multi-modal GUI Session management Multi-modal GUI Session management

Digital Media Grid Project (IHPC, IDA, HP, NTU, SMU) Torque SGE Portal IHPC Meta-scheduler JES Submit job SOAP JES LSF Meta-scheduler LSF Grid Submit job web browser Maya submit ssh http ssh animator Web service IHPC IS

Future Work Build a powerful platform to support science & engineering applications – A*STAR Digital Nervous System (ADNS) –Share resources for inter-research institutes across A*STAR –Provide software as utility services

Status Updates 2005 Information Service: –Sensors –Data Collection Server –Grid/Web Service Execution Management: –Execution Service Container can execute: Sequential jobs Concurrent jobs –Metascheduler: Resource selection based on workload and speed. Grid/Web service DRMAA standard Data Service: Sending data over Grid/Web service protocol Grid Portal Applications: –In house codes –Commercial software 2006 Information Service: –Multi-domain –P2P –Hot-plugable sensors Execution Management: –Resource reservation for parallel/interactive applications –Interface to Local Resource Managements: SGE, LSF, Torque Data Service: –Reliable data transfer –Transfer/Stream data in parallel Data management Multi-modal GUI Workflow editor Virtualization technologies Applications: –18 GB of data –More than 30 hours of processing time –Windows applications Start design and implement ADNS ADNS in action