© Geodise Project, University of Southampton, Geodise & GeodiseLAB Simon Cox University of Southampton 21 st April 2005

Grid Enabled Optimisation and Design Search for Engineering (G EODISE ) Southampton, Oxford and Manchester Simon Cox- Grid/ W3C Technologies and High Performance Computing Global Grid Forum Apps Working Group Andy Keane- Director of Rolls Royce/ BAE Systems University Technology Partnership in Design Search and Optimisation Mike Giles- Director of Rolls Royce University Technology Centre for Computational Fluid Dynamics Carole Goble- Ontologies and DARPA Agent Markup Language (DAML) / Ontology Inference Language (OIL) Nigel Shadbolt- Director of Advanced Knowledge Technologies (AKT) IRC BAE SYSTEMS- Engineering Rolls-Royce- Engineering Fluent- Computational Fluid Dynamics Microsoft- Software/ Web Services Intel- Hardware Compusys- Systems Integration Epistemics- Knowledge Technologies Condor- Grid Middleware

© Geodise Project, University of Southampton, The GEODISE Team... Richard Boardman Sergio Campobasso Liming Chen Mike Chrystall Trevor Cooper-Chadwick Simon Cox Mihai Duta Clive Emberey Hakki Eres Matt Fairman Mike Giles Carole Goble Ian Hartney Tracey Hunt Zhuoan Jiao Andy Keane Marc Molinari Graeme Pound Colin Puleston Nicola Reader Angus Roberts Mark Scott Nigel Shadbolt Wenbin Song Paul Smart Barry Tao Lisa Telfer-Brunton Jasmin Wason Fenglian Xu Gang “Luke” Xue

© Geodise Project, University of Southampton, Distributed Systems 2005 Network IP HTTP (HTML) Compute/ Data Moore’s Law Future Services Flexible, Seamless, and Secure Sharing Software (HTML) XML Web Services ASCII

© Geodise Project, University of Southampton, Negative Scarf Inlet Conventional Inlet 0 12 Applications - Geodise CFD 3D engine nacelle optimisation  Goal - reduce ground noise generated by fan when plane takes off.  Optimise aerodynamic performance when scarf angle is varied.

Problem definition Design of Experiment Response surface modelling Optimisation on Response surface Validation ProEngineer CAD (Condor Pool) Gambit Meshing (Globus Compute) Fluent CFD (Globus Compute) Workflow for aerodynamic shape optimisation using CAD, Gambit, and Fluent

© Geodise Project, University of Southampton, Applications – GEM Electromagnetic optimisation Large number of designs, parameters and solutions Query for a particular data range to post- process Transmission of light properties through a photonic crystal Parameters are radius of holes and light wavelength

Geodise will provide grid-based seamless access to an intelligent knowledge repository, a state-of-the-art collection of optimisation and search tools, industrial strength analysis codes, and distributed computing & data resources G EODISE

© Geodise Project, University of Southampton, Grid-Enabled Scripting Environment Why use scripting languages? –Flexibility –High-level functionality –Quick application development –Extend the user’s existing PSE –… Acts as execution/ enactment engine too Approach –Matlab initially chosen as the hosting environment –Now working with Python/ Jython Matlab Python/ & /Jython

Engineering Applications of the Grid via Geodise Problem Solving Environment Bringing underlying infrastructure into the Problem Solving Environment “Useful into useable” Grid Hardware Matlab, Python, etc “Heartbeat” Enterprise “Dial-tone” Geodise Toolkits

Geodise deliverables Grid Compute toolbox v Cycle stealing v High-performance computing v Integration with heterogeneous environments (Windows, Linux, Solaris, …) Grid Data toolbox v Raw files into journaling filesystem v Metadata in relational database (Oracle, DB2, SQL Server, mySQL…) v XML processing capability Intelligent Workflow composition, execution and monitoring tool v Rule and case-based composition of ‘best’ workflow (semantics and knowledge driven) v Deployment to compute/ data resources v System-level “heartbeat” v User/ Application level “dial-tone” (security, resource availability, network weather, …) Problem Solving Environments e.g. Matlab, iSight Python, ModelCenter,… Grid-enabled Toolboxes for Engineers Middleware (Grid) Hardware Optimisation `Tools Designed around web-services to inherit and leverage Technologies (e.g. security, reliability) and Development tools

© Geodise Project, University of Southampton, Geodise Tools A number of Grid-enabled design tools have been developed –Compute Toolbox: certificate based authentication, job submission, and file transfer, etc. –Database Toolbox: archive files and data, custom metadata, locate and retrieve data, etc. –XML Toolbox: convert Matlab structures to and from XML –Workflow Editor: graphical problem solving environment which allows users to construct and configure workflows for the Grid –OptionsMatlab: Matlab interface to the Options design exploration system –Knowledge Toolbox: Intelligent Design advice

Geodise Database Globus Server Graphical Workflow Tool Matlab Scripting Environment Compute ToolboxDatabase Toolbox OptionsMatlab Compute Toolbox API Database Toolbox API Knowledge Toolbox API Knowledge Repository Metadata and Structure Repository Computation Globus Server #1 Globus Server #N Condor Cluster Globus Server / Condor Node Condor Master Condor Node #1 Engineer Knowledge Toolbox Condor Node #N Applications Pro/Engineer ICAD Catia IDEAS CAD Gambit/Fluent Abaqus Catia HYDRA Analysis Globus Servers Globus and Condor Job Managers JavaMatlab Metadata Webservice Physical files, scripts, and outputs Results by GridFTP Results Local Filesystem Matlab Script Grid Job Submission Results Metadata by webservice Geodise Architecture XML Toolbox

© Geodise Project, University of Southampton, Sponsors

© Geodise Project, University of Southampton, GeodiseLab Activities for the OMII Harden & deliver Geodise toolboxes –Geodise Compute Toolbox –Geodise Database Toolbox –XML Toolbox for Matlab Documentation Testing Extend existing functionality Train and support users... GGF: Simple API for Grid Applications (SAGA) Working Group

© Geodise Project, University of Southampton, Compute Toolbox: Aims Support access to whichever compute resources users wish to exploit –Globus GT2 –OMII services –Condor Provide access from alternative environments –Matlab –Jython –cPython Leverage existing APIs wherever possible

© Geodise Project, University of Southampton, Compute Toolbox: Hit-list Matlab Geodise Java API JythoncPython Condor Native ?.py files Windows HPC/Grid Condor Web Service Globus GT2 OMII Services Java ClientJava CoGOMII API.NET Client.m files

© Geodise Project, University of Southampton, GeodiseLab Compute Toolbox Available Now

© Geodise Project, University of Southampton, Compute Toolbox Release Matlab Globus GT2 OMII Services Geodise Java API JythoncPython Condor Native Condor Web Service Java CoGOMII APIJava Client ?.py files.m files

© Geodise Project, University of Southampton, Toolbox Architecture Matlab Environment Globus GT2 Resource GRAM Service GridFTP Service gd_jobsubmit.m Java CoG gd_getfile.m script.m GridFTP file transfer GRAM job submission

© Geodise Project, University of Southampton, Certificate Management Functions gd_certinfo Returns information about the user's certificate gd_createproxy Creates a Globus proxy certificate gd_proxyinfo Returns information about the user's proxy certificate gd_proxyquery Queries whether a valid proxy certificate exists gd_destroyproxy Destroys the local copy of the user's Globus proxy certificate

© Geodise Project, University of Southampton, Certificate Management C:\GEODISE >> gd_createproxy

© Geodise Project, University of Southampton, Certificate Management C:\GEODISE >> subject = gd_certinfo subject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound issuer : support.ac.uk start date : Wed Sep 22 16:31:24 BST 2004 end date : Thu Sep 22 16:31:24 BST 2005 subject = /C=UK/O=eScience/OU=Southampton/L=SeSC/CN=graeme pound >> isvalid = gd_proxyinfo subject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound,CN=proxy issuer : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound type : full legacy globus proxy strength : 512 bits timeleft : 11 h, 59 min, 30 sec isvalid = 1 >>

© Geodise Project, University of Southampton, Job Submission Functions gd_jobstatus Gets the status of a Globus GRAM job gd_jobsubmit Submits a compute job to a Globus GRAM job manager gd_jobpoll Queries the status of a Globus GRAM job until complete gd_jobkill Kills a Globus GRAM specified by a job handle gd_chmod Changes file permissions of a file on a Globus resource gd_condorsubmit Submits a job to a Condor pool via a Globus resource gd_submitunique Submits a GRAM job to a unique working directory

© Geodise Project, University of Southampton, Job Submission C:\GEODISE >> RSL = '&(executable="/bin/date")(stdout="date.out")'; >> host = 'grid-compute.oesc.ox.ac.uk'; >> jobhandle = gd_jobsubmit(RSL, host) jobhandle = >> isdone = gd_jobpoll(jobhandle) isdone = 1 >>

© Geodise Project, University of Southampton, GridFTP File Transfer gd_getfile Retrieves a remote file using GridFTP gd_putfile Puts a remote file using GridFTP gd_transferfile Performs a third-party file transfer using GridFTP gd_makedir Creates a remote directory using GridFTP gd_listdir Lists the contents of a directory on a GridFTP resource gd_fileexists Tests the existence of files on a Globus resource gd_rmdir Deletes a remote directory using GridFTP gd_rmfile Deletes a remote file using GridFTP gd_rmuniquedir Deletes a remote directory and its contents

© Geodise Project, University of Southampton, File Transfer C:\GEODISE >> exists = gd_fileexists(host, 'date.out') exists = 1 >> gd_getfile(host,'date.out','localfile.txt'); >> type('localfile.txt') Mon Oct 25 09:52:46 BST 2004 >> gd_rmfile(host, 'date.out'); >> exists = gd_fileexists(host, 'date.out') exists = 0 >>

© Geodise Project, University of Southampton, Resource Testing gd_servermetrics Performs tests upon a list of Globus resources gd_testauthentication Tests authentication with a Globus resource gd_testfiletransfer Tests file transfer to a Globus resource gd_testjobsubmission Tests the job submission to a Globus resource gd_timeauthentication Times authentication to a Globus resource gd_timefiletransfer Times file transfer to a Globus resource gd_timejobsubmission Times a job submission to a Globus resource

© Geodise Project, University of Southampton, Resource Testing C:\GEODISE >> servers(1).name = 'grid-compute.leeds.ac.uk'; >> servers(2).name = 'escience-dept2.sesnet.soton.ac.uk'; >> results = gd_servermetrics(servers); >> disp(results(1)) name: 'grid-compute.leeds.ac.uk' authentication: 188 jobsubmission: 672 filetransfer: 984 >> disp(results(2)) name: 'escience-dept2.sesnet.soton.ac.uk' authentication: 125 jobsubmission: 485 filetransfer: 4938 >>

© Geodise Project, University of Southampton, GeodiseLab XML Toolbox Available Now

© Geodise Project, University of Southampton, Engineer’s familiar scripting environment 3C0003 2B0002 1A0001 ValueNameID XML Toolbox Web Technology & Services Databases Access Communication

© Geodise Project, University of Southampton, Features Powerful Matlab to XML interface Allows transparent conversion between proprietary Matlab data types and standard XML Brings flexibility of open XML Standard to Matlab Easy-to-use interface (4 simple core functions: xml_save, xml_load, xml_format, xml_parse ) Extensive documentation & direct help functionality Large range of potential applications: –Science, Engineering, IT, Web, … Toolbox eng_opt_design eng_opt_design DB Filestore Standard File Formats (eg. STEP-XML, …)

© Geodise Project, University of Southampton, XML Toolbox in action Jan 2004 v2.0 released Nov 2004 –Downloads from soton.ac.uk/~gridem: 227 –Downloads from mathworks.com/matlabcentral: 882 Apr 2005 v3.0 –> 1500 downloads total Large user base –Academic & Commercial –Extensive testing & feedback community –Market pull So far very positive feedback from users Matlab Central user rating: 5 / 5 Linked by Mathworks help for XML in Matlab

© Geodise Project, University of Southampton, Aalesund College: Read data from database Aberdeen University: Export analysis results from Matlab to XML ADVA: to read existing XML-files in a Matlab structure and do some calculations Advanced Tech Center: configuring xml input into matlab Alfred-Wegener-Institut: Handle data from echo sounder AMAT: Evaluation of XML input in matlab Anna University: We are programmed to develop a new tool for analysing discrete event system using matlab. Applied Math Labolatory, Ecole Centrale Paris Aston University: To read date from xml files in matlab Avantium Technologies BV: Transferring HTS data stored as XML into Matlab BAE Systems: Data import/export between MATLAB and Delphi; Generation of XML files to control a BAE SYSTEMS Software Defined Radio Beb Corp Bosch GmbH Calypso Medical: looking at xml files in matlab Carnegie Mellon university: Use XML in our facial expression analysis project Center for Advanced Spatial Technologies: Writing least squares adjustments for geodetic observations as part of research in new techniques for parameter estimation combining photogrammetric and terrestrial observations. Data is varied but structured and I am interested in using XML Chouaib Doukkali Universit: Numericals simulations of the physics phenomenons Christian-Albrechts-Universitat zu Kiel (CAU): City College of CUNY: Projects using Matlab and XML technology. CMS: For trial Cold Spring Harbor Laboratory: Storing and parsing neuronal data Conestoga college: intro to telecomme Crystal Engineering CU: Sending sending commands to matlab via xml, outputting results of calculations from matlab in xml to be used by other products Daimler Chrysler: test DeAM: xml reading DECO DEIMOS Engenharia: Reading/writing configuration files DGA Defence, France: For managing data files used in simulation's input DSTO: input parameters to radar model EADS CRC: Information Emyo: For teaching ENI: Reading & Modifying XML documents EPFL: Evaluation for data exchange between matlab and other software Fairchild Dornier: Currently I make a research, how I can efficiently store my matlab data structure in a text editable data format, that's why I'am looking for a xmlread and a xmlwrite function. Financial Engines FIOT Fraunhofer Germany: The last aim of our research is to provide user a platform for maintenance oriented decision via web service. 38 sample institutions

© Geodise Project, University of Southampton,

© Geodise Project, University of Southampton, GeodiseLab Database Toolbox Available Now

© Geodise Project, University of Southampton, Database Toolbox – Aims Make data easier to manage, find and share Store data with descriptive information –Standard and user defined metadata –Query over metadata to easily locate data Familiar interface for engineers –Wrap toolbox as Matlab/Jython functions –Easy integration into user scripts

© Geodise Project, University of Southampton, Database Toolbox – Features Archive files and data structures with metadata Group related data Simple query interface Retrieve data based on ID rather than location Security –User grants permission for others to access their data –Certificate-based authentication and authorisation –Secure Web service access to central database

© Geodise Project, University of Southampton, Functions Overview gd_archive Stores a file or structure with metadata into an archive gd_query Queries over metadata or structures in the archive gd_retrieve Retrieves a file or structure from the archive to the local machine gd_addusers Grants users permission to access some data gd_datagroup Creates a new datagroup to aggregate files and structures gd_datagroupadd Adds a file or structure to a datagroup gd_display Displays the results of a query

© Geodise Project, University of Southampton, GeodiseLab Database Toolbox Worked Example

Grid Client Query Service Globus Server XML ToolboxJava CoG Location Service Authorisation Service Metadata Service Database Generate Data

Grid Client Query Service Globus Server XML ToolboxJava CoG Location Service Authorisation Service Metadata Service Database Metadata

Grid Client Query Service Globus Server XML ToolboxJava CoG Location Service Authorisation Service Metadata Service Database Archive XML ToolboxJava CoG Globus Server Location Service Authorisation Service Metadata Service Database

Grid Client Query Service Globus Server XML ToolboxJava CoG Location Service Authorisation Service Metadata Service Database Generate More Data Java CoG Globus Server XML Toolbox Location Service Authorisation Service Metadata Service Database

Grid Client Query Service Globus Server XML ToolboxJava CoG Location Service Authorisation Service Metadata Service Database Group Files XML Toolbox Authorisation Service Metadata Service Database

Grid Client Query Service Globus Server XML ToolboxJava CoG Location Service Authorisation Service Metadata Service Database Query XML Toolbox Query Service Authorisation Service Database

Grid Client Query Service Globus Server XML ToolboxJava CoG Location Service Authorisation Service Metadata Service Database Retrieve Java CoG Globus Server Location Service Authorisation Service Database

© Geodise Project, University of Southampton, Applications - GENIE  GENIE supports environmental scientists modelling long term climate change.  Optimising a parameterised model over a multi-dimensional state space.  Database used for monitoring, sharing, post-processing and fault tolerance. Earth system model tuning studies

© Geodise Project, University of Southampton, Distributed Systems: 2005 and beyond? Network IP HTTP (HTML) Compute/ Data Moore’s Law Software (HTML) XML Web Services ASCII Experiences Services Robustness Security Reliability Visualization Quality of Service Knowledge Technologies Collaboration tools & frameworks Platform Strategies Development Tools Systems Architecture Distributed Data Management Problem Solving Environments Algorithms, Methods and Libraries Heterogeneous Distributed Computing