ICTP, April 2007 CIMA in Australia Ian Atkinson HPRC Manager, ITR School of Maths, Physics and IT James Cook University

ICTP, April 2007

DART: Dataset Acquisition, Acquisition & e-Research Technologies A DEST-SII funded project ( & to develop tools to handle the data & information management needs of diverse research environments. DART is developing tools to handle typical research data & information management needs, such as: 1. collecting data 2. managing data 3. analyzing that data to produce useful information 4. managing that information 5. collaboration & annotation on the information 6. publishing information 7. searching on the information DART has built three demonstrators: Climate Research X-ray crystallography Indigenous Digital History

ICTP, April 2007

DART - workflows

ICTP, April 2007 Instrument Taxonomy Rick McMullen

ICTP, April 2007 Goal: Develop a Grid enabled and Web services based portal system for collaborative remote access to scientific instruments and their data. Why? Maximise resource utilization and return on investment/s Outreach, diffusion, training and education Grid? Access to compute cycles for processing and modeling and flexible distributed storage management that may further facilitate remote access Multi-Site Collaboration Goal for production level service: Significant collaboration between USyd, JCU, IU (and many others: UAdel, Monash, UQ, DART…) Deployment and user focus issues (this is hard) Remote Instrumentation

ICTP, April 2007 Common Instrument Middleware Architecture (CIMA) Offers a general and re-usable model for instrument access and management using Web/Grid services; adaptable to different instrument settings Flexible and extensible with modular use of plug-ins, Common programmable interface – reusable code base Publish-subscribe or registration model Basis for a standardised implementation/deployment system Grid enablement of instruments & sensors

ICTP, April 2007 Common Instrument Middleware Architecture (CIMA)

ICTP, April 2007 Integrate instruments and sensors (e.g. real-time data sources) into a grid environment with Web Services interfaces Abstract instrument capabilities and functions to reduce data acquisition and analysis applications dependence on specialized knowledge about particular instruments Move production of metadata as close to instruments as possible and facilitate the automatic production of metadata Develop a standard, reusable methodology for grid enabling instruments Rick McMullen, Knowledge Acquisition and Projection Laboratory, Indiana University Common Instrument Middleware Architecture (CIMA)

ICTP, April 2007 CIMA Elements Schema for instrument functionality in WSDL Data model for representing instrument metrics and calibration A small, high performance Web Services stack –Service implementation for accessing the instruments functionality Ability to dynamically insert new protocols into running instances OGSA compliant functions to register with a directory service, authenticate users, provide access control to instrument controls and data, and co- schedule the instrument into a Grid computing and storage context;

ICTP, April 2007 CIMA X-ray Portal

ICTP, April 2007 Indiana CIMA X-Ray system 4 Components i. Instrument Representative, ii.My Manager, iii.Data Manager, iv.Portal Interface ● Instrument Rep, My Manager, Data Manager Service implemented in c++, gSoap, Perl ● Storage – ext3 file-system, MySQL database ● Portal – jrs168 portlets, gridsphere Issues – My Manager & Portal implementations domain specific i.e. Crystallography vs oceanographic monitoring

ICTP, April 2007 Video feeds CIMA Instrument Representative Data Manager CIMA SOAP Interface GridSphere Portal SRB data repository (data and metadata) HSM Storage SRB MCAT Data Processing Applications, Annontation and visualisation Instruments Live data feeds CIMA core code from IU DART/JCU/UQ/Usyd/MU Implementation Workflow & comp pipelines. Automatic metadata generation. PGL: SRB access & schemas SOAP DART Implementation

ICTP, April 2007  NFS and MySQL replaced with Storage Resource Broker (SRB)  Kepler workflow used for MyManager  Ability to customise data storage via a workflow system  Highly extensible, not restricted to SRB as repository Data Management - workflow

ICTP, April 2007 Data Grid middleware from SDSC (San Diego Supercomputing Centre) Uniform interface to heterogeneous resources over a network. Logical View of data User defined metadata structures Works in conjunction with MCAT. Storage Resource Broker (SRB) Data Management - datagrid

ICTP, April 2007  Use of Personal Grid Library (PGL) for SRB data manipulation / metadata display  Metadata schema definitions applied to experimental data - chitter chatter  Stored experimental data is able to be easily retrieved, secured and annotated  Visual diagnostics for sensors (e.g. Graphs, Visual Indicators)  Live CCD image analysis (experimental status) DART - User Portal

ICTP, April 2007

A. Beitz, et al. Monash

ICTP, April 2007 J. Hunter, et al. UQ

ICTP, April 2007  Initial Instrument Representatives deployed at JCU, USyd, Monash & UQ Goal  Each site having their own Data Manager and SRB storage facility which is federated across sites  Shibbolith based AAA and Virtual Organisations  Federate/Replicate data into a a national SRB store… Data Federations with SRB

ICTP, April 2007 –Data federations (SRB) –Security and Rights Management Shibboleth –Analysis tools (Portals/portlets) –Workflows Goal is an end-to-end workbench Extensions to larger and smaller instruments (e.g. sensors) Future Developments

ICTP, April 2007 Acknowledgements: JCUSydney UniIndiana Uni Adelaide Uni S. NessD. du Boulay K. Chiu (SUNNY) A. Wendelborn M. Morgan C. Chee P. Coddington F. Eilert D. Zhang Department of Education Science and Training: Dataset Acquisition, Accessibility, and Annotation e-Research Technologies (DART) project Australian Research Council: e-Research Seed Funding Programme and the Research Networks Programme (MMSN: Molecular and Materials Structure Network).

ICTP, April 2007