Mouse BIRN - Jagadeeswaran Rajendiran Pipeline Processing Environment

Motivation … Algorithms that have been implemented for different computer architecturesfor different computer architectures with varying input/ output needswith varying input/ output needs Data characteristics high volumeshigh volumes located on different machineslocated on different machines have different formatshave different formats Computational needs Complex preprocessing/ analysis tasksComplex preprocessing/ analysis tasks Single, flexible all-encompassing frameworkSingle, flexible all-encompassing framework High volume of measurements/ resultsHigh volume of measurements/ results It would be nice to auto-store intermediate results!

Motivation Researchers are expected to have: Expert knowledgeExpert knowledgeTools Access to computing powerAccess to computing power Collaboration is difficult data and analyses (communication) data and analyses (communication) Seamless translation between data formats Visualization framework PortablePortable Context sensitiveContext sensitive

LONI Pipeline Processing Environment data flow execution environmentdata flow execution environment links independent programslinks independent programs auto-parallelizes sectionsauto-parallelizes sections repeatable on different data-setsrepeatable on different data-sets Uses:Uses: Integrate your program as part of an analytic process Send a collaborator an analytic process Validate data formats Parallel batch processing with error handling

Architecture Modular Generic domain of applications Platform-independent (Java) Extensible and flexible design Clean code base No unnecessary dependenciesNo unnecessary dependencies Rapid prototyping of new functionalities Single encompassing GUI User friendlyUser friendly Data centricData centric

Feature rich GUI Zoom Pan Connections Tooltips Module Lists Runtime updates Workspaces

Execution Environment Executes modules Parallelization Data-independent sectionsData-independent sections If inputs are active and readyIf inputs are active and ready Data-centric Data-centric Intermediate data-setsIntermediate data-sets Can save for later analysis: OntologyCan save for later analysis: Ontology Remote Execution Client-Server using Java RMIClient-Server using Java RMI Seamless integration (external apps)

Data Visualization Additional functionality Is integrated via the extension architecture. Viewer Extensions Configurable

Data Mediation Seamless Format conversion

Grid Engine Integration Horizontal vs. Vertical Parallelization VERTICAL — Pipeline HORIZONTAL — Grid

BREAK FOR DEMO!

Protocols and Applications Data/ Functionality specification XMLXML Network protocols HTTP, FTPHTTP, FTP Grid engine capabilitiesGrid engine capabilities SUN Grid system UCSD Security Client - ServerClient - Server Diffie Hellman key agreement Advanced Encryption Standard (AES) User AuthenticationUser Authentication SHA-1 digest of passphrase AES key from digest Applications SlicerSlicer ShivaShiva

Upcoming features … Improved usability Fault tolerance (system & network) Efficiency Overlay grid implementationOverlay grid implementationProvenance Data validationData validation Information recoveryInformation recovery

Resources Wiki-Page Personnel Dr. Arthur Toga: Arthur Toga: Michael Pan: Pan: Ron Nor: Nor: Jags: THANK YOU