1. Wrapping Scientific Applications As Web Services Using The Opal Toolkit Wrapping Scientific Applications As Web Services Using The Opal Toolkit Nadya Williams nadya@sdsc.edu Sriram Krishnan, Ph.D. sriram@sdsc.edu nadya@sdsc.edu sriram@sdsc.edu

2. Motivation Enable access to scientific applications on Grid resources –Seamlessly via a number of user interfaces –Easily from the perspective of a scientific user Enable the creation of scientific workflows –Possibly with the use of commodity workflow toolkits Overcome complexity of Grid access: –User account creation –Management of credentials –Installation and deployment of scientific software –Interaction with Grid schedulers –Data management

3. Philosophy Focus on application-level services Provide access to virtualized resources Enable access via multiple user interfaces

4. The Opal2 Toolkit: Key Contributions Scientific Software as a Service (SaaS) –Providing higher-level scientic services, focusing on improving scientic pipelines and workows, rather than lower-level infrastructure services –Building repeatable solutions for scientific software that can be leveraged by multiple clients Scientific Service as Software (SaS) –Packaging of scientific services as deployable units (Amazon Machine Images and Rocks Rolls) –Customizable and usable by a broad community of scientific software developers and service providers

5. Application-level Services Traditional model: services for middleware tools –job launch –data transfer –… Our model: Services Oriented Science –Scientific applications as first class services (e.g. APBS, AutoDock, MEME) –Delegation of middleware management to the services back-end –End-users are presented with science-oriented, and not middleware-oriented interfaces

6. Access to Virtualized Resources Enabling access to raw resources is quite complex –Does not scale very well with the number of users, and number of systems Instead, access should be provided to virtualized resources –Provide Software as a Service –Leverage Infrastructure & Storage as Services

7. Multiple User Interfaces

8. Condor poolSGE Cluster PBS Cluster Globus Application Services Opal GUIPMV/VisionKepler Big Picture Grid/Cloud Middleware

9. Possible Approaches Write application services by hand –Pros: More flexible implementations, stronger data typing via custom XML schemas –Cons: Not generic, need to write one wrapper per application Use a Web services wrapper toolkit, such as Opal –Pros: Generic, rapid deployment of new services –Cons: Less flexible implementation, weak data typing due to use of generic XML schemas

10. The Opal2 Toolkit: Overview Enables rapid deployment of scientific applications as Web services (< 2 hours) Steps –Application writers create configuration file(s) for a scientific application –Deploy the application as a Web service using Opals simple deployment mechanism (via Apache Ant) –Users can now access this application as a Web service via a unique URL

11. Opal2: Key Features Standard WSDL and SOAP API for management of scientific applications –Operations to launch jobs, query status, retrieve outputs –Ability to transfer input files using Base64 encoded binary format, MIME attachments, and 3 rd party transfers A Dashboard for accessing applications and usage statistics –Automatic interface generation using the XML specification for command-line arguments Ability to connect to multiple resource providers at the back-end –Fork, Globus GRAM, DRMAA, Condor, CSF4 State management for jobs (and usage statistics) provided via Hibernate

12. Opal2 Configuration: The Plug-in Model

13. Opal2: Configuration tomcat.url=http://rocce.ucsd.edu:8080 num.procs=96 working.dir=opal-jobs mpi.run=/opt/openmpi/bin/mpirun mail.enable=true mail.smtp.host=smtp.gmail.com mail.smtp.auth=true mail.smtp.debug=false mail.smtp.from=noreply@rocce.ucsd.edu mail.smtp.user= mail.smtp.password= opal.datalifetime=4 days opal.hard_limit=3600 opal.jobmanager=edu.sdsc.nbcr.opal.manager.DRMAAJobManager drmaa.pe=mpich globus.gatekeeper=host:2119/jobmanager-sge globus.service_cert=/path/to/your/globus/cert globus.service_privkey=/path/to/your/globus/key globus.gridftp_base=gsiftp://host:2811/working_dir csf4.workingDir=opal_runs mpi.script=/opt/condor/etc/examples/mp1script opal.ip.processing=true opal.ip.limit=35 opal.ip.blacklist=66.102.7.104 opal.ip.whitelist=66.102.7.105, 127.0.0.1

14. Opal2: Deploying Applications No WSDL, no Java code to be written One xml configuration file for an application

15. Sample Application Configuration <appConfig xmlns="http://nbcr.sdsc.edu/opal/types" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> ]]> <![CDATA[ --help : Display this text --CFAC= : Factor by which to expand mol dims to get coarse grid dims [default = 1.7]... ]]> /homes/apbs_user/bin/psize.py --GMEMCEIL=1000 false

16. XML Specification for Command-line Arguments Used for automatic interface generation, argument validation, auto-generation of WSDL (community contribution) help --help = cfac --CFAC FLOAT …… pqr FILE INPUT true PQR Usage: psize.py [opts] Optional Arguments: --help : Display this text --CFAC= : Factor by which to expand mol dimsto get coarse grid dims [default = 1.7] --FADD= : Amount to add to mol dims to get fine grid dims [default = 20] --SPACE= : Desired fine mesh resolution [default = 0.5] --GMEMFAC= : Number of bytes per grid point required for sequential MG calculation [default = 160] --GMEMCEIL= : Max MB allowed for sequential MG calculation. Adjust this to force the script to perform faster calculations (which require more parallelism). [default = 400] …..

17. Application Service Deployment & Undeployment Deploy service onto a local Tomcat container: ant -f build-opal.xml deploy -DserviceName= -DappConfig= Undeploy service : ant -f build-opal.xml undeploy -DserviceName=

18. Service Operations Get application metadata: Returns metadata specified inside the application configuration Launch job: Accepts list of arguments and input files (Base64 encoded), launches the job, and returns a jobID Query job status: Returns status of running job using the jobID Get job outputs: Returns the locations of job outputs using the jobID Get output as Base64: Returns an output file in Base64 encoded form Destroy job: Uses the jobID to destroy a running job

19. Opal2 Toolkit: Client Side Several clients APIs available: Java, Python, Perl Generic command-line client: # java edu.sdsc.nbcr.opal.GenericServiceClient -l http://localhost:8080/axis/services/PDB2PQRServicePort -r launchJob -a "-ipdb sample.pdb -h -opdb output.pdb -f etc/sample.pdb Too complex for beginners Need a Graphical User Interface to simplify access

20. getAppMetadata launchJob getOutput PDB2PQR getAppMetadata launchJob getOutput MEME Opal server Vision Kepler User clients Browser User Interfaces and Workflow Tools

21. Case Study: Opal2 Dashboard Main characteristics: –Working out of the box (No additional configuration) –Multi-platform -> Web interface –Implemented in Java Key features: –Service Registry/Listing –Simple and advanced submission forms –Usage statistics –Documentation and tutorials

22. List of Services http://ws.nbcr.net/

23. Problem: user has to input command line - too error prone. Solution: submission form must be customized for command line arguments Simple Submission Form

24. Advanced Submission Form An optional tag in the appConfig file to describe input parameters (types) It is a command line syntax description language

25. Command Line Input Arguments Taxonomy Flag: not ordered, type Boolean, a dash followed by characters Tagged: not ordered, Prefixed by some tag Untagged: ordered, not prefixed Parameters: many types: integer, float, string, file, enumeration, etc.

26. Advanced Submission Form Grouping capability: –To group several parameters together –A group can be exclusive Default values An example of the command line syntax description language and of the form…

27. nodebump --nodebump Do not perform the debumping operation … = ffout --ffout STRING Instead of using the standard canonical naming scheme for residue and atom names, use the names from the given forcefield … Flags Tagged Parameters

28. output-path FILE OUTPUT The desired output name of the PQR file to be generated … inputParam inFile inId true Input file to be used (choose one of the two options) … Untagged Parameters Groups

29. Group 2 Group 3 Exclusive group Input file String Exclusive enumeration PDB2PQR Advanced Submission Form Group 1

30. getAppMetadata launchJob getOutput PDB2PQR getAppMetadata launchJob getOutput MEME Opal server Opal GUI displayForm PDB2PQR getAppMetadata appMetadata submissionForm User On user submit: invokes launchJob with right input parameters 1 5 4 3 2 Usage Scenario

31. Summary Opal enables rapid exposing legacy applications as Web services –Provides features like Job management, Scheduling, Security, and Persistence More information, downloads, documentation: –http://opal.nbcr.net/http://opal.nbcr.net/