1. Web services for Production Cyberenvironment for a A Computational Chemistry Grid University of Hyderabad, India 17 March 07 Sudhakar Pamidighantam NCSA, University of Illinois at Urbana-Champaign sudhakar@ncsa.edu

2. Acknowledgements

3. Outline Historical Background Grid Computational Chemistry Production Environments Current Status Web Services Usage Brief Demo Future

4. Motivation Software - Reasonably Mature and easy to use to address chemists questions of interest Community of Users - Need and capable of using the software Some are non traditional computational chemists Resources - Various in capacity and capability

5. Background Qauntum Chemistry Remote Job Monitor ( Quantum Chemistry Workbench) 1998, NCSA Chemviz 1999-2001, NSF (USA) http://chemviz.ncsa.uiuc.edu Technologies Web Based Client Server Models Visual Interfaces Distributed computing (Condor)

6. GridChem NCSA Alliance was commissioned 1998 Diverse HPC systems deployed both at NCSA and Alliance Partner Sites Batch schedulers different at sites Policies favored different classes and modes of use at different sites/HPC systems

7. Extended TeraGrid Facility www.teragrid.org

8. NSF Petascale Road Map Track I Scheme Multi petaflop single site system to be deployed by 2010 Several Consortia Competing (Now under review) Track 2 Sub petaflop systems Several to be deployed until Track 1 is online First one will be at TACC ( 450 TFlops) Available Fall 2007 ( 50 000 Processors/Cores) NCSA is deploying a 110 TFlops in April 2007 (10000 Processors/cores) Second subpetaflops systems being reviewed

9. Grid and Gridlock Alliance lead to Physical Grid Grid lead to TeraGrid Homogenous Grid with predefined fixed software and system stack was planned (Teragrid) but it was difficult to keep it homogenous Local preferences and diversity leads to heterogeneous grids now! (Operating Systems, Schedulers, Policies, Software and Services ) Openness and standards that lead interoperability are critical for successful services

10. Current Grid Status Grid Hardware Middleware Scientific Applications

11. User Community Chemistry and Computational Biology User Base Sep 03 – Oct 04 NRAC AAB Small Allocations ------------------------------------------------------------- #PIs 26 23 64 #SUs 5,953,100 1,374,100 640,000

13. Some User Issues Addressed by the new Services New systems meant learning new commands Porting Codes Learning new job submissions and monitoring protocols New proposals for time (time for new proposals) Computational modeling became more popular and number of users increased (User Management) Batch queues are longer / waiting increased Finding resources where to compute complicated - probably multiple distributed sites Multiple proposals/allocations/logins Authentication and Data Security Data management

14. Computational Chemistry Grid This is a Virtual Organization Integrated Cyber Infrastructure for Computational Chemistry Integrates Applications, Middleware, HPC resources, Scheduling and Data management Allocations, User services and Training

15. Resources System (Site)Procs Avail Total CPU Hours/Year Status Intel Cluster (OSC)36315,000 SMP and Cluster nodes HP Integrity Superdome (UKy) 33290,000 TB Replaced with an SMP/ Cluster nodes IA32 Linux Cluster (NCSA) 64560,000 Allocated Intel Cluster (LSU)10241,000,000 Allocated IBM Power4 (TACC)16140,000 Allocated Teragrid (Multiple Institutions) 2-10000 250,000New Allocations Expected The initial Acesss Grid Testbed Nodes (38) and Condor SGI resources (NCSA, 512 nodes) have been retired this year.

16. Other Resources Extant HPC resources at various Supercomputer Centers (Interoperable) Optionally Other Grids and Hubs/local/personal resources These may require existing allocations/Authorization

18. Grid Middleware Proxy Server GridChem System user Portal Client Grid Services Grid applicationapplication Mass Storage http:// www.nsf.gov/awardsearch/showAward.do?AwardNumber=0438312

19. Applications GridChem supports some apps already –Gaussian, GAMESS, NWChem, Molpro, QMCPack, Amber Schedule of integration of additional software –ACES-3 –Crystal –Q-Chem –Wein2K –MCCCS Towhee –Others...

20. Gridchem Middleware Service (GMS)

21. GrdiChem Web Services Quick Primer XML is used to tag the data, SOAP is used to transfer the data, WSDL is used for describing the services available and UDDI is used for listing what services are available. Web Services is different from Web Page Systems or Web Servers: There is no GUI Web Services Share business logic, data & processes through APIs with each other (not with user) Web Services describe Standard way of interacting with “web based” applications A client program connecting to a web service can read the WSDL to determine what functions are available on the server. Any special datatypesdatatypes used are embedded in the WSDL file in the form of XML Schema. Universal Description, Discovery, and Integration. WSRF Standards Compliant.

22. GridChem Web Services Client  Objects  Database Interaction WS Resources DTO ObjectsHibernate Databasehb.xml Client DTO (Data Transfer Object) Serialize transfer through XML DAO (Data Access Object) How to get the DB objects hb.xml (Hibernate Data Map) describes obj/column data mapping Business Model DAO

23. GridChem Data Models UsersProjectsResources UserProjectResource SoftwareResources ComputeResources NetworkResources StorageResources Resources resoruceID Type hostName IPAddress siteID userID projectID resourceID loginName SUsLocalUserUsed Jobs jobID jobName userID projID softID cost UsersResources

24. Computational Chemistry Resource

25. GMS_WS Use Cases Authentication Job Submission Resource Monitoring Job Monitoring File Retrieval … http://www.gridchem.org:8668/space/GMS/usecase

26. GMS_WS Authentication WSDL (Web Service Definition Language) is a language for describing how to interface with XML-based services. It describes network services as a pair of endpoints operating on messages with either document-oriented or procedure-oriented information. The service interface is called the port type WSDL FILE: <definitions name="MathService" targetNamespace="http://www.globus.org/namespaces/examples/core/MathService_instance" xmlns="http://schemas.xmlsoap.org/wsdl/" … http://www.gridchem.org:8668/space/GMS/usecase Contact GMS Creates Session, Session RP and EPR Sends EPR ( Like a Cookie, but more than that) Login Request (username:passwd) Validates, Loads UserProjects Sends acknowledgement Retrieve UserProjects (GetResourceProperty Port Type [PT]) GridChem ClientGMS

27. GMS_WS Authentication http://www.gridchem.org:8668/space/GMS/usecase Selects project LoadVO port type (w. MAC address) Verifies user/project/MACaddr Load UserResources RP Retrieve UserResources [as userVO/ Profile] (GetResourceProperty port Type PT) GridChem ClientGMS Validates, Loads UserProjects Sends acknowledgement

28. GMS_WS Job Submission Create Job object PredictJobStartTime PT + JobDTO JobStart Prediction RP PT = portType RP = Resource Properties DTO = Data Transfer Object Completion: Email from batch system to GMS server cron@GMS  DB Submission CoGKit GAT “gsi-ssh” If decision OK, SubmitJob PT + JobDTO Create Job object API—Submit Store Job Object Send Acknowledgement Need to check to make sure allocation-time is available. GC ClientGMS

29. GMS_WS Monitoring Parse XML, Display PT = portType RP = Resource Properties DTO = Data Transfer Object DB = Data Base cron@GMS server cron@HPC Servers Job Launcher Notifications VO Admin email parses email  DB (status + cost) Request for Job, Resource Status Alloc. Balance UserResource RP Updated from DB GC ClientGMSResources/Kits/DB Send info Discover Applications (Software Resources) Monitor System Monitor Queues

30. GMS_WS Job Status Job Status jobDTO.status Job Launcher Status Update Estimate Start time Scheduler emails/ notifications Notifications: Client, email, IM GC ClientGMSResources/Kits/DB

31. GMS_WS File Retrieval (MSS) GetResourceProperty PT FileDTO(?) LoadFile PT (project folder+job) Validates project folder owned by user. Send new listing PT = portType RP = Resource Properties DTO = Data Transfer Object MSS = Mass Storage System Job Completion: Send Output to MSS LoadFile PT MSS query UserFiles RP + FileDTO object Retrieve Root Dir. Listing on MSS with CoGKit or GAT or “gsi-ssh” API file request Store locally Create FileDTO Load into UserData RP RetrieveFiles PT (+file rel.path) Retrieve file: CoGKit or GAT or “gsi-ssh” GetResourceProperty PT GC ClientGMSResources/Kits/DB

32. GMS_WS File Retrieval PT = portType RP = Resource Properties DTO = Data Transfer Object MSS = Mass Storage System Create FileDTO (?) Load into UserData RP RetrieveJobOutput PT (+JobDTO) Job Record from DB. Running: from Resource Complete: from MSS Retrieve file: CoGKit or GAT or “gsiftp” GetResourceProperty PT GC ClientGMSResources/Kits/DB

33. GridChem Web Services WSRF (Web Services Resource Framework) Compliant WSRF Specifications: WS-ResourceProperties (WSRF-RP) WS-ResourceLifetime (WSRF-RL) WS-ServiceGroup (WSRF-SG) WS-BaseFaults (WSRF-BF) %ps -aux | grep ws /usr/java/jdk1.5.0_05/bin/java \ -Dlog4j.configuration=container-log4j.properties \ -DGLOBUS_LOCATION=/usr/local/globus \ -Djava.endorsed.dirs=/usr/local/globus/endorsed \ -DGLOBUS_HOSTNAME=derrick.tacc.utexas.edu \ -DGLOBUS_TCP_PORT_RANGE=62500,64500 \ -Djava.security.egd=/dev/urandom \ -classpath /usr/local/globus/lib/bootstrap.jar: /usr/local/globus/lib/cog-url.jar: /usr/local/globus/lib/axis-url.jar org.globus.bootstrap.Bootstrap org.globus.wsrf.container.ServiceContainer -nosec Logging Configuration Where to find Globus Where to get random seed for encryption key generation Classpath (required jars)

34. GridChem Software Organization Open Source Distribution CVS for GridChem

35. Package: org.gridchem.service.gms GMS_WS

36. + Should these each be a separate package?

37. model dto credential job notification filefile.task job.task user exceptions resource persistence synch query test util dao gpir crypt enumerators gat proxy GMS_WS client audit gms Classes for WSRF service implementation (PT) Cmd line tests to mimic client requests Data Access Obj – queries DB via persistent classes (hibernate) Data Transfer Obj – (job,File,Hardware,Software,User) XML How to handle errors (exceptions) CCG Service business mode (how to interact) Contains user’s credentials for job sub. file browsing,… “ Oversees correct” handling of user data (get/putfile). Define Job & util & enumerations (SubmitTask, KillTask,…) CCGResource&Util, Synched by GPIR, abstract classes NetworkRes., ComputeRes., SoftwareRes., StorageRes., VisualizationRes. User (has attributes – Preference/Address) DB operations (CRUD), OR Maps, pool mgmt,DB session, Classes that communicate with other web services Periodically update DB with GPIR info (GPIR calls) JUnit service test (gms.properties): authen. VO retrieval, Res.Query,Synch, Job Mgmt, File Mgmt, Notification Contains utility and singleton classes for the service. Encryption of login password Mapping from GMS_WS enumeration classes  DB GAT util classes: GATContext & GAT Preferences generation Classes deal with CoGKit configuration. Autonomous notification via email, IM, textmesg.

38. GMS_WS external jars Testing For XML Parsing “Java” Document Object Model –Lightweight –Reading/Writing XML Docs –Complements SAX (parser) & DOM –Uses Collections**

39. Authentication

40. Resource Status

41. Job Editor

42. Job Submission

43. Job Monitoring

44. Gradient Monitoring

45. Energy Monitoring

46. Post Processing

47. Visualization Molecular Visualization Electronic Properties Spectra Vibrational Modes

48. Molecular Visualization Better molecule representations (Ball and Stick/VDW/MS) In Nanocad Molecular Editor Third party visualizer integration Chime/VMD Export Possibilities to others interfaces Deliver standard file formats (XML,SDF,MSF,Smiles etc…)

49. Eigen Function Visualization Molecular Orbital/Fragment Orbital MO Density Visualization MO Density Properties Other functions Radial distribution functions

50. Some example Visuals Arginine Gamess/6-31G* Total electronic density 2D - Slices

51. Electron Density in 3D Interactive (VRML)

52. Orbital 2D Displays N2 6-31g* Gamess

53. Orbital 3D VRML

54. Spectra IR/Raman Vibrotational Spectra UV Visible Spectra Spectra to Normal Modes Spectra to Orbitals

55. GridChem Use Allocation Community and External Registration Consulting/User Services Ticket tracking, Allocation Management Documentation Training and Outreach FAQ Extraction, Tutorials, Dissemination

56. Users and Usage 170 Users Include Academic PIs, two graduate classes And about 15 training users NCSA 57000 SUs + A 7 node dedicated system UKy around 106766 SUs OSC 13,820 SUs + A 14 node dedicated system Usage at LSU and TACC as well More than a 335000 CPU Wallhours since Jan 06.

57. Science Enabled Chemical Reactivity of the Biradicaloid (HO...ONO) Singlet States of Peroxynitrous Acid. The Oxidation of Hydrocarbons, Sulfides, and Selenides. Bach, R. D.; Dmitrenko, O.; Estévez, C. M. J. Am. Chem. Soc. 2005, 127, 3140-3155. The "Somersault" Mechanism for the P-450 Hydroxylation of Hydrocarbons. The Intervention of Transient Inverted Metastable Hydroperoxides. Bach, R. D.; Dmitrenko, O. J. Am. Chem. Soc. 2006, 128(5), 1474-1488. The Effect of Carbonyl Substitution on the Strain Energy of Small Ring Compounds and their Six-member Ring Reference Compounds Bach, R. D.; Dmitrenko, O. J. Am. Chem. Soc. 2006,128(14), 4598.

58. Science Enabled Azide Reactions for Controlling Clean Silicon Surface Chemistry: Benzylazide on Si(100)-2 1 Semyon Bocharov, Olga Dmitrenko, Lucila P. Mendez De Leo, and Andrew V. Teplyakov* Department of Chemistry and Biochemistry, UniVersity of Delaware, Newark, Delaware 19716 Received April 13, 2006; E-mail: andrewt@udel.edu http://pubs.acs.org.proxy2.library.uiuc.edu/cgi- bin/asap.cgi/jacsat/asap/pdf/ja0623663.pdf [May require ACS access] http://pubs.acs.org.proxy2.library.uiuc.edu/cgi- bin/asap.cgi/jacsat/asap/pdf/ja0623663.pdf

59. Possible H-bonds network for P450 cam hydroperoxy intermediate Suggested: THR252 accepts an H-bond from the hydroperoxy (Fe(III)- OOH that promotes the second protonation on the distal oxygen, leading to the O-O bond cleavage Nagano, S.; Poulos, T.L. J. Biol. Chem. 2005, 250, p.1668 Auclair, K.; Hu, Z.; Little, D. M.; Ortiz de Montellano, P. R.; Groves, J. T. J. Am. Chem. Soc. 2002, 124, 6020.

60. The Somersault Isomerization of Model Cpd0 Robert Bach and Olga Dmytrenko, 2006

61. Energy Diagram for the Concerted Non-synchronous Hydroxylation of Isobutane Energy diagram (kcal/mol) for the oxidation of the isobutane with ground state, 24a (GS-8 hydrogen bonded to isobutane). MIN-24b [model oxidant MIN-10 (PorFe(SH)O  HO) hydrogen bonded to isobutene] is not necessarily on the reaction pathway.

62. Somersault Mechanism Summary for Isobutane Hydroxylation

63. Unsymmetrical Mo(CO) 4 Crown Ethers

64. Dibenzaphosphepin based  bis(phosphorous)polyether chelated Mo(CO) 4

65. Crystal Structures CSD:XAPZAP cis-(6,6'-((1,1'-Binaphthyl)-2,2'- diylbis(oxy))bis(dibenzo(d,f)(1,3,2)dioxaphosp hepin))-tetracarbonyl-molybdenum(0) C48 H28 Mo1 O10 P2 CSD:DEQDOS cis-Tetracarbonyl-(P,P'-(6-(2'-oxy-2-biphenyl)-3,6- dioxa-hexanolato)-bis(dibenzo (d,f)(1,3,2)dioxaphosphepine)-P,P')-molybdenum C44 H32 Mo1 O12 P2

66. Reference Structure for Comparison

67. Starting Structure

68. Optimized Structure

69. Reference Structure for Comparison 8 7

70. Structural Comparisons C-C Torsion Angles for the OCH 2 CH 2 O Fragments and for the Axially Chiral Biaryl Groups Atoms PCMODEL* UFF Ab Initio Amber C37-C42-C43-C48 -49.9 -26.4 -43.0 -40.4 C1-C6-C7-C12 45.4 22.3-22.3 -72.8 C13-C22-C23-C32 75.6 74.7-85.9 -81.2 C32-O-C33-C34 -178.4 -140.8 159.7 -171.2 O-C33-C34-O 62.4 -64.5 -87.3 -82.4 C33-C34-O-C35 -80.6 -118.9 67.8 64.9 C34-O-C35-C36 174.6 118.9 -153.4 60.1 O-C35-C36-0 66.2 56.0 64.0 67.3 *Hariharasarma, et al. Organomet., 1232-1238, 2000. Ab Initio=B3LYP/3-21G* Amber9 ff03, GAFF, chloroform, 300K, median over 1ns MD

71. MD OCH 2 CH 2 O Structure 8 7

72. MD Biaryl Structure

73. 1 H NMR Chemical Shift Comparison For Aromatic Protons Reference 32ppm (from TMS B3LYP/6-31g*) Atom Exp.AbinitioAtomExp.Abinitio H27.0255.6H256.5785.7 H37.0265.8H266.7375.9 H47.0495.9H277.0186.1 H57.1816.0H287.6236.5 H87.1106.1H307.7906.7 H96.8906.0H317.2896.9 H106.7216.0 H116.2375.7H387.3276.2 H397.2746.1 H147.9255.8H407.1696.0 H157.8086.3H417.3506.3 H177.7416.0H447.3606.1 H187.2545.6H457.1605.9 H197.0915.1H467.1766.0 H206.9894.6H477.0607.0

74. Third Year Plans Post Processing Spectra and related entities New Application Support Aces3, Dmol3, Vasp Expansion of Resources Teragrid, OSG, Pragma Systems and New resources at Partner Sites Extension Plan Two Proposals in review for Extension

75. Future Plans Preparations for Petaflop computing High throughput massively parallel applications Complex workflows for integrating multiple interdependent applications Multiscale Computing Archiving and annotating data for future use Open Data initiatives by NIH and NSF

76. Acknowledgments Rion Dooley, TACC Middleware Infrastructure Stelios Kyriacou, OSC Middleware Scripts Chona Guiang, TACC Databases and Applications Kent Milfeld, TACC Database Integration Kailash Kotwani, NCSA, Applications and Middleware Scott Brozell, OSC, Applications and Testing Michael Sheetz, UKy, Application Interfaces Vikram Gazula, UKy, Server Administration Tom Roney, NCSA, Server and Database Maintenance