Clarens Toolkit Building Blocks for a Simple TeraGrid Gateway Tutorial Conrad Steenberg Julian Bunn, Matthew Graham, Joseph Jacob, Craig Miller, Roy Williams California Institute of Technology TeraGrid 2007 Madison, WI

June 2007TeraGrid Introduction Clarens Toolkit architecture Shell service basics User mapping Job submission Web interface hosting

June 2007TeraGrid Clarens Toolkit Architecture High-performance Apache web server “handler” Written from the ground up in Python since 2001 Multi-protocol communication (XML- RPC/JSON/SOAP) Geared towards scientists – requires very little programming to create web services Supports X509/proxy-based authentication Role-based and Gridmap file based authorization File Upload/download – AJAX hosting See presentation on Tuesday by Julian Bunn

June 2007TeraGrid Clarens Architecture IIClient Web server Web serverService 3 rd party application Clarens Clarens http/https XML-RPC SOAP Java RMI JSON/RPC The Clarens Toolkit Supports: Authentication – User certificates and proxies Access control – to Web Services, remote files Discovery - Web Services and Software Shell – Remote shells with ACL Virtual Organization – membership, management and roles Services – Potentially infinite :-) JavaScript (AJAX), Java, Python, Root (analysis framework)

June 2007TeraGrid Shell Service Basics Shell service can be used to run an executable on the server –Mapping user's credentials to a local system user External API: –id=shell.cmd(“some command”, id) –Each command executed in a sandbox owned by local system user –stdout and stderr files created in sandbox –Sandboxes can be re-used by supplying optional id argument –Files can be uploaded and downloaded to sandbox via web service or HTTP PUT using a form in a browser

June 2007TeraGrid Shell Service Basics II

June 2007TeraGrid User Mapping Map credentials (X509/proxy) to local system user Use classical gridmap-file – "/DC=org/DC=doegrids/OU=People/CN=Conrad Steenberg " conrad Use external callout Site Authorization Service Use Clarens hierarchical VO-based mapping –Hierarchical group structure –Any user or group can be mapped to local user –E.g. “Caltech” group consisting of Roy, Joe, Julian mapped to “caltech” user on remote machine No credential can be mapped to web server user

June 2007TeraGrid Job Submission Client-side framework can be developed using only shell service basics –Needs to be repeated for every access method and every client –Higher level services can be developed using code that make up shell service as a library –Higher level service can then be exposed to outside world E.g. rootlet service Higher level services can provide complex user mapping –E.g. NVO has anonymous, weak, and strong certificates mapping to guest account, nvo account or individual account resp.

June 2007TeraGrid Job Submission II Shell service library use on server: –Write service using tutorial on clarens.sf.net –Import shell service as library: shell_mod=clarens_util.import_module(“shell”) –Obtain desired user mapping: user_name,uid,gid,dn=shell_mod.lookup_user(req) –Execute shell command: shell_mod.exe_worker(req, method_name, cmd, request_id, user_name) –Or wrap user executable with script to submit to PBS/LSF/Condor

June 2007TeraGrid Web Interface Hosting Clarens server handles ordinary web page hosting –Can host html/js files to build AJAX interfaces –Use any popular client-side toolkit (yahoo, google, dojo, etc.) –Clarens provides support for method invocation from javascript –Uses same authentication/authorization for every HTTP GET/PUT request Example: – –jsonrpc("shell.cmd", [“date”,id], date_callback, error_cb)

June 2007TeraGrid Web Interface Hosting II

June 2007TeraGrid Summary Clarens Toolkit allows portal builders a flexible way to build web services –Uses wrist-friendly Python language –Simple design and implementation –Open Source Supports advanced portal requirements –Range of user mapping methods from simple to complex –Powerful access control mechanisms –Web interface hosting Examples of use include –NESSSI services –rootlets in HEP

June 2007TeraGrid Contact Info Clarens: –Complete manual –One-liner installation for x86-Linux –Service tutorial –Bug-tracking/mailing lists NVO: –Example services

June 2007TeraGrid

June 2007TeraGrid

June 2007TeraGrid

June 2007TeraGrid