1. OGCE Consortium Component-Based Portals for Grid Computing Marlon Pierce Community Grids Lab Indiana University

2. OGCE Consortium NSF NMI Project for Reusable Portal Components: Who We Are University of Chicago –Gregor von Laszewski Indiana University –Marlon Pierce, Dennis Gannon, Geoffrey Fox, and Beth Plale University of Michigan –Charles Severance, Joseph Hardin NCSA/UIUC –Jay Alameda, Joe Futrelle Texas Advanced Computing Center –Mary Thomas

3. OGCE Consortium What Is Grid Computing? Grid Computing provides an overlay infrastructure that can be used to bind computing and data resources from multiple organizations into “virtual organizations”. –Security, information services, resource access protocols, file transfer, etc. Open Grid Services Architecture recasts Grid capabilities as Web Services –WSDL descriptive conventions, advanced features for transient services, etc. –Service hosting environments manage service lifecycles, interactions with requestor agents. But what about the clients? –And what about user centric services?

4. OGCE Consortium Towards A Common Grid Client Hosting Environment Grid portal background and emerging common frameworks

5. OGCE Consortium What Is a Computing Portal? Browser based user interface for accessing grid and other services –“Live” dynamic pages for accessing grid services –Use(d) Java/Perl/Python COGs –Manage credentials, launch jobs, manage files, etc. –Hide Grid complexities –Can run from anywhere –Unlike user desktop clients, connections go through portal server, so overcome firewall/NAT issues Combine “Science Grid” with traditional web capabilities –Get web pages for news feeds –Post and share documents –And other more traditional web page features Customizable interfaces and user roles/views

6. OGCE Consortium Let 10000 Flowers Bloom Many portal projects have been launched since late ’90s. –HotPage from SDSC, NCSA efforts, DOD, DOE Portals, NASA IPG –2002 Special Issue of Concurrency and Computation Continue to be important component of many large projects –NEESGrid, DOE SciDAC projects, NASA, NSF, many international efforts Global Grid Forum’s Grid Computing Environments Research Group –Community forum

7. OGCE Consortium Portal User Interface Grid Resource Broker Service Grid and Web Protocols Information and Data Services Database Service Database HPC or Compute Cluster Grid Information Services, SRB Portal Client Stub Portal Client Stub Portal Client Stub JDBC, Local, or Remote Connection Three-Tiered Architecture Three-tiered architecture is accepted standard for accessing Grid and other services

8. OGCE Consortium Problem with Portals GCE revealed two things –Everyone was doing the same thing Not quite, but significant Everyone builds secure logins, remote file manipulation, command execution, access to info servers. Everyone would at least like support for multiple user roles (administrators, users) and customization –No one could share components with other groups No well defined way of sharing UI components or making services interoperate. No well defined interfaces to portal services. A research opportunity! –Two levels of integration: user interfaces and services Our challenges –Stop reinventing things and provide ways for groups to reuse components. –Provide a portal marketplace for competing (advanced) services. –Provide APIs for service integration

9. OGCE Consortium A Solution based on components A software component is object defined by –A precise public interface –A semantics that includes a set of “standard” behaviors. A Software component architecture is: –A a set of rules for component behavior & –A framework in which components can be easily installed and interoperate. The component architecture of choice for the Portal community is the one based on portlets –Java components that generate content, make local and remote connections to services. –Portal containers manage portlet lifecycles

10. OGCE Consortium A Portlet Approach to Grid Services A Portlet is a portal server component that provides basic services rendered in a user-configurable window in a portal pane. Portal Server MyProxy Server Metadata Directory Service(s) Directory & index Services Application Factory Services Messaging and group collaboration Event and logging Services Portlet 1 Portlet 2 Portlet 3 Portlet 4 Portlet 5 Portlet 6

11. OGCE Consortium The Grid Portal Provides Portlets for –Management of user proxy certificates –Remote file Management via Grid FTP –News/Message systems for collaborations –Grid Event/Logging service –Access to OGSA services –Access to directory services –Specialized Application Factory access Distributed applications Workflow –Access to Metadata Index tools User searchable index

12. OGCE Consortium OGCE Foundations: Portal Containers and Grid Access

13. OGCE Consortium Portlet Component and Container Technologies Jakarta Jetspeed –Open source Java portlet project –Jetspeed is both a framework and reference implementation –Defines portlets, portal service APIs (login, authorization, customization, etc.) CHEF from University of Michigan –Uses Jetspeed as a framework Reimplements many of the core classes –Basis for UM CourseTools –NEESGrid portal –CMCS Portal

14. OGCE Consortium Background CHEF is organized around groups of users Portals in CHEF are group based (a group can consist of only one person!) A user sees the Portals for each group of which that person is a member The Portal is a collection of Portal pages Each Portal page contains one of more teamlets

15. OGCE Consortium Portal Engine: Jetspeed Velocity CHEF Teamlets: Written in JAVA Responsible for GUI Operate in the context of a session. Rely on services for any persistent or “cross-user” information. Services Persistent System-wide Multiple implementations of services Configurable as to what implementation provides what service Servlets: Access services outside of the portal engine: AccessServlet and WebDavServlet Services API CHEF Architecture Web Server: Tomcat Turbine Non-HTTP Components (i.e. E-Mail)

16. OGCE Consortium What is a Teamlet? A teamlet is a portal-like presentation of information and possible user actions It can be placed in multiple places with a portal in across multiple portals; each placement is independent Each placement is configurable Each placement belongs to a portal; and is therefore associated with a group

17. OGCE Consortium Design Process - Elements The design of a teamlet consists of three elements –A service (the Java class or classes that implement the interface to a source/store of information) –An action (the Tool in CHEF; one of more Java classes that present information to the user and respond to user actions) –The GUI (usually a set of Velocity templates)

18. OGCE Consortium Java CoG Kit Provides interfaces to elementary Grid functionality –Copy a file from here to there –Execute a remote job on the Grid –Authenticate to the Grid Provides interfaces to more advanced Grid functionality such as simple job queues and task graphs Provides a convenient API level interface that protects you from many changes in the Grid such as GT1.x to GT2.x to GT3.x to GT4.x

19. OGCE Consortium What does the user see? Portlet Java CoG Kit High-Level GT2GT3GT4CondorSSH Java CoG Kit Low-Levelinterface Portal Interface

20. OGCE Consortium Portal Capabilities A survey of current portal capabilities.

21. Portal CapabilitiesDescription Grid Proxy Certificate ManagerGet MyProxy certs after logging in. ScheduleInteractive individual and group calendars DiscussionPersistent topic-based discussion for groups ChatLive chat services and interfaces Document managersWEBDav based document system for group file sharing MDS/LDAP BrowsersBasic Globus MDS browsing and navigating GridContext PortletsAccess context services for managing metadata GRAM Job SubmissionRun simple executables on remote hosts GridFTPUpload, download, crossload remote files. GPIR PortletsView, interact with HPC status, job, etc information. AnabasAccess to Anabas shared display applets Newsgroups and citation portletsPost topics to newsgroup, manage group references and citations with access controls User Portlets

22. OGCE Consortium Grid Portlet Examples We’ll next overview several portal capabilities. Jetspeed/CHEF acts as a clearing house for portal capabilities –User interface components can be added in well defined ways. –First level of integration All Grid access goes through the Java COG.

23. OGCE Consortium Example Capability: Portals for Users The MyProxy Manager –The user contacts the portal server and asks it to do “grid” things on behalf of the user. –To make this possible the server needs a “Proxy Certificate” The user has previously stored a proxy cert in a secure MyProxy Server stored with a temporary password. User give the portal server the password and the portal server contacts the proxy server and loads the proxy. The portal server will hold the proxy for the user for a “short amount of time” in the user’s session state. Portal Server 1. Load my Proxy Certificate! User “Beth” MyProxy Server 2. Give me Beth’s proxy certificate I am Beth’s Proxy 3. COG MyProxy Portlet

24. OGCE Consortium Java COG Example Capability: File Management Grid FTP portlet– Allow User to manage remote file spaces –Uses stored proxy for authentication –Upload and download files –Third party file transfer Request that GridFTP server A send a file to GridFTP server B Does not involve traffic through portal server Portal Server User “Beth” GridFTP Server A GridFTP Server B GridFTP portlet

25. OGCE Consortium Example Capability: Grid Context Service User’s want to be able to use the portal to keep track of lots of things –Application and experiment records File metadata, execution parameters, workflow scripts –“Favorite” services Useful directory services, indexes, links to important resources –Notes and annotations “Scientific Notebooks”

26. OGCE Consortium XDirectory: A Grid Context Service XDirectory is itself a Grid Service that is access by the portal. –An index over a relational database –Each node is either a “directory node” or a leaf. –Leaf nodes are xml elements which contain metadata as well as html annotations.

27. OGCE Consortium Portlet Interfaces to Grid Context Services A Remote Service Directory Interface –Holds references and metadata about application services. User selects interface to application service from the directory browser. Examples: (near completion) –Select a link to a Dagman document and invoke the Condor service on the script. –Same for GridAnt/Ogre or BPEL workflow script. –Factory services for any grid apps that have interactive user interfaces. Portal Server Portal Server Remote Grid Application Service Remote Service Directory Service

28. OGCE Consortium Example Capability: Topic Based Messaging Systems Indiana University has implemented a XML metadata system based on messages. Newsgroups –Topic based posting and administration Citation/reference browsers –Topic based, export/import bibtex Portlets sit atop JMS-based message system.

29. OGCE Consortium

30. OGCE User Privileges for Group Channels Users request access to specific topics/channels. –Granted by administrator for that topic Can request –Read/write by browser –Read/write by email (newsgroups) –Receive/don’t receive attachments. Topic admin can edit these requests. Super admins can manage administrators to topics

31. OGCE Consortium GPIR Data Load - aggregated CPU Downtime data for a machine –Jobs: aggregated queue MOTD Nodes: job usage for each machine node NWS: based on VO and Click model Grid Monitoring –Based on TACC GMS System –Custom providers –Plans to include MDS3.0 and INCA data uderway Expanding to include: –queuing system –application profiles –performance data –Application profiles –Doc links Model allows generic inclusion of any XML data from any recognized source –Need schema –Need query

32. OGCE Consortium Grid Portal Information Repository (GPIR 1.1)

33. OGCE Consortium GPIR Components Web Services Ingestor –Web Services Ingestor and clients –XML Schemas - can be changed Data Repository –Local Cache –Archival --> PostgreSQL Web Service Query –retrieve data – XML Queries –Retrieving current snapshot and archived data Clients –GridPort services –Portal/Web Interface (Portlets, servlets, JSP) –Command line –Any that speak web services

34. OGCE Consortium Future Capabilities

35. OGCE Consortium Major Theme: Grid Application Support Current portal’s job submission capabilities are vanilla –Type desired machine, executable, output file –Generates RSL, runs command Actual job management requires more –Integration of information, scheduling services, file transfers, job sequencers, events

36. OGCE Consortium Capability: Job Sequencer Portlets User uses Portal to generate XML description of sequence. " xsi:schemaLocation="http://grids.tacc.utexas.edu/schemas/sequencer/jobSequence C:\DOCUME~1\Maytal\Desktop\Maytal\Work\GP-IR\GP-IRX~1\motd.xsd"> < New Unscheduled CSFJob http://129.116.218.36:15080/ogsa/services/metascheduler/ JobFactoryService normal pam -g 1 mpichp4_wrapper /home/monitor/mpi_jobs/mpimd_5 /home/monitor/mpi_jobs 4 /dev/null /home/monitor/mpi_jobs/tomislavSequencerJobOut /home/monitor/mpi_jobs/tomislavSequencerJobErr GridFTP [Previous] blanco.tacc.utexas.edu:2811 /home/monitor/mpi_jobs/tomislavSequencerJobOut /home/monitor/mpi_jobs/tomislavSequencerJobOutCopied /ho me/monitor/mpi_jobs/tomislavSequencerJobErr /home/monitor/mpi_jobs/tomislavSequencerJobErrCopied Currently, sequence steps can consist of File Transfers and Job Submissions to the CSF meta scheduler GPIR The XML is then decomposed and persisted to GPIR where the status information of each step in the sequence and of the sequence as a whole can be stored Sequencer GridPort returns a Sequence ID to the Portal immediately and then begins executing the Sequence to completion or to error. Status information can be obtained at any time with the Sequence ID

37. OGCE Consortium Capability: Community Scheduling Framework Portlets CSF Use Case Researcher submits job through User Portal User Portal uses GridPort to –authenticate user –optionally make advanced reservation to visualization system –submit job to CSF CSF selects compute cluster with best fit and forwards job Gridport sends results to visualization system User Workstation User Portal GridPort CSF Visualization System Bandera Blanco Buda

38. OGCE Consortium O.G.R.E.—A Job Management Engine See Thursday Demo O.G.R.E. = Open Grid Computing Environments Runtime Engine What Ant lacked, but we needed: Broader conditional execution, Ant: based on write-once String properties. A general “loop” structure for Task execution. Data-communication between Tasks (and with their containers). Specialized tasks File reading and writing Local and remote file management (gridftp) Web service related tasks Event- and process-monitoring-tasks

39. OGCE Consortium Data and Metadata Management When the job is through… Simulations, experiments generate both data and metadata –Metadata includes from code input parameters, host machines, data formats, owners of data, generators of data,… NEESGrid metadata system will be integrated into the portal release. Another example of integrated Grid services –GridFTP, CAS and other services

40. OGCE Consortium Metadata Repository Capabilities Data store –Files –Logical naming –Format translation Metadata store –Structured (RDF-like schemas) –Random-access (tuple store) –Version control Archiving –Mass store –“nar” archive format Security –Single signon –Secure reliable file transfer with GridFTP –Authorization via CAS Grid service interfaces –NFMS: NEESgrid File Management Service –NMDS: NEESgrid Metadata Service –Repo. service (Façade) –Secure remote access by applications

41. OGCE Consortium Portal Repository architecture NFMS NMDS File system GridFTP Repo browser File xfer servlet user repository HTTPS JDBC, File I/O Repo. service (Façade) GSDL, GridFTP Java API, GridFTP NMDS DB CAS CAS DB

42. Access Grid and Related Portlets

43. OGCE Consortium Architectural Upgrades Portlet standards, service managers, event standards

44. OGCE Consortium “A Bag of Portlets…” Portlet/container systems provide a simple level of user interface integration. –A clearing house for pluggable components of all sorts User interfaces are actually to a diverse set of backend services. –A mixture of UIs to Web services, grid services, communication/collaboration services,…. We are a portlet marketplace… But we need closer integration

45. OGCE Consortium OGCE Initial Architecture Portal Local Portlets Teamlets Proxy Portlets Jetspeed Internal Services Java COG API Java CoG Kit Grid Services Grid Protocols GRAM, MDS-LDAD MyProxy Service API CHEF Services Remote Interfaces CoG Stubs HTTP Grid Services Other Services SOAP Initial architecture aggregates multiple services into a single portal using portlet containers

46. OGCE Consortium Integration Points and Service Abstractions Internal portal service abstractions –Service layer abstractions to define how to interact with in-memory proxy certificates. Authorization –Internal and external roles need to be integrated. Events –Share events between services –Job submissions should automatically update the calendar operation, for example

47. OGCE Consortium TeraGrid Integrated Architecture Diagram demonstrates how existing software projects (such as GridPort) can be adapted to support NMI Portals software system Portal Portlets and Teamlets Jetspeed Internal Services Grid Service Stubs Remote Content Services Remote Content Servers HTTP Grid Services Java CoG Kit Local Portal Services Service API GridPort Toolkit Web Services

48. OGCE Consortium Portlet Standards Current portal uses Jetspeed portlet API Other portlet systems available –Websphere->GridSphere Portlet standard: JSR 168 –A common API for all next generation portlet systems. –Compliant portlet components may be shared between systems. Open Source Implementation (Pluto) is available –We will be adopting this, will be part of our SC2004 release –Will leverage education portal work from CHEF team.

49. OGCE Consortium OGCE Portals in Action Some early applications

50. OGCE Consortium New Starts: TeraGrid Portal Access to TeraGrid Services –Version 0: Collecting Initial Services Public Information about Resources Private Information for the developers. –Version 1: A User centered portal (Q2 2004) Hotpage/Gridport style access to user accounts, credentials, job submission & management. –Version 2: Portals for Science Collaborations (Q3 2004) Shared spaces, whiteboards, AG access, group authorization, shared application services

51. OGCE Consortium LEAD Portal

52. OGCE Consortium OGCE Collaboratory We Can’t Do It All We’ll Take Credit For It, Though

53. OGCE Consortium OGCE Collaboratory Hopefully, we have convinced you to not rebuild portals from scratch. –Time to use pluggable components in consistent frameworks. Our award is not just to release our own software. –We want to foster the portal community –Contributed third party components will be sought. Initial contributions will be from similar projects –CMCS and other SciDAC projects are closely allied

54. OGCE Consortium Additional Information OGCE Web site: www.ogce.orgwww.ogce.org –Download the portal software –Join news lists, get announcements OGCE Demo Portal: www.collab- ogce.orgwww.collab- ogce.org –See our demo Thursday night Contact us –marpierc@indiana.edumarpierc@indiana.edu