OGCE Consortium Reusable Grid Portal Components Mary Thomas

OGCE Consortium Outline The Portal Challenge The Solution: –The Open Grid Computing Environments Consortium (OGCE) The OGCE Portal Architecture Application Portal Examples NMI OGCE Plans

OGCE Consortium The Portal Challenge

OGCE Consortium Portal User Interface Grid Resource Broker Service Grid and Web Protocols Information and Data Services Database Service Database HPC or Compute Cluster Grid Information Services, SRB Portal Client Stub Portal Client Stub Portal Client Stub JDBC, Local, or Remote Connection Three-Tiered Architecture Three-tiered architecture is accepted standard for accessing Grid and other services

OGCE Consortium Portal are Gateways to Grid Early Grid Portals were great success: –Applications oriented, –Provided scientists with some useful tools –Encapsulated some complexities. But they were: –All “stovepipe” solutions. Very little reusable code and infrastructure Nothing could be shared between portals –Many ignored emerging architecture of the Grid A framework based on distributed ubiquitous services

OGCE Consortium Impact of Distributed Grid and Web Services Resource layer 1000s of PCs ->massive supercomputers Grid Portals Launch, configure And control Application Interface Workflow service App Instance Security Data Management Service Data Management Service Accounting Service Accounting Service Logging Event/Mesg Service Event/Mesg Service Policy Administration & Monitoring Administration & Monitoring Grid Orchestration Registries and Name binding Registries and Name binding Reservations And Scheduling Reservations And Scheduling Open Grid Service Architecture Layer Open Grid Service Infrastructure (web service component model) Online instruments (slide by Dennis Gannon)

OGCE Consortium Advantage of Portlet Architecture Portlet/component approach very compatible with Web/Grid services model Each Grid service can be associated with a unique portlet –Very easy to add new services –Many different Groups can contribute portlets which can be plugged into a portal. Currently Indiana, Argonne, Michigan, NCSA,Texas + GridSphere portal group following this model. Most likely others as well. –Jetspeed is Apache standard. Basis for IBM WebSphere portal. Also Oracle, Sun and others. Now a Java Standard (waiting for JSR 168) User can select and configure portlets he/she wishes to use –Selection becomes part of persistent context

OGCE Consortium The Next Generation Grid Portal Building on Standard Technologies –Portlet Design (JSR-168) IBM, Oracle, Sun, BEA, Apache –Grid standards: Java CoG, Web/Grid Services –Web server: JetSpeed (open source) User configurable, Service Oriented –Philosophy: The Portal is a gateway to distributed Grid and Web Services With common API, portlets can be exchanged, interoperate Portal Server MyProxy Server Metadata Directory Service(s) Directory & index Services Application Factory Services Messaging and group collaboration Event and logging Services

OGCE Consortium User Requirements Have Not Changed Users & Clients are: –Portal clients/users –Portal developers Security –Single sign-on, authentication to all grid resources Certificate based / uses MyProxy –VO management File Management –Grid FTP –File staging –Metadata management Remote Job Management –Job submission, scheduling, and tracking. Log and output mgmt. –Workflow management tools

OGCE Consortium User Requirements Information Services –Grid resource and status information directories –Data and metadata catalogs –Personal and VO metadata and service index/directories –Database Interfaces Application Interfaces –Configure and launch workflow from domain specific interface Access to Collaboration Tools –VO collaboration. Synchronous and asynchronous –Access Grid venues Developers need common API

OGCE Consortium The Solution: The Open Grid Computing Environment (OGCE) Consortium

OGCE Consortium Open Grid Computing Environments (OGCE) Consortium Motivation: –Emerging technologies impacting portal developers: Grid/Web services; ubiquitous database use, XMl –Lessons learned from GCE-RG and other projects (GridPort/HotPage, Alliance Portal, DOE Components, NEESGrid, Gateway, GRaDS, others) –Multiple projects repeating similar tasks, work, development; OGCE established in Fall 2003 to foster collaborations and sharable components with portal developers worldwide.

OGCE Consortium OGCE Funded by: –NSF NMI: software releases will be part of NMI and synchronized with GRIDS releases –Also: NCSA Alliance, NPACI, DOE, NASA, others Release Schedule: –First Release: –Next Release: April, 2004 Website:

OGCE Consortium OGCE Tasks/Goals Development of interoperable Grid portal architecture –Define standards for common API’s between portlet service and remote services Establishment of a Grid Portal Collaboratory –Repository of portlet and portal service components An online forum for developers of Grid Portals The building of reusable portal components: –Can be integrated in a common portal container system Contribute to standards organizations: –NSF NMI program  software releases –GGF if appropriate or applicable

OGCE Consortium The OGCE Portal Architecture

OGCE Consortium OGCE Initial Architecture Portal Local Portlets Teamlets Proxy Portlets Jetspeed Internal Services Java COG API Java CoG Kit Grid Services Grid Protocols GRAM, MDS-LDAD MyProxy Service API CHEF Services Remote Interfaces CoG Stubs HTTP Grid Services Other Services SOAP Initial architecture aggregates multiple services into a single portal using portlet containers

OGCE Consortium OGCE Initial Arch: Features & Capabilities OGCE approach encompasses multiple portal development systems: –Preserving development investments Central portal system (Jetspeed) acts as a central portal container: –Portal responsible for managing the lifecycle of pluggable portal components, or portlets. Grid/computational portlets must provide services on the behalf of the user: –Not just a collection of links/static pages User view (from the left) is of a single portal composed of portlets.

OGCE Consortium Supports Aggregation of Diverse Tasks Portlets provide a uniform management interface to the portal (on the left) connect to heterogeneous set of APIs on the right. –First horizontal thread Portlets interact with web interfaces written in other languages such as Perl, Python, PHP, ColdFusion, etc. –Architecture for bridged support is depicted in the second horizontal thread Examples: –Locally running Myproxy Grid portlet uses Java CoG kit to interact with remote Grid services –Job Scheduler portlet uses GridPort Toolkit to submit job via Community Scheduling Frameworks (CSF) service –Teamlets (sharable portlets) share collaboration and Grid services, such as calendars, message boards, file services.

OGCE Consortium OGCE - Revised Architecture Revised portal architecture provides a unified interface for portal services. Supports aggregation of services in backend Portal Portlets and Teamlets Jetspeed Internal Services Service API Grid Protocols Grid Service Stubs Remote Content Services Remote Content Servers HTTP Grid Services Java CoG Kit Local Portal Services

OGCE Consortium Features All services will employ a single (generic) service API that allows them to be deployed and accessed in a uniform fashion. Developer then chooses (during portal configuration) the implementations of particular services. Services may be local, may access remote sites via Grid protocols, or may access remote content, such as web interfaces written in Perl. OGCE Twist: –Entire portal will auto-install and include all required software components

OGCE Consortium Key Technologies???? Core Portal Framework: –CHEF/JetSpeed (U. Mich) Core Grid Technologies: –JavaCoG (provides Globus interface) –GridPort Toolkit (aggregates JavaCoG, and other grid services)

OGCE Consortium Portal/Application Gallery

OGCE Consortium TeraGrid Portal Access Point for TeraGrid Services Version 0: Collecting Initial Services –Public Information about Resources –Private Information for the developers. Version 1: User centered portal –HotPage/GridPort style access to user accounts, credentials, job submission & management. Version 2: Portals for Science Collaborations (Q3 2004) –Shared spaces, whiteboards, AG access, group authorization, shared application services

OGCE Consortium TeraGrid Integrated Architecture Diagram demonstrates how existing software projects (such as GridPort) can be adapted to support NMI Portals software system Portal Portlets and Teamlets Jetspeed Internal Services Grid Service Stubs Remote Content Services Remote Content Servers HTTP Grid Services Java CoG Kit Local Portal Services Service API GridPort Toolkit Web Services

OGCE Consortium Create job description Create job description Store and record results Store and record results Generate and store derived products Generate and store derived products monitoring myMEAD personal information space myMEAD personal information space Broker Service Teragrid data service Teragrid data service Teragrid Globus Toolkit 3.0 GridFTP MetaCatalog Service(MCS) MetaCatalog Service(MCS) NEESgrid Meta- Catalog service NEESgrid Meta- Catalog service Storage Resource Broker (SRB) Storage Resource Broker (SRB) Gather data files, resources Gather data files, resources Execute application Execute application Need ‘x’ put to ‘l’ by time y:00 Need to store ‘y’ and ‘z’ Provides context to resolve ‘x’, ‘y’, ’z’ Retrieve customized run description template from data management services Monitoring results captured in event services persistent channel. myMEAD records address of service and channel Delayed invocation of GridFTP Teragrid Data Service- single point of contact to server-side data servers Alliance scientific portal

OGCE Consortium DOE Fusion Grid Portal Portal Development on the DOE SciDAC Grid –Initially, Fusion Grid Multi-site collaborative project: –TACC, Indiana, SDSC, General Atomics Developing interoperable portal and Web services Portal architecture based on NMI Portal software

OGCE Consortium LEAD Portal

OGCE Consortium CHEF - NEESGrid NEES Tools –Data Viewers (Live/Stored) –Telepresense –Video as Data –Electronic Notebook –Data/Metadata Repository Grid Technologies –Grid sign on using myproxy –Grid computational portal –GridFTP –..Many more

OGCE Consortium OGCE Plans

OGCE Consortium OGCE Portal System System Integration OGSA Globus COG CHEF Toolkit Gridport Alliance Portal Workflow Testing NMI Dist Slide courtesy of Chuck Severance

OGCE Consortium NMI Features List (April 04 Release) Content Management: –Resources: Users and groups can upload and share documents and URLs. –News: Several news feeds are available, including XMLHack and BBC News. –Grid Context: Store arbitrary web objects (movies, web pages, audio files) in a customizable, annotated directory tree. Collaborative: –Chat: Engage other members of your group in online discussion. –Discussion: Participate in discussions with other members of your group –Calendar Program:schedule appointments and attach associated documents and URLs. Calendars can be used to schedule both user and group activities. –Anabas Impromptu: system for real time shared display, audio, and chats.

OGCE Consortium NMI Features List (April 04 Release) GT 3.0/Interactive: –Proxy Manager and Job Submit: Obtain and manage one or more Grid credentials that can be used to access Grid resources through your browser. –GridFTP: Use Grid credential to browse remote directories and to upload and download files between your desktop and a remote host. –Application Manager –Application Factory Manager Remote Job Management –GridAnt for workflow –Job submission portal –Community Scheduling Framework (CSF): submit jobs to scheduler Information Services: –LDAP Browser: –Navigate the LDAP server of your choice. –GridPort Information Repository Portlets (GPIR): Portlets are available for browsing Machine Summaries, System Status, and Total Grid Capabilities. Application Portlets: –Specialized portlets for projects

OGCE Consortium Key Feature – Download a Portal & Install Prerequisite Software –Java 2 Standard Edition version 1.4.x or higher. –Apache Ant version 1.5.x or higher. Preparing the Installation –Optionallly, create a "portal" account on your web server. –Download the portal bundle –Put the download bundle into the home directory of "portal" account. –Untar or unzip the portal bundle. –Globus: installation process will create the directories.globus and.globus/certificates in the Software automatically installed includes: –Globus/Java CoG –CHEF Install the Portal –Move into the portal/chef directory. –Edit file "build.properites" –Type "ant deploy" to build in and install the portal. –Start the Tomcat web server. –Point your browser at :10081/ogce and create an account. OS/Machines Tested: –MAC OSX –PC/Windows XP –Linux –Cygwin

OGCE Consortium Current Status SC03 Demonstration portal and software download available –Beta1 Updated Release available now –Bug fixes –Now supporting GT3.x with J2SDK Working on NMI April 2004 release –Including new features to support science portal applications Open GCE Runtime Engine ( OGRE) workflow services and portlets Sample template portlets for simple science applications –Based on aggregation of existing projects –OGSA/OGSI/GT3.x compatible

OGCE Consortium Future Directions Moving towards new portlet standard –JSR 168 –Should allow compatibility with GridSphere Academic education and business portals Oncourse, OneStart, CHEF (of course), and uPortal Major commercial products offerings from Sun, IBM, Oracle, etc. Incorporate new features: –Grid Services (CSF, OGSA-DAI portlets, Job Sequencer) –Access Grid, Polycom and other AV Portlets Collaborations and Contributions –DOE SciDAC projects (such as CMCS) Expand online repository –Develop process for third party contributions

OGCE Consortium Future Directions Moving towards new portlet standard –JSR 168 –Should allow compatibility with GridSphere Academic education and business portals Oncourse, OneStart, CHEF (of course), and uPortal Major commercial products offerings from Sun, IBM, Oracle, etc. Incorporate new features: –Grid Services (CSF, OGSA-DAI portlets, Job Sequencer) –Access Grid, Polycom and other AV Portlets Collaborations and Contributions –DOE SciDAC projects (such as CMCS) Expand online repository –Develop process for third party contributions

OGCE Consortium Participants Indiana University University of Michigan University of Texas at Austin Argonne National Labs JavaCoG National Center for Supercomputing Applications the globus alliance

OGCE Consortium References OGCE Website – NSF Middleware Initiative –