PROGRESS – Computing Portal and Data Management in the Cluster of SUNs Michał Kosiedowski Sun HPC Consortium Heidelberg 2003.

Slides:



Advertisements
Similar presentations
POZNAN SUPERCOMPUTING AND NETWORKING CENTER Poznan Supercomputing and Networking Center Portals and Content Cezary Mazurek, Andrzej.
Advertisements

PIONIER 2003, Poznan, , PROGRESS Grid Access Environment for SUN Computing Cluster Poznań Supercomputing and Networking Center Cezary Mazurek.
LEAD Portal: a TeraGrid Gateway and Application Service Architecture Marcus Christie and Suresh Marru Indiana University LEAD Project (
W w w. h p c - e u r o p a. o r g HPC-Europa Portal: Uniform Access to European HPC Infrastructure Ariel Oleksiak Poznan Supercomputing.
The National Grid Service and OGSA-DAI Mike Mineter
Digital Object Lifecycle in dLibra Digital Library Framework Cezary Mazurek, Marcin Werla
Global Grid Access Cezary Mazurek, PSNC. Cezary Mazurek, PSNC, Enable access to global grid, Supercomputing 2003, Phoenix, AZ 2 Agenda Introduction PROGRESS.
Seminar Grid Computing ‘05 Hui Li Sep 19, Overview Brief Introduction Presentations Projects Remarks.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
PROGRESS: ICWS'2003 Web Services Communication within the PROGRESS Grid-Portal Environment Michał Kosiedowski.
Chapter 9: Moving to Design
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
W w w. h p c - e u r o p a. o r g The HPC-Europa project and GridSphere Dawid Szejnfeld Poznan Supercomputing.
The SAM-Grid Fabric Services Gabriele Garzoglio (for the SAM-Grid team) Computing Division Fermilab.
“This presentation is for informational purposes only and may not be incorporated into a contract or agreement.”
SUN HPC Consortium, Heidelberg 2004 Grid(Lab) Resource Management System (GRMS) and GridLab Services Krzysztof Kurowski Poznan Supercomputing and Networking.
- 1 - Grid Programming Environment (GPE) Ralf Ratering Intel Parallel and Distributed Solutions Division (PDSD)
Chapter 9 Elements of Systems Design
Copyright © cs-tutorial.com. Introduction to Web Development In 1990 and 1991,Tim Berners-Lee created the World Wide Web at the European Laboratory for.
OASIS ebXML Registry Standard Open Forum 2003 on Metadata Registries 10:30 – 11:15 January 20, 2003 Kathryn Breininger The Boeing Company Chair, OASIS.
Integration of the Biological Databases into Grid-Portal Environments Michal Kosiedowski, Michal Malecki, Cezary Mazurek, Pawel Spychala, Marcin Wolski.
Flexibility and user-friendliness of grid portals: the PROGRESS approach Michal Kosiedowski
Grids and Portals for VLAB Marlon Pierce Community Grids Lab Indiana University.
Grid Resource Allocation and Management (GRAM) Execution management Execution management –Deployment, scheduling and monitoring Community Scheduler Framework.
3rd June 2004 CDF Grid SAM:Metadata and Middleware Components Mòrag Burgon-Lyon University of Glasgow.
Information Grid Services in the Polish Optical Internet PIONIER Cezary Mazurek, Maciej Stroiński, Jan Węglarz.
1 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre The Data Portal Glen Drinkwater.
A Model of the Environment for Flexible Access to Complex Distributed Applications Michal Kosiedowski
GLOBAL GRID FORUM 10 Workflows in PROGRESS and GridLab environments Michał Kosiedowski.
Facilitating access to the scientific data service with the use of the Data Management System Cezary Mazurek
The PROGRESS Grid Service Provider Maciej Bogdański Portals & Portlets 2003 Edinburgh, July 14th-17th.
Migrating Desktop The graphical framework for running grid applications Bartek Palak Poznan Supercomputing and Networking Center The.
1 Welcome to CSC 301 Web Programming Charles Frank.
GEM Portal and SERVOGrid for Earthquake Science PTLIU Laboratory for Community Grids Geoffrey Fox, Marlon Pierce Computer Science, Informatics, Physics.
Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.
Application portlets within the PROGRESS HPC Portal Michał Kosiedowski
Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, October 2003.
PROGRESS: ICCS'2003 GRID SERVICE PROVIDER: How to improve flexibility of grid user interfaces? Michał Kosiedowski.
Migrating Desktop Marcin Płóciennik Marcin Płóciennik Kick-off Meeting, Santander, Graphical.
Grid Architecture William E. Johnston Lawrence Berkeley National Lab and NASA Ames Research Center (These slides are available at grid.lbl.gov/~wej/Grids)
Metadata harvesting in regional digital libraries in PIONIER Network Cezary Mazurek, Maciej Stroiński, Marcin Werla, Jan Węglarz.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
9 Systems Analysis and Design in a Changing World, Fourth Edition.
A university for the world real R © 2009, Chapter 9 The Runtime Environment Michael Adams.
GRID Overview Internet2 Member Meeting Spring 2003 Sandra Redman Information Technology and Systems Center and Information Technology Research Center National.
Terena conference, June 2004, Rhodes, Greece Norbert Meyer The effective integration of scientific instruments in the Grid.
The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.
Migrating Desktop Bartek Palak Bartek Palak Poznan Supercomputing and Networking Center The Graphical Framework.
Module: Software Engineering of Web Applications Chapter 2: Technologies 1.
Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing
SUPERCOMPUTING 2002, Baltimore, , SUN „Grid Day” PROGRESS Access environment to computational services performed by cluster of SUNs Poznań Supercomputing.
GRID ANATOMY Advanced Computing Concepts – Dr. Emmanuel Pilli.
Partnerships in Innovation: Serving a Networked Nation Grid Technologies: Foundations for Preservation Environments Portals for managing user interactions.
PEPC 2003, Geneva, , PROGRESS Computing Portal Poznań Supercomputing and Networking Center (PSNC) Poland Poland Cezary Mazurek.
A computer contains two major sets of tools, software and hardware. Software is generally divided into Systems software and Applications software. Systems.
PROGRESS: GEW'2003 Using Resources of Multiple Grids with the Grid Service Provider Michał Kosiedowski.
BalticGrid-II Project EGEE UF’09 Conference, , Catania Partner’s logo Framework for Grid Applications Migrating Desktop Framework for Grid.
Migrating Desktop Uniform Access to the Grid Marcin Płóciennik Poznan Supercomputing and Networking Center Poznan, Poland EGEE’07, Budapest, Oct.
© Geodise Project, University of Southampton, Workflow Support for Advanced Grid-Enabled Computing Fenglian Xu *, M.
Migrating Desktop Uniform Access to the Grid Marcin Płóciennik Poznan Supercomputing and Networking Center Poland EGEE’08 Conference, Istanbul, 24 Sep.
Remote Api Tutorial How to call WS-PGRADE workflows from remote clients through the http protocol?
1 Copyright © 2008, Oracle. All rights reserved. Repository Basics.
InSilicoLab – Grid Environment for Supporting Numerical Experiments in Chemistry Joanna Kocot, Daniel Harężlak, Klemens Noga, Mariusz Sterzel, Tomasz Szepieniec.
9 Systems Analysis and Design in a Changing World, Fifth Edition.
University of Technology
Knowledge Based Workflow Building Architecture
Analysis models and design models
Data Management Components for a Research Data Archive
Introduction to the SHIWA Simulation Platform EGI User Forum,
Presentation transcript:

PROGRESS – Computing Portal and Data Management in the Cluster of SUNs Michał Kosiedowski Sun HPC Consortium Heidelberg 2003

Sun HPC Consortium R & D Center PSNC was established in 1993 and is an R&D Center in: –New Generation Networks POZMAN and PIONIER networks 6-NET, SEQUIN, ATRIUM projects –HPC and Grids GRIDLAB, CROSSGRID, PROGRESS projects –Portals and Content Delivery Tools Polish Educational Portal Multimedia City Guide, dLibra framework

Sun HPC Consortium Center of Excellence PSNC became the Sun CoE in New Generation Networks, Grids and Portals in November 2002

Sun HPC Consortium PROGRESS Duration: December 2001 – May 2003 (R&D) Budget: ~4,0 MEuro Project Partners –SUN Microsystems Poland –PSNC IBCh Poznań –Cyfronet AMM, Kraków –Technical University Łódź Co-funded by The State Committee for Scientific Research (KBN) and SUN Microsystems Poland

Sun HPC Consortium PROGRESS (2) Cluster of 80 processors Networked Storage of 1,3 TB Software: ORACLE, HPC Cluster Tools, Sun ONE, Sun Grid Engine

Sun HPC Consortium PROGRESS – architecture

Sun HPC Consortium PROGRESS – the talk HPC Window – the access environment to grid resources: user interfaces (Computing Portal and Migrating Desktop) + Grid Service Provider Data Management System – a distributed system designed for storing scientific data used in experiments performed in the computing cluster (grid)

Sun HPC Consortium HPC Window - motivation grid access environments lacked flexibility: one grid -> one portal user must have relied on the grid as far as the history is concerned user interfaces were not that much functional

Sun HPC Consortium Grid-Portal Environment PORTAL HPC RESOURCES GRID MANAGEMENT SYSTEM GRID SERVICE PROVIDER 4-tier new grid-portal environment PORTAL HPC RESOURCES GRID MANAGEMENT SYSTEM 3-tier classical grid-portal environment

Sun HPC Consortium PROGRESS GPE Content Provider Webservice Session Bean Entity Beans

Sun HPC Consortium PROGRESS HPC Portal interaction with users presentation of data obtain from services

Sun HPC Consortium PROGRESS HPC Portal (2)

Sun HPC Consortium PROGRESS HPC Portal (3)

Sun HPC Consortium PROGRESS HPC Portal (4)

Sun HPC Consortium PROGRESS HPC Portal (5)

Sun HPC Consortium PROGRESS HPC Portal (6)

Sun HPC Consortium PROGRESS HPC Portal (7)

Sun HPC Consortium PROGRESS HPC Portal (8)

Sun HPC Consortium Grid Service Provider the use of the grid resources more comfortable to the end users allows for easy building of numerous portals and other user interfaces; users can switch from one to another and use the same GSP services various thematic scientific web portals sharing the same grid resources possibility of providing all clients (user interfaces) with computing resources belonging to two or more different grids

Sun HPC Consortium Grid Service Provider (2) Necessary services to provide: –job submission service managing the creation of user jobs, their submission to the grid and the monitoring of their execution (typically through reverse reporting performed by the Grid Management System about events connected with the execution of jobs) –application management service storing information about applications available for running in the grid assisting application developers in adding new applications to the application repository –provider management service keeping up-to-date information on the services available within the provider

Sun HPC Consortium GSP: Job submission service computing job building, submitting them to the grid for execution and viewing the results job description is prepared using the XRSL language and transferred to the grid resource broker for the execution of the job grid resource broker reverse reports on grid events connected with the job "workflowed" jobs: sequences and parallels

Sun HPC Consortium GSP: Application mgmt. srv. application repository management application descriptor contains a reference to the application executable: a reference to a file stored in the DMS or a path to a binary on grid computing server filesystems also included in the application descriptor: available (required or optional) arguments, required environment variables and required input and output files applications in PROGRESS may be unconfigured or configured: one executable -> multiple configured applications virtual applications

Sun HPC Consortium GSP: Provider mgmt. service enables keeping up-to-date information on services available in the grid service provider a service descriptor contains information on the Web Service interface: URL at which the service is available, the service namespace reference (URN) and the service WSDL reference services may have multiple instances: informational services

Sun HPC Consortium GSP: Informational services examples of instance enabled services intended for use by web portals PROGRESS example: short news service other: document directory, discussion forum (under development)

Sun HPC Consortium GSP: XRSL Language Extended Resource Specification Language (XRSL) is an XML based language designed for description of computing jobs the XML documents describing grid computing jobs are passed to the grid resource broker, which analyzes them and executes jobs in accordance with requirements included

Sun HPC Consortium Data Mgmt. Syst. - motivation data management systems were not oriented towards grid access environment data management systems concentrated on distributing data between grid computers

Sun HPC Consortium Data Management System Data Broker Data Storage Mirror & Proxy Data Storage Metadata Repository SRS WS GASS FTP Grid FTP (...) Clients Portal Grid broker Migrating desktop

Sun HPC Consortium Data Management System (2) provides seamless access to data and information for grid computing uses metadata for describing stored data stores data on various media such as files, tapes and databases serves as the source of input data and the destination for the results of computing experiments

Sun HPC Consortium DMS: Data broker Data Broker Data Storage Mirror & Proxy Data Storage Metadata Repository SRS WS GASS FTP Grid FTP (...) Clients Portal Grid broker Migrating desktop

Sun HPC Consortium DMS: Data broker (2) serves as an interface (Web Services based) for external clients, such as the HPC Portal and the grid resource broker delivers DMS functions for directory, file and metadata management

Sun HPC Consortium DMS: Data broker (3) directory mgmt.: add, remove and rename directories, retrieve root and current path, change path, list contents file mgmt.: add, remove and rename files, add, remove and retrieve physical location, add and remove archives, add and remove symbolic links metadata mgmt.: metadata scheme mgmt., retrieve list of schemes and attributes, assign schemes to files and edit values and attributes, search metadata repository

Sun HPC Consortium DMS: Metadata repository Data Broker Data Storage Mirror & Proxy Data Storage Metadata Repository SRS WS GASS FTP Grid FTP (...) Clients Portal Grid broker Migrating desktop

Sun HPC Consortium DMS: Metadata repository (2) responsible for storing and managing metadata format of the metadata scheme associated with a file can be defined by the user or chosen from the predefined formats like the Dublin Core

Sun HPC Consortium DMS: Data storage modules Data Broker Data Storage Mirror & Proxy Data Storage Metadata Repository SRS WS GASS FTP Grid FTP (...) Clients Portal Grid broker Migrating desktop

Sun HPC Consortium DMS: Data storage mod. (2) enable access to physical data data are arranged in data containers and can be stored on all media types and accessed by a uniform interface data can be organized as files on generic filesystems, BLOBs in databases or files on data tapes GASS, GridFTP and FTP as the data transport protocols

Sun HPC Consortium DMS: Mirror & proxy module Data Broker Data Storage Mirror & Proxy Data Storage Metadata Repository SRS WS GASS FTP Grid FTP (...) Clients Portal Grid broker Migrating desktop

Sun HPC Consortium DMS: Mirror & proxy mod. (2) enables access to external scientific databanks mirrors earlier defined sets of data (like the SRS databank in PROGRESS) serves as a proxy to internet based data resources (with caching functions)

Sun HPC Consortium Web Services Communication HPC Portal Grid Service Provider Data Management System Grid Resource Broker saveJob() getApplications() saveTaskOfJob() saveStdOfTask() submitJob() getUserJobs() getJobStatus() listUserDirectory() addUserFile() getUserFileLocation() submitJob() changeJobStatus()

Sun HPC Consortium Authentication & authorization utilize the services available within the Sun One Portal Server 6.0 package: authentication techniques, user database, portlet access control, identity server design an authorization system for the grid service provider and the data management system: based on the RAD model apply a Single Sign-On mechanism

Sun HPC Consortium Authorization scheme Portal GRID SERVICE PROVIDER Identity server RAD based authorization system Logon Authentication Request Method invocation Token validation Resource access authorization

Sun HPC Consortium Visualization of job results

Sun HPC Consortium Where to go now? Project: Research & Development finished; the test and deployment phase now We will continue the R&D on the tools, including the grid service provider and the data management system

Sun HPC Consortium Where to go now? (2) PROGRESS HPC Portal is a bioinformatic thematic portal: other thematic scientific portals possible to deploy, using the same grid service provider, data management system and grid resources and utilizing the same portal tools

Sun HPC Consortium Where to go now? (3) The grid service provider may be equipped with means of communication with multiple grids: –cooperation with GRIDLAB –perhaps some Sun Grid Engine based grids?

Sun HPC Consortium Where to go now? (4) encourage application developers to add their applications to the PROGRESS repository encourage visualization module developers to add their software to the PROGRESS repository encourage bioinformatic experts to take over as information services editors

Sun HPC Consortium PROGRESS