User interaction and workflow management in Grid enabled e-VLBI experiments Dominik Stokłosa Poznań Supercomputing and Networking Center, Supercomputing.

Slides:

Advertisements

Similar presentations

PIONIER 2003, Poznan, , PROGRESS Grid Access Environment for SUN Computing Cluster Poznań Supercomputing and Networking Center Cezary Mazurek.

Advertisements

Dominik Stoklosa Poznan Supercomputing and Networking Center, Supercomputing Department EGEE 2007 Budapest, Hungary, October 1-5 Workflow management in.

SCARIe: Realtime software correlation Nico Kruithof, Damien Marchal.

Dominik Stokłosa Pozna ń Supercomputing and Networking Center, Supercomputing Department INGRID 2008 Lacco Ameno, Island of Ischia, ITALY, April 9-11 Workflow.

The Australian Virtual Observatory e-Science Meeting School of Physics, March 2003 David Barnes.

European VLBI Network of Radio Telescopes - a killer application for Grid ? - Mike Garrett FP6 GRID Meeting, March 2003.

Global Grid Access Cezary Mazurek, PSNC. Cezary Mazurek, PSNC, Enable access to global grid, Supercomputing 2003, Phoenix, AZ 2 Agenda Introduction PROGRESS.

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.

6th Biennial Ptolemy Miniconference Berkeley, CA May 12, 2005 Distributed Computing in Kepler Ilkay Altintas Lead, Scientific Workflow Automation Technologies.

UNICORE UNiform Interface to COmputing REsources Olga Alexandrova, TITE 3 Daniela Grudinschi, TITE 3.

CHEP 2015 Analysis of CERN Computing Infrastructure and Monitoring Data Christian Nieke, CERN IT / Technische Universität Braunschweig On behalf of the.

ORNL is managed by UT-Battelle for the US Department of Energy Data Management User Guide Suzanne Parete-Koon Oak Ridge Leadership Computing Facility.

SUN HPC Consortium, Heidelberg 2004 Grid(Lab) Resource Management System (GRMS) and GridLab Services Krzysztof Kurowski Poznan Supercomputing and Networking.

Apache Airavata GSOC Knowledge and Expertise Computational Resources Scientific Instruments Algorithms and Models Archived Data and Metadata Advanced.

Hungarian Supercomputing GRID

Poznań city, PSNC - who are we? Poznań city, PSNC - who are we? Introduction into Virtual Laboratory Introduction into Virtual Laboratory VLab architecture.

Marcin Okoń Pozna ń Supercomputing and Networking Center, Supercomputing Department e-Science 2006, Amsterdam, Dec Virtual Laboratory as a Remote.

ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.

UNIT - 1Topic - 2 C OMPUTING E NVIRONMENTS. What is Computing Environment? Computing Environment explains how a collection of computers will process and.

ENTERPRISE COMPUTING QUIZ By: Lean F. Torida

© 2008 Open Grid Forum Independent Software Vendor (ISV) Remote Computing Primer Steven Newhouse.

EDAGRID TUTORIAL Dr. Tuan-Anh Nguyen Ho Chi Minh City University of Technology Prof. Pierre Kuonen University of Applied Sciences of Fribourg, Switzerland.

The PROGRESS Grid Service Provider Maciej Bogdański Portals & Portlets 2003 Edinburgh, July 14th-17th.

7April 2000F Harris LHCb Software Workshop 1 LHCb planning on EU GRID activities (for discussion) F Harris.

Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, October 2003.

PROGRESS: ICCS'2003 GRID SERVICE PROVIDER: How to improve flexibility of grid user interfaces? Michał Kosiedowski.

 Apache Airavata Architecture Overview Shameera Rathnayaka Graduate Assistant Science Gateways Group Indiana University 07/27/2015.

TERENA 2003, May 21, Zagreb TERENA Networking Conference, 2003 MOBILE WORK ENVIRONMENT FOR GRID USERS. TESTBED Miroslaw Kupczyk Rafal.

Update on the Software Correlator Nico Kruithof, Huseyin Özdemir, Yurii Pydoprihora, Ruud Oerlemans, Mark Kettenis, JIVE.

May http://cern.ch/hep-proj-grid-fabric1 EU DataGrid WP4 Large-Scale Cluster Computing Workshop FNAL, May Olof Bärring, CERN.

Scheduling Interactive Tasks in the Grid-based Systems M. Okoń, M. Lawenda, N. Meyer, D. Stokłosa, T. Rajtar, D. Kaliszan, M. Stroiński TERENA Networking.

A B A B AR InterGrid Testbed Proposal for discussion Robin Middleton/Roger Barlow Rome: October 2001.

Westerbork Netherlands Dedicated Gbit link Onsala Sweden Gbit link Jodrell Bank UK Dwingeloo DWDM link Cambridge UK MERLIN Medicina Italy Chalmers University.

EURO-VO: GRID and VO Lofar Information System Design OmegaCEN Kapteyn Institute TARGET- Computing Center University Groningen Garching, 10 April 2008 Lofar.

Hiding Grid Complexity Behind SSH Session Server framework Tomasz Kuczyński (1,2) 1) Poznan Supercomputing and Networking Center.

ISG We build general capability Introduction to Olympus Shawn T. Brown, PhD ISG MISSION 2.0 Lead Director of Public Health Applications Pittsburgh Supercomputing.

Securing the Grid & other Middleware Challenges Ian Foster Mathematics and Computer Science Division Argonne National Laboratory and Department of Computer.

Tier3 monitoring. Initial issues. Danila Oleynik. Artem Petrosyan. JINR.

PROGRESS: GEW'2003 Using Resources of Multiple Grids with the Grid Service Provider Michał Kosiedowski.

Migrating Desktop Uniform Access to the Grid Marcin Płóciennik Poznan Supercomputing and Networking Center Poznan, Poland EGEE’07, Budapest, Oct.

© Geodise Project, University of Southampton, Workflow Support for Advanced Grid-Enabled Computing Fenglian Xu *, M.

Migrating Desktop Uniform Access to the Grid Marcin Płóciennik Poznan Supercomputing and Networking Center Poland EGEE’08 Conference, Istanbul, 24 Sep.

New Workflow Manager Katarzyna Bylec PSNC. Agenda Introduction WLIN Workflows DEMO KIWI Design Workflow Manager System Components descripton ▫ KIWI Portal.

INFSO-RI Enabling Grids for E-sciencE File Transfer Software and Service SC3 Gavin McCance – JRA1 Data Management Cluster Service.

Scenario use cases Szymon Mueller PSNC. Agenda 1.General description of experiment use case. 2.Detailed description of use cases: 1.Preparation for observation.

Shatskaya M, Abramov A., Fedorov N., Kostenko V., Likhachev S., Seliverstov S., Sichev D. Astro Space Center, P. Lebedev Physical Institute, RAS, Moscow,

Introduction to Fabric Kiwi Team PSNC. E-VLBI system – general idea.

EXPReS FABRIC meeting at Poznan, Poland1 EXPReS FABRIC WP 2.2 Correlator Engine Meeting Poznan Poland JIVE, Ruud Oerlemans.

WP7: Computation in a shared infrastructure Mark Kettenis JIVE Presented by Norbert Meyer PSNC.

WfMS and external systems Katarzyna Bylec PSNC. Agenda Introduction Pre-corelation ▫ North Star ▫ NRAO SCHED ▫ Vlbeer FTP ▫ Log2vex ▫ drudg Correlation.

WP7: Computation in a shared infrastructure Mark Kettenis JIVE Presented by Norbert Meyer PSNC.

What is FABRIC? Future Arrays of Broadband Radio-telescopes on Internet Computing Huib Jan van Langevelde, JIVE Dwingeloo.

Distributed Correlation in Fabric Kiwi Team PSNC.

Scientific Data Processing Portal and Heterogeneous Computing Resources at NRC “Kurchatov Institute” V. Aulov, D. Drizhuk, A. Klimentov, R. Mashinistov,

TOG, Dwingeloo, March 2006, A. Szomoru, JIVE Status of JIVE Arpad Szomoru.

WP18, High-speed data recording Krzysztof Wrona, European XFEL

Shatskaya M, Abramov A. , Fedorov N. , Kostenko V. , Likhachev S

Grid Computing.

EGEE NA4 Lofar Lofar Information System Design OmegaCEN

University of Technology

Monitoring of the infrastructure from the VO perspective

Mike Becher and Wolfgang Rehm

Tiers vs. Layers.

Initial job submission and monitoring efforts with JClarens

SCARIe: eVLBI Software Correlation Over Dynamic Lambda Grids

Project Information Management Jiwei Ma

The Design of a Grid Computing System for Drug Discovery and Design

Gordon Erlebacher Florida State University

PolarGrid and FutureGrid

GGF10 Workflow Workshop Summary

Presentation transcript:

User interaction and workflow management in Grid enabled e-VLBI experiments Dominik Stokłosa Poznań Supercomputing and Networking Center, Supercomputing Department

Outline Motivation for distributed correlation E-VLBI System in practice E-VLBI system behind the scene Summary

First glance at distributed correlation Data can be transferred over the network Each stage of the process can be speeded up GRID resources available Software correlator available

E-VLBI System

E-VLBI System – communication

How to conduct an experiment Creating observation workflow using WFM application Submitting observation workflow for execution in the Grid environment Workflow execution Monitoring the VLBI experiment

Observation workflow Constructed: by VLBI operator based on the observation schedule (VEX) with Workflow Manager Application (WFM)

Observation workflow

Submitting the observation workflow

Workflow execution Divided into the following phases: Chunking data from radio telescopes Submitting chunked data sets for correlation Archiving the correlated data sets

(1) Workflow execution: chunking

(1) Workflow execution: chunking

(1) Workflow execution: preparing environment

(1) Workflow execution: sending chunks

(2) Workflow execution: correlation First data chunks are available Step, where the real correlation takes place Computation on the Grid resources

(2) Workflow execution: correlation

scheduling correlation jobs with Correlation Node .. (2) Workflow execution scheduling correlation jobs with Correlation Node ..

(2) Workflow execution: correlation

(3) Workflow execution: archive First data sets have been correlated Store correlated data with Correlated Data Service (CDS) Clean up the Grid resources

(3) Workflow execution: archive

(3) Workflow execution: archive

Workflow monitoring Possibility to monitor the distributed correlation Monitoring perspective in WFM application

Fabric results 25 minutes of data Successful correlation with 2 clusters (PSNC) One File Server (Jive) Stations: Westerbork, Medicina, Torun, Cambridge

Correlation testbed

Correlation time vs chunk size