As computer network experiments increase in complexity and size, it becomes increasingly difficult to fully understand the circumstances under which a.

Slides:



Advertisements
Similar presentations
INDIANAUNIVERSITYINDIANAUNIVERSITY GENI Global Environment for Network Innovation James Williams Director – International Networking Director – Operational.
Advertisements

Experiment Provenance: Towards Links to Network Measurement Data Mehmet Aktas, Beth Plale, Scott Jensen Data to Insight Center Indiana University.
Data Publishing Service Indiana University Stacy Kowalczyk April 9, 2010.
Pulan Yu School of Informatics Indiana University Bloomington Web service based Varuna.Net.
Abstraction Layers Why do we need them? –Protection against change Where in the hourglass do we put them? –Computer Scientist perspective Expose low-level.
TSpaces Services Suite: Automating the Development and Management of Web Services Presenter: Kevin McCurley IBM Almaden Research Center Contact: Marcus.
ASCR Data Science Centers Infrastructure Demonstration S. Canon, N. Desai, M. Ernst, K. Kleese-Van Dam, G. Shipman, B. Tierney.
Lifemapper Provenance Virtualization
A Java Architecture for the Internet of Things Noel Poore, Architect Pete St. Pierre, Product Manager Java Platform Group, Internet of Things September.
Connect. Communicate. Collaborate Click to edit Master title style MODULE 1: perfSONAR TECHNICAL OVERVIEW.
NetKarma Portal Chris Small. Portal Goals Make it much easier for experimenters to capture provenance data with experiment Integrate with: – Measurement.
Workshop on Cyber Infrastructure in Combustion Science April 19-20, 2006 Subrata Bhattacharjee and Christopher Paolini Mechanical.
Ch 12 Distributed Systems Architectures
1 Alternate Title Slide: Presentation Name Goes Here Presenter’s Name Infrastructure Solutions Division Date GIS Perfct Ltd. Autodesk Value Added Reseller.
Community Manager A Dynamic Collaboration Solution on Heterogeneous Environment Hyeonsook Kim  2006 CUS. All rights reserved.
Sponsored by the National Science Foundation netKarma Spiral 2 Year-end Project Review Indiana University Beth Plale (PI) School of Informatics and Computing.
Chapter 3 Database Architectures and the Web Pearson Education © 2009.
GMD German National Research Center for Information Technology Innovation through Research Jörg M. Haake Applying Collaborative Open Hypermedia.
@2011 Mihail L. Sichitiu1 Android Introduction Platform Overview.
18:15:32Service Oriented Cyberinfrastructure Lab, Grid Deployments Saul Rioja Link to presentation on wiki.
National Science Foundation Arlington, Virginia January 7-8, 2013 Tom Lehman University of Maryland Mid-Atlantic Crossroads.
Institute of Computer and Communication Network Engineering OFC/NFOEC, 6-10 March 2011, Los Angeles, CA Lessons Learned From Implementing a Path Computation.
San Diego Supercomputer Center Grid Physics Network (GriPhyN) University of Florida Dataflows in SRB using SDSC Matrix Arun Jagatheesan Architect & Team.
San Diego Supercomputer Center SDSC Storage Resource Broker Data Grid Automation Arun Jagatheesan et al., San Diego Supercomputer Center University of.
INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
Through the development of advanced middleware, Grid computing has evolved to a mature technology in which scientists and researchers can leverage to gain.
Crystal-25 April The Rising Power of the Web Browser: Douglas du Boulay, Clinton Chee, Romain Quilici, Peter Turner, Mathew Wyatt. Part of a.
Towards Low Overhead Provenance Tracking in Near Real-Time Stream Filtering Nithya N. Vijayakumar, Beth Plale DDE Lab, Indiana University {nvijayak,
Contents 1.Introduction, architecture 2.Live demonstration 3.Extensibility.
OOI CI LCA REVIEW August 2010 Ocean Observatories Initiative OOI Cyberinfrastructure Architecture Overview Michael Meisinger Life Cycle Architecture Review.
Monitoring Windows Server 2012
Phase II Additions to LSG Search capability to Gene Browser –Though GUI in Gene Browser BLAST plugin that invokes remote EBI BLAST service Working set.
Grid Execution Management for Legacy Code Applications Grid Enabling Legacy Code Applications Tamas Kiss Centre for Parallel.
Large Scale Nuclear Physics Calculations in a Workflow Environment and Data Provenance Capturing Fang Liu and Masha Sosonkina Scalable Computing Lab, USDOE.
San Diego Supercomputer Center Grid Physics Network (GriPhyN) University of Florida DGL: The Assembly Language for Grid Computing Arun swaran Jagatheesan.
Policy Based Data Management Data-Intensive Computing Distributed Collections Grid-Enabled Storage iRODS Reagan W. Moore 1.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
Grid programming with components: an advanced COMPonent platform for an effective invisible grid © 2006 GridCOMP Grids Programming with components. An.
Presented by Scientific Annotation Middleware Software infrastructure to support rich scientific records and the processes that produce them Jens Schwidder.
ANKITHA CHOWDARY GARAPATI
GRID Overview Internet2 Member Meeting Spring 2003 Sandra Redman Information Technology and Systems Center and Information Technology Research Center National.
Presented by Jens Schwidder Tara D. Gibson James D. Myers Computing & Computational Sciences Directorate Oak Ridge National Laboratory Scientific Annotation.
Cooperative experiments in VL-e: from scientific workflows to knowledge sharing Z.Zhao (1) V. Guevara( 1) A. Wibisono(1) A. Belloum(1) M. Bubak(1,2) B.
Experiences with Measurement Data Collection on R&E Networks Christopher Small Indiana University.
Recording Actor Provenance in Scientific Workflows Ian Wootten, Shrija Rajbhandari, Omer Rana Cardiff University, UK.
August 2003 At A Glance The IRC is a platform independent, extensible, and adaptive framework that provides robust, interactive, and distributed control.
Development of e-Science Application Portal on GAP WeiLong Ueng Academia Sinica Grid Computing
Ceilometer + Gnocchi + Aodh Architecture
GRID ANATOMY Advanced Computing Concepts – Dr. Emmanuel Pilli.
Steven Perry Dave Vieglais. W a s a b i Web Applications for the Semantic Architecture of Biodiversity Informatics Overview WASABI is a framework for.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI How to integrate portals with the EGI monitoring system Dusan Vudragovic.
Click to edit Master title style Click to edit Master text styles –Second level Third level –Fourth level »Fifth level 1 CustomerSoft ESP Contact Operations.
XMC Cat: An Adaptive Catalog for Scientific Metadata Scott Jensen and Beth Plale School of Informatics and Computing Indiana University-Bloomington Current.
OGCE Workflow and LEAD Overview Suresh Marru, Marlon Pierce September 2009.
Sponsored by the National Science Foundation 1 March 15, 2011 GENI I&M Update: I&M Service Types, Arrangements, Assembling Goals Architecture Overview.
CIMA and Semantic Interoperability for Networked Instruments and Sensors Donald F. (Rick) McMullen Pervasive Technology Labs at Indiana University
Use-cases for GENI Instrumentation and Measurement Architecture Design Prasad Calyam, Ph.D. (PI – OnTimeMeasure, Project #1764) March 31.
Grid Execution Management for Legacy Code Architecture Exposing legacy applications as Grid services: the GEMLCA approach Centre.
Physical Oceanography Distributed Active Archive Center THUANG June 9-13, 20089th GHRSST-PP Science Team Meeting GHRSST GDAC and EOSDIS PO.DAAC.
Cyberinfrastructure Overview of Demos Townsville, AU 28 – 31 March 2006 CREON/GLEON.
© Copyright IBM Corporation 2016 Diagram Template IBM Cloud Architecture Center Using the Diagram Template This template is for use in creating a visual.
All Hands Meeting 2005 BIRN-CC: Building, Maintaining and Maturing a National Information Infrastructure to Enable and Advance Biomedical Research.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,
POW MND section.
Database Architectures and the Web
Monitoring of the infrastructure from the VO perspective
San Diego Supercomputer Center University of California, San Diego
GENI Global Environment for Network Innovation
About Thetus Thetus develops knowledge discovery and modeling infrastructure software for customers who: Have high value data that does not neatly fit.
Presentation transcript:

As computer network experiments increase in complexity and size, it becomes increasingly difficult to fully understand the circumstances under which a experiment was run. The provenance of an experiment is its lineage or historical trace [7] that can capture experiment conditions, time ordering, and relationships within the experiment and across the experiment and infrastructure layer. The Karma [5] project at Indiana University is capturing the provenance of experimental activity on the Global Environment for Network Innovations (GENI [3]).  Karma [8] is a standalone provenance collection tool first used in the Linked Environments for Atmospheric Discovery (LEAD) project [2] and since rewritten and applied in pharmaceutical discovery and satellite imagery ingest pipelines as well as in GENI.  Karma is either a standalone web service or a daemon that can be added to existing cyberinfrastructure for purposes of collection and representation of provenance data. It utilizes a modular architecture that permits support for multiple instrumentation plugins that make it usable in different architectural settings, including Axis2 handlers, Adapters which parse log files, and more recently RabbitMQ[6].  GENI Adaptor: an collection client that ingests GENI experiment logs and uses a set of rules to derive provenance notifications that are stored to the Karma repository. The adaptor is comprised of two sub-units:  Log Parser  Notification Generator  RabbitMQ is an Enterprise-class messaging systems that implements the Advanced Message Queuing Protocol (AMQP) [9], an open standard for messaging middleware.  NetKarma Portal: Control, query, and visualize provenance. The Karma framework [4] consists of four layers:  Notification Creation Layer that enables a set of client libraries to generate the provenance notification,  Notification Collection Layer that uses either a web service model or a publish/subscribe model to collect provenance notification,  Notification Process Layer that stores, processes, and correlates notifications against Karma repository, and  Representation Layer that enriches the provenance data and provides a query interface over the provenance repository for access and visualization. Karma: Provenance Aggregation Across Layers of GENI Experimental Networks Beth Plale 1,2, Yuan Luo 1,2 1 School of Informatics and Computing, Indiana University, Bloomington, IN USA 2 Data to Insight Center, Pervasive Technology Institute, Indiana University, Bloomington, IN USA {plale, Overview Karma Server Experiment Repository Provenance Collection Architecture Karma Framework GENI Provenance Adaptor GENI Experimental Tool Provenance Repository GENI Provenance Adaptor Publish Provenance Notification … RabbitMQ Message Bus Publish Provenance Notification NetKarma Portal Provenance Query/Visualization Publish Provenance Notification GENI is a virtual laboratory for at-scale networking experimentation including slice creation, topology of the slice, operational status and other measurement statistics and correlate it with the experimental data. The GENI framework has several layers of the GENI stack: 1)An experimental layer where application code is deployed across the slice and execution is triggered; 2)a control plane which exposes a network topology through assignment of a slice to a user or group of users; and 3)a measurement plane which traces network traffic, for instance GENI Instrumentation and Measurement Systems (GIMS) [3]. GENI GENI Control Plane GENI Measurement Plane Notification Creation Layer Notification Generator Karma Messaging Client RabbitMQ Service Message Receiver Daemon Karma DB Ingester Provenance Repository Asynchronized Notification Processor Karma Query Interface Provenance Client Client Notification Collection Layer Notification Process Layer Provenance Representation Layer Web Service (Client) Web Service (Server) Provenance is collected from multiple sources to obtain a picture of GENI experiments and conditions of experiment including:  Experimental tool commands (RAVEN and GUSH)  Topology created using the control frameworks (PlanetLab)  Operational status on substrate /infrastructure  Code and data contained in the experimental slice  Measurement data obtained  Annotations by experimenters Collection Sources [1]B. Chun et al., “Planetlab: an overlaytestbed for br oad-coverage services”. SIGCOMM Computer Communication Review, [2]K. Droegemeier et al., "Service-oriented environments for dynamically interacting with mesoscale weather", Computing in Science and Engineering, IEEE Computer Society Press and American Institute of Physics, Vol. 7, No. 6, pp , [3]Global Environment for Network Innovations, “GENI Spiral 2 Overview”, GENI-INF-PRO-S2-OV- 1.1, June 3, [4]Karma Provenance Collection Tool, [5]NetKarma, [6]RabbitMQ, [7]Yogesh L. Simmhan, Beth Plale, and Dennis Gannon, “A survey of data provenance in e-Science”, ACM SIGMOD Record, Vol. 34, No. 3, September [8] Yogesh Simmhan, Beth Plale, and Dennis Gannon, 2008: Karma2: Provenance Management for Data Driven Workflows. International Journal of Web Services Research, IGI Publishing, Vol 5, No 2, [9] Vinoski, S. Advanced message queuing protocol. IEEE Internet Computing, 10(6), 87-89, For more information:  Data to Insight Center at Pervasive Technology Institute  School of Informatics and Computing, Indiana University References GENI Experiment Life Cycle Outside the Scope of GENI GENI Tools and Services support these phases of the lifecycle Experiment Design S/w and H/w Development (Lab) Experimenter Registration Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting