Next Generation Network Monitoring Prepared by: Les Cottrell SLAC

Slides:



Advertisements
Similar presentations
Network Systems Sales LLC
Advertisements

Joint CASC/CCI Workshop Report Strategic and Tactical Recommendations EDUCAUSE Campus Cyberinfrastructure Working Group Coalition for Academic Scientific.
High Performance Computing Course Notes Grid Computing.
EInfrastructures (Internet and Grids) US Resource Centers Perspective: implementation and execution challenges Alan Blatecky Executive Director SDSC.
1 Cyberinfrastructure Framework for 21st Century Science & Engineering (CIF21) NSF-wide Cyberinfrastructure Vision People, Sustainability, Innovation,
Connect. Communicate. Collaborate Click to edit Master title style MODULE 1: perfSONAR TECHNICAL OVERVIEW.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
Introduction and Overview “the grid” – a proposed distributed computing infrastructure for advanced science and engineering. Purpose: grid concept is motivated.
MAGGIE NIIT- SLAC On Going Projects Measurement & Analysis of Global Grid & Internet End to end performance.
Next Generation Network Monitoring for Pakistan: Proposal Prepared by: Les Cottrell SLAC, Arshad Ali NIIT For Prof. Dr. Atta-ur-Rehman, Chairman of HEC.
LOGO 1 MAGGIE Measurement & Analysis of the Global Grid & Internet End-to-End Performance Monitoring A Research Collaboration by National University of.
Network Monitoring grid network performance measurement, simulation & analysis Presented by Warren Matthews at the Performance.
SharePoint Portal Server 2003 JAMES WEIMHOLT WEIDER HAO JUAN TURCIOS BILL HUERTA BRANDON BROWN JAMES WEIMHOLT INTRODUCTION OVERVIEW IMPLEMENTATION CASE.
Open Cloud Sunil Kumar Balaganchi Thammaiah Internet and Web Systems 2, Spring 2012 Department of Computer Science University of Massachusetts Lowell.
Network Topologies.
Assessment of Core Services provided to USLHC by OSG.
3 Cloud Computing.
1 ESnet Network Measurements ESCC Feb Joe Metzger
GEANT Performance Monitoring Infrastructure – Joint Techs meeting July Nicolas Simar GEANT’s Performance Monitoring.
Microsoft Active Directory(AD) A presentation by Robert, Jasmine, Val and Scott IMT546 December 11, 2004.
PingER Project Arguably the world’s most extensive active end-to-end Internet Performance Project –Digital Divide emphasis –Partially funded by MoST, US.
Performance Monitoring - Internet2 Member Meeting -- Nicolas Simar Performance Monitoring Internet2 Member Meeting, Indianapolis.
SOA Management Packs & Governance Cheat Sheet (Shared under OPN NDA - Last Updated: 8/3/2009)OPN NDA Target Account Profile Enterprises that: Have IT infrastructure.
The Network Performance Advisor J. W. Ferguson NLANR/DAST & NCSA.
Internet2 Performance Update Jeff W. Boote Senior Network Software Engineer Internet2.
1 Measuring Circuit Based Networks Joint Techs Feb Joe Metzger
DataTAG Research and Technological Development for a Transatlantic Grid Abstract Several major international Grid development projects are underway at.
Stanford University, SLAC, NIIT, the Digital Divide & Bandwidth Challenge Prepared by Les Cottrell, SLAC for the NIIT, March 9, 2007.
Connect. Communicate. Collaborate Implementing Multi-Domain Monitoring Services for European Research Networks Szymon Trocha, PSNC A. Hanemann, L. Kudarimoti,
Russ Hobby Program Manager Internet2 Cyberinfrastructure Architect UC Davis.
Measurement & Analysis of Global Grid & Internet End to end performance (MAGGIE) Network Performance Measurement.
Connect communicate collaborate GÉANT3 Services Connectivity and Monitoring Services by and for NRENs Ann Harding, SWITCH TNC 2010.
Perspectives on Grid Technology Ian Foster Argonne National Laboratory The University of Chicago.
Connect. Communicate. Collaborate GÉANT2 and the GRID Domenico Vicinanza DANTE EGEE 08 Meeting, Istanbul September 2008.
Developer TECH REFRESH 15 Junho 2015 #pttechrefres h Understand your end-users and your app with Application Insights.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
1 Network Measurement Summary ESCC, Feb Joe Metzger ESnet Engineering Group Lawrence Berkeley National Laboratory.
GRID Overview Internet2 Member Meeting Spring 2003 Sandra Redman Information Technology and Systems Center and Information Technology Research Center National.
Ruth Pordes November 2004TeraGrid GIG Site Review1 TeraGrid and Open Science Grid Ruth Pordes, Fermilab representing the Open Science.
Connect. Communicate. Collaborate Click to edit Master title style PERT OPERATIONS.
NORDUnet Nordic Infrastructure for Research & Education Workshop Introduction - Finding the Match Lars Fischer LHCONE Workshop CERN, December 2012.
PerfSONAR-PS Functionality February 11 th 2010, APAN 29 – perfSONAR Workshop Jeff Boote, Assistant Director R&D.
Introduction to Grids By: Fetahi Z. Wuhib [CSD2004-Team19]
13-Oct-2003 Internet2 End-to-End Performance Initiative: piPEs Eric Boyd, Matt Zekauskas, Internet2 International.
Jeremy Nowell EPCC, University of Edinburgh A Standards Based Alarms Service for Monitoring Federated Networks.
Internet2 End-to-End Performance Initiative Eric L. Boyd Director of Performance Architecture and Technologies Internet2.
Cyberinfrastructure: Many Things to Many People Russ Hobby Program Manager Internet2.
PerfSONAR-PS Working Group Aaron Brown/Jason Zurawski January 21, 2008 TIP 2008 – Honolulu, HI.
Ellis Paul Technical Solution Specialist – System Center Microsoft UK Operations Manager Overview.
DICE: Authorizing Dynamic Networks for VOs Jeff W. Boote Senior Network Software Engineer, Internet2 Cándido Rodríguez Montes RedIRIS TNC2009 Malaga, Spain.
July 19, 2004Joint Techs – Columbus, OH Network Performance Advisor Tanya M. Brethour NLANR/DAST.
Cyberinfrastructure Overview of Demos Townsville, AU 28 – 31 March 2006 CREON/GLEON.
Connect communicate collaborate perfSONAR MDM News Domenico Vicinanza DANTE (UK)
Cloud Computing 3. TECHNOLOGY GUIDE 3: Cloud Computing 2 Copyright John Wiley & Sons Canada.
Campana (CERN-IT/SDC), McKee (Michigan) 16 October 2013 Deployment of a WLCG network monitoring infrastructure based on the perfSONAR-PS technology.
1 Deploying Measurement Systems in ESnet Joint Techs, Feb Joseph Metzger ESnet Engineering Group Lawrence Berkeley National Laboratory.
1 Network Measurement Challenges LHC E2E Network Research Meeting October 25 th 2006 Joe Metzger Version 1.1.
Bob Jones EGEE Technical Director
Internet2 End-to-End Performance Initiative
Networking for the Future of Science
Robert Szuman – Poznań Supercomputing and Networking Center, Poland
Grid Computing.
Internet2 Performance Update
3 Cloud Computing.
MAGGIE NIIT- SLAC On Going Projects
Interoperable Measurement Frameworks: Internet2 E2E piPEs and NLANR Advisor Eric L. Boyd Internet2 17 April 2019.
“Detective”: Integrating NDT and E2E piPEs
Internet2 E2E piPEs Project
OU BATTLECARD: WebLogic Server 12c
Presentation transcript:

Next Generation Network Monitoring Prepared by: Les Cottrell SLAC

Needs Advancements in networks improve scientific collaborations, help accelerate discoveries –E.g. High Energy Physics (HEP), seismology, tele-medicine, astro-physics, global weather, education … Modern science relies on global Internet –Data exchange, interaction & teleconferencing, Grids … Network problems have increased significance for science Thus dependent on cyberinfrastructure to support efficient network problem diagnosis along paths traversing multiple network domains –This is an unresolved issue today –Hard to overstate amount of effort today to resolve problems Often duplicated Scientists forced to become part-time network engineers

Why is this hard? Internet very diverse, hard to find “invariants, phone models do not work Constantly changing both short and long-term –Changes are not smooth but usually in steps, findings may be out of date No central organization –Scientific communities span multiple organizations in many countries –Typical path requires crossing at least 5 administrative domains (campus, regional, backbone, regional and campus) –Domains are autonomous Measurement not high on vendor’s priorities ISP’s concerned about privacy, competitive advantage, public embarrassment Diagnosis hard: –Convince ADs there is a problem and that they could/should help –Need multiple pieces of information from multiple sources (ends, multiple middles…), with no coordinating body

Besides Pathology Service Level Agreements (setting up and auditing) Planning, setting expectations Scheduling for grid computing

Past Attempts Over past 20 years many projects –AMP, MonALISA, IEPM-BW, PingER, Surveyor … Tended to bundle data generation, sharing and analysis and visualization Provided many new insights BUT: –Lacked widespread, uniform deployments –Analysis & visualization hampered by lack of standards to share data –Failed to achieve critical mass (packaging, open source, unitary solution imposed, and/or lack of community involvement)

Proposal to Address Widespread demand for net info by: –Researchers to know how network is performing –Advanced net apps such as Grids –Net Ops staffs to diagnose problems Flexibility in extracting net performance data, needed since –Network changes quickly, diagnostic data is moving target –New tools, metrics and types of analysis are constantly developed –Lack of effective ways to share performance data across domains

perfSONAR Partnership of Internet2, GEANT, ESnet –Plus in the US: SLAC, U Delaware, GATech –13 EU related NREN deployments of perfSONAR Provide open set of protocols + reference implementation for cross-domain sharing of network measurements –Common performance middleware –Open Grid Forum NMWG = extensible XML data representation –All development is open source to encourage widespread development, deployment, ownership & involvement Early framework prototypes deployed in Europe, N and S America (Brazil), also adopted by LHC

Components Measurement points (MPs) Measurement Archives (MA) Lookup service: register & discover services Authentication Transformation of existing archives (e.g. IEPM) Resource protector: manages policy details Topology service: offers topo info on networks

Methodology Benefits Provide standard interchange format, allow users to focus on problem solving Easier to extend with new sub-components since standard documented APIs, allows evolution –E.g. MP tool developer can focus on tool operation and not worry about deployment Divide & conquer trouble shooting using Lookup & Topology services Easy to generate trouble reports with access to data in standard format Can scale to global size

Compare with Existing Measurement tool clients must be downloaded when needed, need experts, usually need server at remote end (implicit trust & security challenges) Network measurement projects (AMP, PingER, IEPM etc.) –Require installation, host to run on –Lack community involvement, little ownership (e.g. research team know more about site connectivity that site people, but not involved in trouble-shooting –Projects fade as funding ceases Net & system measurement projects (MonALISA) –Closed development effort, license requires sign of intellectual rights to Caltech, must rely on Caltech to incorporate new measurement tools –Lack of community involvement, and consensus may limit widespread ownership and deployment.

Where are we Now? perfSONAR consortium exists, includes many NRENs, active contributions from large segment of research community Set of protocol standards for interoperability A partially complete reference implementation Shortcomings: –Development of some important infrastructure still to be completed (e.g. authentication/authorization) –All existing services need work to turn into production quality, in particular to make easy to deploy Simple installation can take many hours, and is a big barrier to adoption

Next Steps Develop scalable, distributed, redundant Federated Lookup service (like DNS) Integrate common, existing authentication management into perfSONAR Design and build the Resource Protector to implement policy Provide specific, useful example diagnostic services as high quality examples (e.g. for traceroute, ping, one-way delay, SNMP, Layer-2 link services etc.) Provide a Topology service to provide layer-2 & 3 interconnection information Promote perfSONAR to research community –Students get reliable data from perfSONAR, request on demand measurements, provide new analyses

Impact Science Science relies on reliable networking. –Debugging problems across domains extraordinarily difficult today, Increased switched networks will make harder. –PerfSONAR enables divide and conquer between end & intermediate points: provides access to relevant data, enables on demand measurements reduces need to coordinate multi-domain admins (scientist > local net admin > Regional net admin Backbone admin > …), telephone tag, explaining Reduces participants, hours, days, frustration etc

Impact Net Research Network researchers can build, deploy tools to capture analyze net behavior more easily: –No need for login to test boxes, approval from sysadmin to run servers Handled by authentication, Resource Protector service –Common data exchange formats enables access to archives

Impact Education PerfSONAR eases bringing net into classroom, can interrogate, run measurements etc. Incorporate perfSONAR infrastructure components into learning process. Analyze archived data (do not have to rely on goodwill of end users) Early prototypes of perfSONAR components featured in UDel CS courses –Excellent pedagogical vehicle for distributed systems –Develop perfSONAR plugins

Benefits Better understanding of customer experience and needs: –utilization, use patterns, event detection, problem diagnosis, planning Development of better measurement tools, analysis, visualization Pakistan part of major international community of NRENs –In Europe, U.S. and S. America Pakistan research & education access to data to analyze

Benefits SLAC Extend tools to a new country/NREN Extend diagnosis (important for LHC/Pak collab) Increased resources for tool development, analysis

Benefits Education Proven track record –6 students, all will return to Pakistan –3 at SLAC now –1 In Silicon valley start-up, 1 in Oxford, 1 returned to NIIT to pursue PhD Students get exposure to National Lab and world leading researchers Courses at Stanford Hands on exposure to production high speed networks such as are planned for Pakistan

More information/Questions Acknowledgements: –Harvey Newman and ICFA/SCIC for a raison d’etre, ICTP for contacts and education on Africa, Mike Jensen for Africa information, NIIT/Pakistan, Maxim Grigoriev (FNAL), Warren Matthews (GATech) for ongoing code development for PingER, USAID MoST/Pakistan for development funding, SLAC for support for ongoing management/operations support of PingER PingER –www-iepm.slac.stanford.edu/pinger, sdu.ictp.it/pinger/africa.htmlwww-iepm.slac.stanford.edu/pinger sdu.ictp.it/pinger/africa.html Human Development – Role of Internet Exchanges –event-africa- networking.web.cern.ch/event%2Dafrica%2Dnetworking/workshop/slides/The% 20Role%20of%20Internet%20Exchanges.pptevent-africa- networking.web.cern.ch/event%2Dafrica%2Dnetworking/workshop/slides/The% 20Role%20of%20Internet%20Exchanges.ppt Case Studies: – Sahara+Case+Studyhttps://confluence.slac.stanford.edu/display/IEPM/Sub- Sahara+Case+Study –