Download presentation
Presentation is loading. Please wait.
1
Brian L. Tierney, Dan Gunter BLTierney@lbl.gov, DKGunter@.lbl.gov
NetLogger Summarizer and the nlioperf tool Brian L. Tierney, Dan Gunter Lawrence Berkeley National Laboratory Center for Enabling Distributed Petascale Science
2
NetLogger Toolkit We have developed the NetLogger Toolkit (short for Networked Application Logger), which includes: tools to make it easy for distributed applications to log interesting events at every critical point NetLogger client library (C, Java, Perl, Python) tools for host and network monitoring event visualization tools that allow one to correlate application events with host/network events NetLogger event archive and retrieval tools (new) NetLogger combines network, host, and application-level monitoring to provide a complete view of the entire system. Open Source, available at
3
NetLogger Analysis: Key Concepts
NetLogger visualization methodology is based on time correlated and object correlated events. In order to associate a group of events into a “lifeline”, you must assign an “Event ID” to each NetLogger event Sample Event ID: file name, block ID, frame ID, etc.
4
Time-based summarization
Imagine you want detailed instrumentation of a simple loop that reads data, processes it, then writes out some result. DO for i=1, N log(“read.start”, ...) read_data() log(“read.end”, ...) process_data() log(“write.start”, ...) write_data() log(“write.end”, ...) DONE
5
Time-based summarization
Reporting only a periodic summary (mean, sd) of times and values between “start” and “end” events (of a given type and ID). .. to here Go from here.. Orders of magnitude data reduction with, for many purposes, no loss in explanatory power. Since the original instrumentation is still there, you can “turn on” and off the full detail in a running program.
6
Sample Use: GridFTP Bottleneck Detector
GridFTP today By default only logs single throughput number GridFTP throughput might be limited by Sending/receiving disk NFS/AFS mounted partition? network currently no way to tell Experimental GridFTP enhancement Use NetLogger summarization library to monitor all I/O streams
7
nlioperf Started out as a test tool for the NetLogger summarization library Actually quite useful test tool Supports multiple TCP streams file input/output TCP buffer tuning bottleneck detection multiple levels of logging full, interval summary, final summary only
8
nlioperf: network vs disk performance
9
Bottleneck Detection: WAN Transfer
10
Ideas Add NetLogger summarizer to iperf? Add disk testing to iperf
11
More Information NetLogger 4.0beta just released
includes new summarization Library: or
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.