1. BISmark: the Broadband Internet Service benchmark
Walter de Donato
Department of Computer Engineering and Systems
University of Napoli Federico II

2. The Internet scenario (1/2)
2 billion Internet users worldwide
500 million broadband users
Linux Day, Napoli, 27 October 2012

3. The Internet scenario (2/2)
Heterogeneous access technologies
Wired
Wireless
UMTS
xDSL
Fiber
WiMax
LTE
PSTN
Satellite
Cable
Wi-Fi
ISDN
Internet applications are more and more complex and resource demanding
Linux Day, Napoli, 27 October 2012

4. What performance do users see?
Important for regulators, consumers, ISPs
Are users really getting what they pay for?
Can visibility into problems help reduce service calls?
Notion of performance is fuzzy
What metrics should we measure?
How to measure them?
Linux Day, Napoli, 27 October 2012

5. IP Performance evaluation
Measurement methodologies and techniques
Active → purposely forge synthetic traffic
Passive → exploit user generated traffic
Most common Quality of Service metrics
Throughput
VoIP Quality
Acceptable
Good
Latency
< 300 ms
< 150 ms
Jitter
< 50 ms
< 20 ms
Packet loss
< 3%
< 1%
Latency
Jitter
Packet loss
Linux Day, Napoli, 27 October 2012

6. Studying Internet access networks
Taxonomy of approaches
Server-based
Router-based
Host-based
ISP-based
Core-based
Client-based
Web-based
Service-based
plugin-based
embedded
Last Mile
Linux Day, Napoli, 27 October 2012

7. The BISmark project University of Napoli Federico II
Walter de Donato, Antonio Pescapè
University of Napoli Federico II
Srikanth Sundaresan, Nick Feamster
Georgia Institute of Technology

8. BISmark: a router-based platform
continous periodical measures
observes all traffic passing through network
can take into account interfereces
difficult to obtain large scale deployments
BISmark
firmware
Last Mile
Linux Day, Napoli, 27 October 2012

9. BISmark: architecture overview
Management
Server
Measurement
Server
BISmark
daemon
BISmark
measurement
services
Hi-speed
Network DB
Internet
ISP
BISmark
firmware
Control
Measure
Linux Day, Napoli, 27 October 2012

10. BISmark: main features
Completely open-source (sources maintained on github.com)
Built on top of OpenWRT Linux
Real-time monitoring of routers (60 sec heartbeat)
Remote router firmware update
Server-side configuration (pull mode)
On-demand remote troubleshooting
Geographic selection of measurement servers
Linux Day, Napoli, 27 October 2012

11. BISmark: the platform growing up
Atlanta
USA
Worldwide
2010
2010
2011
2012
2013 6 20 60
260
Routers deployed
Routers deployed 1 2 4 6 12
Developers involved
Netgear
WNDR3700v2
Linksys WRT54GL
NoxBox
Router models
Linux Day, Napoli, 27 October 2012

12. BISmark: challenges and solutions
Limited memory and CPU resources
Hard to let OS, tools and data coexits
Solution: optimizations
optimized and stripped binaries
most binaries on ram-disk (pull mode)
reuse of ash interpeter for most tasks
processing on-the-fly of results
Remote management through NATs
Unable to directly contact routers from the Internet
Solution: ad-hoc management protocol
UDP heartbeat packets allow the server to reply with simple commands
Optimized OpenWRT firmware
Netgear WNDR3700
Linksys WRT54GL
680Mhz MIPS CPU
64 MB RAM
8MB Flash
200Mhz MIPS CPU
16 MB RAM
4MB Flash
Linux Day, Napoli, 27 October 2012

13. BISmark: active measurements
Parameters
Type
L4 Proto
Frequency
Tool
Latency
End-to-end
Last-mile
Upstream load
Downstream load
DNS
ICMP, UDP
ICMP
UDP
5 min
30 min
ping, D-ITG
ping
nslookup
Packet loss
15 min
D-ITG
Jitter
Downstream throughput
Single-thread
Passive
Capacity
TCP
N/A
12 hrs, 60 min
Curl (GET), D-ITG
/proc/net/dev
shaperprobe, D-ITG
Upstream throughput
Curl (PUT), D-ITG
Collected statistics
minimum, maximum, average, standard deviation, median, inter quartile range
Measurement servers
hosted by GeorgiaTech, University of Napoli and MLab
Linux Day, Napoli, 27 October 2012

14. BISmark: scientific results
Srikanth Sundaresan, Walter de Donato, Nick Feamster, Renata Teixeira, Sam Crawford, Antonio Pescapè, Measuring Broadband Home Performance, Communications of ACM, Volume Issue 11, November 2012.
Srikanth Sundaresan, Walter de Donato, Nick Feamster, Renata Teixeira, Sam Crawford, Antonio Pescapè, Broadband Internet Performance: A View From the Gateway, ACM SIGCOMM 2011, Toronto, ON, Canada, August 15-19, 2011
Selected by IRTF for the Applied Networking Research Prize (ANRP) 2012
Srikanth Sundaresan, Nick Feamster, Renata Teixeira, Anthony Tang, W. Keith Edwards, Rebecca E. Grinter, Marshini Chetty, Walter de Donato, Helping Users Shop For ISPs with Internet Nutrition Labels, ACM SIGCOMM Workshop on Home Networks proceedings, Toronto, ON, Canada, August 15-19, 2011
Linux Day, Napoli, 27 October 2012

15. BISmark: live examples
Official website
Virtual router
Linux Day, Napoli, 27 October 2012

16. Thank you! ?

17. Remote troubleshooting
Linux Day, Napoli, 27 October 2012