Building the Measurement Infrastructure and Virtual Data Observatory in the European Future Internet Experimental Facility Péter Hága Eötvös Loránd University

why to build a new experimental facility? in the Internet it is often not possible to measure traffic flows and other aspects of usage models and analysis of measurement data should play crucial role in the research on a Future Internet the design of new network architectures should be amenable to modeling and measurement the network itself (in contrast of the todays Internet) the Experimental Facility should provide a platform with a rich set of tools for measurement and monitoring the EF should offer full observability of the experiment and the related data theoretical analysis and modeling can benefit from the rich capture and logging of data FuturICT 2009, Budapest, Hungary, 29 June 2009.

FI research initiatives GENI/FIND/NetSE, US AKARI Project, Japan Future Internet Forum, Korea ANR, France G-Lab Initiative, Germany SHOK, Fin Internet del Futuro, Spain EU Future Internet Research and Experimentation (FIRE) FuturICT 2009, Budapest, Hungary, 29 June GENI infrastructure

Best Testbed Award The E uropean T raffic O bservatory M easurement I nfrastru C ture (etomic) was created in 2004 within the Evergrow Integrated Project. Since 2005 also supported by the Hungarian Office for Research and Technology Its goals: – to provide open access, public testbed for researchers experimenting the Internet –to serve as a Virtual Observatory active measurement data on the European part of the Internet Since 2008 in Onelab2 Etomic is federated in the Advanced Network Monitoring Equipment etomic history FuturICT 2009, Budapest, Hungary, 29 June 2009.

The GENI connection FuturICT 2009, Budapest, Hungary, 29 June 2009.

Infrastructure stations Best Testbed Award

Future Internet Research and Experimentation (FIRE) programme OneLab2: An Open Federated Laboratory Supporting Network OneLab2 EU Integrated Project participants Europe wide + 1 from Japan budget 7.5 M aims to federate several measurement infrastructures (including Planetlab Europe) includes building (extending) and maintaining a monitoring and measurement infrastructure (based on the Etomic) partners involved: –Université Pierre et Marie Curie, France –Universidad Autonoma de Madrid, Spain –Tel Aviv University, Israel –Eötvös Loránd University, Hungary FuturICT 2009, Budapest, Hungary, 29 June 2009.

Advanced Network Monitoring Equipment Web interface available via Account application -> own measurement design Free access to periodic measurement end-to-end data Measurement time slot reservation for registered users, unique slot (experiment are not affected by other users) Programming DAG and ARGOS cards via a user friendly API Controlling APE box Fully configurable active measurement scenarios Measurements are distributed automatically to the measurement stations Measurement data is stored in a VO fashion FuturICT 2009, Budapest, Hungary, 29 June 2009.

Hardware setup Advanced Network Monitoring Equipment (ANME) FuturICT 2009, Budapest, Hungary, 29 June 2009.

Main components

ANME hardware components Etomic –Precise active measurements with DAG 3.6GE and ARGOS FPGA CoMo –Monitoring the traffic of Planetlab nodes APE –lightweight measurement box –standalone GPS receiver to provide time synchronization FuturICT 2009, Budapest, Hungary, 29 June 2009.

Etomic/CoMo architecture Server PC architecture Linux OS Endace DAG 3.6 GE card or ARGOS FPGA measurement card with packet sending capability (packet offset ~60ns) GPS antenna for time synchronization FuturICT 2009, Budapest, Hungary, 29 June 2009.

APE lightweight measurement box standalone based on Blackfin programmable board dedicated IP packet timestamping module (< μsec) low cost (300 )

Infrastructure management etomic Central Management System FuturICT 2009, Budapest, Hungary, 29 June 2009.

Central Management System FuturICT 2009, Budapest, Hungary, 29 June Etomic CMS IBM blade server User management Node maintenance Experiment scheduling Storing experiment results (temporally) Web GUI

Slices vs. unique timeslots no virtualization balancing sliceability and fidelity is one of the most fundamental challenges facing the EF virtualization allows many researchers to share a common set of resources while virtualization introduces too much unpredictability in timing measurements dedicated measurement hardware elements can be allocated to some slices. Measurement hardware should operate under temporal partitioning principle.

realization FuturICT 2009, Budapest, Hungary, 29 June 2009.

Experimental use cases Some ongoing basic topics FuturICT 2009, Budapest, Hungary, 29 June 2009.

Use cases FuturICT 2009, Budapest, Hungary, 29 June one way delay (60nanosec resolution) tracking topology changes available bandwidth meter transport protocol testing queuing delay tomography geolocation experiments …

Data handling Store & share FuturICT 2009, Budapest, Hungary, 29 June 2009.

Experimental facilities most network measurement projects: use a single dedicated infrastructure scan only narrow subsegments analyze a limited set of network characteristics centralized and separated from each other key idea: try to interconnect separate measurement data! large-scale behaviour long-term evolution

Traditional approach Traditionally measurements are designed to collect only specific data, important from the point of view of the researchers agenda. FuturICT 2009, Budapest, Hungary, 29 June 2009.

Sharing science Genome databases Astronomy

Related work: CAIDA/DatCat

Related work: MoMe database

Related work: MAWI repository

Data publication efforts 1.DatCat (USA): searchable catalog of metadata about measurements passive traffic traces, traceroutes, BGP tables, virus propagation studies 2.MOME (EU): database for meta-measurement data packet & flow traces, routing data, HTTP traces standardization efforts sharing of analysis tools is possible (e.g. jitter calculation) 3.MAWI (Japan): repository of passive traces from the WIDE backbone (collected since 1999) raw data is not stored raw data from single infrastructure

you can: GREP 1 MB in ~ seconds GREP 1 GB in ~ minutes GREP 1 TB in ~ days GREP 1 PB in ~ years for 1 PB you need ~ 1000 disks at some point you need –to limit search –parallel data search and analysis – move the analysis to the data storage this is where databases can help FTP and GREP are not adequate FTP ~ 1 MB/sec FTP ~ 1 GB/min (= 1 $/GB) FTP ~ 1 TB/ days and 1K$ FTP ~ 1 PB/ 3 years and 1M$

Key ideas in data handling store & share raw data –joint analysis of different types of measurement data –reanalysis (with new evaluation methods) –reference data (historical comparison) share analysis tools –server side processing simplifies client applications –no need to transfer bulk data packages: online processing standardization, network XML Network Measurement Virtual Observatory ( nmVO ) FuturICT 2009, Budapest, Hungary, 29 June 2009.

VO approach The modern approach is to collect and store all measurable data and make it available for virtual observation. Virtual measurements can have set of goals different from the original FuturICT 2009, Budapest, Hungary, 29 June 2009.

Unified interface VO can be realized by collecting measurement data from different infrastructures. Data structures should be standardized netXML FuturICT 2009, Budapest, Hungary, 29 June 2009.

Casjobs - database interface the nmVO concept: efficient way to organize and share different types of measurement data and corresponding analysis tools prototype node implementation client application example a semantic extension to integrate multiple datasources

Summary Experimental facility: –old etomic is running –new etomic will be launched this summer Main features: –Precise active measurements –Unique time slot reservation –Easy to use GUI –Data is collected in VO fashion European deployment (planned): FuturICT 2009, Budapest, Hungary, 29 June 2009.

Thanks and register! Visit: org