© 2009 – RNP Large-scale testbeds for network R&D PlanetLab Everywhere Rio de Janeiro, April 2009 Michael Stanton Rede Nacional de Ensino e Pesquisa - RNP

Summary We discuss R&D testbeds for networking and distributed systems, seen from the point of view of RNP In addition to activities carried out in Brazil, we also examine some testbeds in use or planned in other countries, especially in relation to future Internet development In conclusion suggestions are made about future steps to be followed in Brazil. Large-scale testbeds for R&D in Brazil2

Introduction to Brazil In 1494 Spain and Portugal had divided between themselves undiscovered lands by the Treaty of Tordesillas –The Tordesillas Line was to be the frontier between the dominions of Spain (W) and Portugal (E) Brazil is the successor country to the Portuguese dominions in South America –Rather over one half of present Brazil lies to the WEST of the Tordesillas Line Brazil is a BIG place! –diameter of about 4,200 km –42 x 42 = 1764 ms 2 Current population of about 180 millions, unevenly distributed –most of the population and communications infrastructure concentrated to the EAST of the Tordesillas Line Tordesillas Line to Spain to Portugal 3

Large-scale testbeds for R&D in Brazil RNP – Rede Nacional de Ensino e Pesquisa RNP is the Brazilian NREN –maintained by the Brazilian government (since 1989) to enable network access to the national research and education community –provides national (inter-state) and international R&E connectivity for more than 300 public and private universities and research centers through the provision of advanced networking infrastructure also provides commodity access – one-stop shopping –promotes the development of advanced networking and applications Since 2000, RNP is managed for the federal government by a non-profit private company, RNP-OS, legally recognised as an “Organização Social”, which allows the government to contract its services without competitive tender. 4

Large-scale testbeds for R&D in Brazil RNP’s service networks RNP includes the following funded connectivity: National backbone network – Rede IPÊ –1 PoP (Point of Presence) in each state – usually a federal university –Link capacity depends on the available telco infrastructure –Currently from 2 Mbps to 10 Gbps Direct intercity connections between state PoP and non-local federal instituions (education, science and technology) –Currently from 2 to 155 Mbps (depends on the institution) Community-based optical metro networks connected to PoPs –Currently being built out – 9 out of 27 already in operation No service charges are made to end user institutions Non-federal institutions are normally required to fund their own access links 5

National integration

RedClara, beyond 2008 Extend RedClara to all LA&C countries Promote applications in education and health Start of a new project (ALICE2), with the support of the EC, in 2009 Source: 7Large-scale testbeds for R&D in Brazil

A world-class network – RNP in GLIF Source: 8Large-scale testbeds for R&D in Brazil

A world-class network – worldwide GLIF Source: 9Large-scale testbeds for R&D in Brazil

Rede IPÊ – national backbone network Last big reform in 2005 (5 th phase) Capacity reflects available telco infrastructure Currently composed of: Multigigabit core network –4 PoPs at 10 Gbps, and 6 PoPs at 2.5 Gbps –IP over lambdas ( km) Terrestrial SDH connections to 15 PoPs –Most links are 34 Mbps –Some at 2 Mbps –Some upgrades in 2007 to 102, 155 and 622 Mbps 2 PoPs connected by satellite at 2 Mbps 10

Large-scale testbeds for R&D in Brazil Evolution of academic networks in Brazil RNP Phase YearTechnologyLink capacitiesComment 1988BITNET up to 9.6 kbps first national network 11992Internet 9.6 and 64 kbps first national IP network (RNP) up to 2 Mbps also: commercial IP deployed 31999IP/ATM, IP/FR VC up to 45 Mbps, access up to 155 Mbps RNP2 national backbone; testbed metro networks in 14 cities (using ATM/dark fiber) 42003IP/SDH 34, 155, 622 Mbps also: IP/WDM interstate testbed network (Project GIGA) 52005IP/WDM 2.5 and 10 Gbps IPÊ national backbone; metro networks in 27 capitals 11

Large-scale testbeds for R&D in Brazil Evolution of academic networks in Brazil Phase 0 BITNET Phase 1 Internet Phase 2 comercial Internet Phase 3 RNP2 Phase 4 RNP2+ Phase 5 Ipê (Link capacity) 12

RNP and its innovating networks The first version of RNP’s network was deployed in 1992, and pioneered in using TCP/IP technology nationwide Since then, the different generations of network deployed and operated by RNP have innovated technically, at least within Brazil With the development of the network industry here, since the beginning of commercial IP networks starting in 1995, RNP is no longer the only IP network operator in Brazil, but continues to lead technologically, in the pursuit of new models of infrastructure and applications Large-scale testbeds for R&D in Brazil13

Technology changes in the network The beginning of each new technology phase was a step in the dark –new circuits (or service, in the case of ATM/FR) were ordered and delivered, and equipment configured –after configuration by the network engineers, the new network would begin to work, and would be put into operation as soon as possible. –users could then begin to use the resources of the new network Problems: –lack of familiarity with the new technologies before operational deployment –the technology transition became a singularity Large-scale testbeds for R&D in Brazil14

New application services Internet technology is very accommodating of new application services –any user can develop a new service, impelemnted as a distributed application using the sockets API –this permits and encourages experimentation with applications that can be built initially in the laboratory (in a LAN environment) and then migrated to the wide area network Some problems which arise: –some distributed services require componentes “within the network” – security problems –performance monitoring may be a problem Large-scale testbeds for R&D in Brazil15

Alternative solutions Both for –preparing an upcoming network technology change –developing a a new large-scale distributed application we really need a testbed facility, isolated from the production network, and which reproduces its characteristics of scale and performance This isolation can be real or virtual –Real – the testbed is based on separate physical infrastructure, independent of the production network example: Project GIGA –Virtual – the testbed shares the same infrastructure used by the production network example: PlanetLab Large-scale testbeds for R&D in Brazil16

Large-scale testbeds for R&D in Brazil17 Project GIGA – optical networking testbed Partnership between –RNP, CPqD (telco industry R&D centre in Campinas, SP), R&D community in networks and distributed systems –Financed by FUNTTEL between 2002 and 2007 – US$20M –telcos – provide optical fiber at no cost Objectives: –build an advanced networking testbed for development and demonstration purposes –support R&D subprojects in optical and IP networking technology and advanced applications and services Network support (since May 2004) –R&D subproject consortia provided with internal connectivity using VLANs – 20 institutions connected –however: the testbed (with some exceptions) did not provide exteral connectivity, limiting its usefulness FUNTTEL

Large-scale testbeds for R&D in Brazil18 GIGA testbed network - location dark fibre-based 700-km inter-city backbone in states of São Paulo and Rio de Janeiro (south-east Brazil) Initially 20 universities and R&D centers in 7 cities 2.5G DWDM in the inter-city backbone 2.5G CWDM used in the metropolitan area testbed network

Large-scale testbeds for R&D in Brazil19 GIGA testbed network - location Universities IME PUC-Rio PUC-Campinas UERJ UFF UFRJ Mackenzie UNICAMP USP R&D Centers CBPF CPqD CPTEC INCOR CTA FIOCRUZ IMPA INPE LNCC LNLS

Large-scale testbeds for R&D in Brazil20 Testbed network design 2.5G DWDM inter-city network between Campinas and Rio de Janeiro (some upgrades to 10G) – up to 6 waves per link (can use 8) 2.5G CWDM metro networks in São Paulo, Campinas and Rio de Janeiro all links currently 1 Gigabit Ethernet –optical equipment from the Brazilian firm, Padtec ( –IP equipment from Extreme Networks S.J. dos Campos São Paulo Campinas Rio de Janeiro Campinas São PauloS. José dos Campos Rio de Janeiro Cachoeira Paulista MAN CP MAN SP MAN RJ Petrópolis Niterói 2λ2λ 2λ2λ 2λ2λ 1λ1λ 3λ3λ 1λ1λ 2λ2λ 1λ1λ 3λ3λ

What is PlanetLab? (base d on slides by Marc Fiuczynski, Sept. 2007) Facility: Planetary-scale “overlay” & “underlay” network –900+ Linux-based servers at 400+ sites in 40+ countries –Currently there exist a handful of PL sites in Brazil operated by RNP and a few universities and research institutions

PlanetLab Facility Today servers at 450+ sites in 40+ countries Co-located throughout the Uni. & Companies Co-located at network crossroads (Internet2, RNP, CERNET, …) 22Large-scale testbeds for R&D in Brazil

Virtualization in PlanetLab? Research Community: Distributed Sys. & Networking –Researchers can get a set of “virtual machines” across these servers (SLICE) –In a SLICE researchers can deploy & evaluate … –… distributed systems services and applications “The next Internet will be created as an overlay in the current one” –… network architectures and protocols “The new Internet will be created in parallel next to the current one” (see later)

Example Network Services Scalable Large-File Transfer: CoBlitz—Princeton, LoCI—Tennessee Content Distribution: Coral—NYU, CoDeeN—Princeton, CobWeb— Cornell Distributed Hash Tables: OpenDHT—UC Berkeley; Chord-MIT Routing Overlays: I3 Internet Indirection Infrastructure—UC Berkeley Multicast Delivery Nets: End System Multicast—CMU, Tmesh-U. Michigan Serverless ePOST—Rice University Publish-Subscribe News Access: CorONA—Cornell Robust DNS Resolution: CoDNS—Princeton, CoDoNs—Cornell Mobile Access: DHARMA—U. of Pennsylvania Location/Anycast Services: OASIS—NYU, Meridian—Cornell Internet Measurement: ScriptRoute—U. of Maryland Above services communicate with >1M real users and transmit ~4TB of data per day 24Large-scale testbeds for R&D in Brazil

PlanetLab Node Software Architecture Slice Manager (SM) Virtualization Software x86 Server Hardware Slice 25Large-scale testbeds for R&D in Brazil

Slices 26Large-scale testbeds for R&D in Brazil

Slices 27Large-scale testbeds for R&D in Brazil

Slices 28Large-scale testbeds for R&D in Brazil

Thoughts about the future Internet The success of the Internet has been so enormous that it is tempting to imagine its future by extrapolating from the present However, there are consequences of its design, based on decisions taken in the 1970s, which severely limit its security, availability, flexibility and manageability These limitations can not be removed through small incremental adjustments of the existing network, and, if they are not removed, they will create huge impediments to the ability to use and exploit the Internet in the future Large-scale testbeds for R&D in Brazil29

Removing these limitations For many years, combatting Internet limitations has been carried out through a series of “patches”, introduced to solve specific problems. Unfortunately, these patches result in increased complexity, resulting in a less robust system, whose operation has become more difficult and costly A growing consensus exists in the network research community that we have already reached the point where patches are inadequate, and a fundamental rethink of the Internet is required (from the GENI Research Plan, 2007) Large-scale testbeds for R&D in Brazil30

GENI - (Global Environment for Network Innovations) An initiative of NSF (USA) to create a shared testbed environment to allow the validation of new network architectures –Initial phase: 2005 to 2007 – design of the GENI environment –Present phase: since 2008, deployment and use GENI will support research which can lead to a future Internet with improved chacteristics –more comprehensive security –greater generality –better integration of optical and wireless technologies –integration of the world of sensors and embedded processors –improved options for the economic health of ISPs Large-scale testbeds for R&D in Brazil31

The GENI testbed environment GENI will: allow experiments with alternative large-scale network architectures, services and applications under realistic conditions use vitualizations to permit carrying out multiple independent and simultaneous experiments permit long-duration experiments, allowing mature prototypes to serve “living” user communities facilitate experimental research through the use of extensive tools for measurement and data collection In summary, GENI will provide support for the taking ideas on large-scale ideas from their conception to their deployment, by means of experimental validation Large-scale testbeds for R&D in Brazil32

How the GENI environment will be built The GENI environment is inspired on PlanetLab, and especially the “Meta-Testbed” VINI - VINI extends the scope of PlanetLab to allow –“slicing” of links between the nodes (link virtualization) –substitution of level 3 protocols (IP) Large-scale testbeds for R&D in Brazil33

Next Step: Meta Testbed (base d on slides by Marc Fiuczynski, Sept. 2007) Goals –support experimental validation of new architectures simultaneously support real users and clean slate designs allow a thousand flowers to bloom –provide plausible deployment path Key ideas –virtualization multiple architectures on a shared infrastructure shared management costs –opt-in on a per-user / per-application basis attract real users demand drives deployment / adoption 34Large-scale testbeds for R&D in Brazil

VINI: Our Meta Testbed approach Infrastructure –PlanetLab provides “access network” with global reach user desktops run proxy that allows them to opt-in treat nearby PlanetLab node as ingress router –NLR/I2 provides high-speed backbone in the United States populate with programmable routers extend slice abstraction to these routers Usage model –each architecture (service) runs in its own slice –two modes of use short-term experiments long-running stable architectures and services 35Large-scale testbeds for R&D in Brazil

Extending Slices to a VINI testbed 36Large-scale testbeds for R&D in Brazil

Extending Slices to a VINI testbed 37Large-scale testbeds for R&D in Brazil

Extending Slices to a VINI testbed 38Large-scale testbeds for R&D in Brazil

User Opt-in Client Server NAT wireless 39Large-scale testbeds for R&D in Brazil

Internet in a Slice (IIAS) XORP (routing protocols) vif1vif2vif0 IPv4 Fwd table User Kernel Filters, shapers PlanetLab VM E-GRE tunnels XORP in Network Container –Adds routes to copy of kernel IPv4 forwarding table –Kernel forwards packets between virtual interfaces Filters and shapers –Add delay and loss, constrain bandwidth Virtual interfaces –Appear as Ethernet devices in a slice –Reduce MTU for tunneling E-GRE tunnels –Hack standard GRE tunnels to preserve MAC headers 40Large-scale testbeds for R&D in Brazil

GENI Extenssion of VINI – key ideas: –virtualization multiple network architectures sharing a common infrastructure –user opt-in: per user / per application intended to attract real users Infrastructure –NLR/Internet2 provide high-capacity backbone in the US populate with programmable resources (processors, storage) populate with programmable routers –more sophisticated than the PCs used in VINI extend the “slice” abstraction to these routers include “extensions” to wireless and sensor networks Large-scale testbeds for R&D in Brazil41

GENI: The Physical Network (slides by C. Qiao, 2008) Large-Scale Facility of Networked Systems –Reasonable Representation of the Internet’s Complexity –A Nationwide Optical Network ~ 200 Universities –Clusters for Processing/Storage –Wireless Access Networks Mobility, Location Awareness –Sensor Networks –Connected to a large number of User Communities Partnerships to Extend GENI within the US –Add Technologies and Users Federation to Extend GENI on a Global Scale 42Large-scale testbeds for R&D in Brazil

Core Nodes Schematic GENI Network Mobile Wireless Network Edge Site Sensor Network Edge Nodes Federated International Facility 43Large-scale testbeds for R&D in Brazil

Programmability All network elements programmable via open interfaces and/or downloadable user code Programmable Sensor Node Open API Radio platform Programmable Edge Node Programmable Core Node GENI Control & Management Plane 44Large-scale testbeds for R&D in Brazil

Slicing and Virtualization Mobile Wireless Network Edge Site Sensor Network - share resources to support many simultaneous experiments 45Large-scale testbeds for R&D in Brazil

GENI Design Principles Physical network ‘substrate’ –building block components –elements / nodes / links / subnets Software control & management framework –knits building blocks together –allows many parallel experiments (slices) –creates arbitrary logical topologies (virtualization) Programmable for ‘Clean Slate’ research Instrumented for accurate analysis Flexible and Phased Design –Support Technology Introduction during GENI Lifetime 46Large-scale testbeds for R&D in Brazil

GENI Phase 1(2008-9) (based on slides by Chip Elliott – GENI program director) Provides the very first, national-scale prototype of an interoperable infrastructure suite for Network Science and Engineering experiments Creates an end-to-end GENI prototype in 6-12 months with broad academic and industrial participation, while encouraging strong competition in the design and implementation of GENI’s control framework and clearinghouse Includes multiple national backbones and regional optical networks, campuses, compute and storage clusters, metropolitan wireless and sensor networks, instrumentation and measurement, and user opt-in Because the GENI control framework software presents very high technical and programmatic risk, the GPO has funded multiple, competing teams to integrate and demonstrate competing versions of the control software in Phase 1 Large-scale testbeds for R&D in Brazil47

GENI phase 1: integration: 5 competing control schemes Large-scale testbeds for R&D in Brazil48

Research using GENI - FIND Prior to the building of the GENI testbed, NSF launched the initiative FIND – Future Internet Design, with the aim of identifying and financing research activities One product of this initiative is the GENI Research Plan, which details the research motivation for GENI, and some of the research goals which will become possible: Large-scale testbeds for R&D in Brazil49

Experimental activities in Europe FIRE - Future Internet Research & Experimentation European initiative directed towards the design of the future Internet, similar to the FIND and GENI initiatives of the NSF promotes the concept of experimental research, combining visionary academic research with validation and experimentation typical of industry aims to create a multidisciplinary research environment to investigate and validate experimentally innovative ideas for new paradigms of networks and services plans to create a “European Experimental Facility” (EEF), formed by the interconnection and federation of both existing and future testbeds, for emerging and future Internet technologies First projects selected in 2008 Large-scale testbeds for R&D in Brazil50

FIRE/Panlab (FOKUS/DE)– Large-scale testbeds for R&D in Brazil51

FIRE/OneLab (UPMC, FR) – History: –Mar/04:based on ENEXT (Identification of critical testbeds for networking research) –Sep/06:OneLab1 project funded as an IST project under the FP6 funding program:10 partners, 2 years –Sep/08: proposal funded as an IST project under the FP7 funding program: 26 partners, 2 years OneLab1 – Goals –extend PlanetLab into new environments, beyond the traditional wired internet –improve PlanetLab’s monitoring capabilities –provide a European administration for PlanetLab nodes in Europe Large-scale testbeds for R&D in Brazil52

FIRE/Federica – Large-scale testbeds for R&D in Brazil53

Comparison of FIRE prototypes Large-scale testbeds for R&D in Brazil54

Future Internet activities in Japan AKARI - –plans to deploy a new generation network by 2015, based on a new architecture –supposes the use of an experimental testbed incorporating virtualization techniques, as in GENI Large-scale testbeds for R&D in Brazil55

Perspectives in Brazil Phase 2 of Project GIGA (RNP- CPqD) –original Project GIGA funding ended in Dec/2007, although the experimental network continues to operate –since 2007 RNP and CPqD have been seeking to maintain an experimental facility for their research communities –RNP’s proposal is to place greater emphasis on research into architectures for a future Internet Large-scale testbeds for R&D in Brazil56

Perspectives in Brazil INCT/Web Science (consortium led by UFRJ) – more than 100 researchers – approved in 2008 (3 to 5 years) A group of 8 researchers from (RNP, UFF, UFPA, UNIFACS, USP) included a research proposal in “Future Internet Architectures” –main emphasis on experimental research, using an environment based on PlanetLab / VINI, with extensions for wireless access networks –RNP network will be used for long-distance communications (within Brazil and externally) Note: –VINI requires access to a “lower than level 3” network – this depends on the next phase of the RNP network (FuturaRNP), expected in 2010 Large-scale testbeds for R&D in Brazil57

FuturaRNP and 2010 RNP is engaged in several initiatives, which are changing the face of its infrastructure: –adoption of a layer 2 national backbone, and introduction of static and dynamic end-to-end circuits as well as routed IP (Hybrid Network) –extension to all 27 capitals and 10 other cities of optical metro networks based on Ethernet technology (layer 2) –huge increase in capacity, when possible through multiple lambdas in DWDM systems A possible desirable consequence would be the permanent reservation of capacity for experimental activities, segregated from production traffic (à la GENI) Large-scale testbeds for R&D in Brazil58

Federation with other initiatives The deployment of an experimental facility in Brazil to support research into new architectures and applications will simplify international collaboration with similar initiatives abroad This would be brought about by the interconnection (federation) of the national facility with similar facilities in other countries It should be noted that this style of federation is one of the characteristics of the projects we have described from the USA and Europe. Large-scale testbeds for R&D in Brazil59

The Future Internet will be polymorphic Large-scale testbeds for R&D in Brazil60

The Future Internet will be polymorphic Large-scale testbeds for R&D in Brazil61

Michael Stanton Thank you! Yellow ipê in blossom