The Status and Plans for SDN IHEP Fazhi Qi CC,IHEP Fazhi Qi/IHEPCC 1

Agenda About IHEP & IHEPCC About IHEP & IHEPCC Why Why How How Progress & Plan Progress & Plan Summary Summary Fazhi Qi/IHEPCC

Agenda About IHEP & IHEPCC About IHEP & IHEPCC Why Why How How Progress & Plan Progress & Plan Summary Summary Fazhi Qi/IHEPCC

IHEP at a Glance ~1500 staffs, with ~1200 scientists and engineers ~1500 staffs, with ~1200 scientists and engineers Four(Six) sites currently Four(Six) sites currently Beijing, Dongguang(CSNS), Shenzhen (dayabay), Tibet (Yangbajing), Jiangmen (JUNO), Chengdu (LHASSO)Beijing, Dongguang(CSNS), Shenzhen (dayabay), Tibet (Yangbajing), Jiangmen (JUNO), Chengdu (LHASSO) The largest fundamental research center in China with following research fields: The largest fundamental research center in China with following research fields: Experimental Particle PhysicsExperimental Particle Physics Theoretical Particle PhysicsTheoretical Particle Physics Astrophysics and cosmic-raysAstrophysics and cosmic-rays Accelerator Technology and applicationsAccelerator Technology and applications Synchrotron radiation and applicationsSynchrotron radiation and applications Nuclear analysis techniqueNuclear analysis technique Computing and Network applicationsComputing and Network applications …

Fazhi Qi/IHEPCC Major Projects BEPCII/BESIII BEPCII/BESIII 36 Institutes from China, US, Germany, Russian, Japan,…36 Institutes from China, US, Germany, Russian, Japan,… 5PB data in 5 years 5PB data in 5 years Daya Bay Neutrino experiment Daya Bay Neutrino experiment 39 Institutes from China, US, …39 Institutes from China, US, … 400TB/year data collected400TB/year data collected Yangbajing in Tibet Yangbajing in Tibet Cosmic-ray observatory,Cosmic-ray observatory, Collaborations of China, Italy, JapanCollaborations of China, Italy, Japan ~200TB raw data per year.~200TB raw data per year. JUNO JUNO Jiangmen Underground Neutrino ObservatoryJiangmen Underground Neutrino Observatory ~ 1PB Raw date per year~ 1PB Raw date per year LHAASO LHAASO the Large High Altitude Air Shower Observatorythe Large High Altitude Air Shower Observatory ~ 2PB Raw data pear year~ 2PB Raw data pear year CSNS CSNS Chinese Spallation Neutrons SourceChinese Spallation Neutrons Source LHC LHC Members of ATLAS and CMSMembers of ATLAS and CMS WLCG Tire-2 at IHEPWLCG Tire-2 at IHEP AMS (Alpha Magnetic Spectrometer) AMS (Alpha Magnetic Spectrometer)

Other Projects hosted by Chinese universities and institutes CDEX, China Dark-matter EXperiment CDEX, China Dark-matter EXperiment Direct Detection of cold dark matter with ultra-low energy thresholdDirect Detection of cold dark matter with ultra-low energy threshold SRF, Shanghai SRF, Shanghai SRF, USTC, Hefei SRF, USTC, Hefei … Fazhi Qi/IHEPCC

* Computing Center, IHEP Computing Center, IHEP Staffs, 20 project staffs ， 15 Students Staffs, 20 project staffs ， 15 Students Serve for the HEP Experiments Serve for the HEP Experiments Infrastructure Operation Operation Network and Security Network and Security Computing & Storage Computing & Storage Basic IT services Basic IT services Database Database Applications Development Applications Development …… …… IHEP CC Fazhi Qi/IHEPCC 7

Agenda About IHEP & IHEPCC About IHEP & IHEPCC Why Why How How Progress & Plan Progress & Plan Summary Summary Fazhi Qi/IHEPCC

HEP Computing Challenges Rapid access to data stores & efficient data exchange Rapid access to data stores & efficient data exchange Secure, efficient, managed access to worldwide computing resources Secure, efficient, managed access to worldwide computing resources Tracking state and usage patterns Tracking state and usage patterns Matching resources to demands Matching resources to demands Enable physicists in all world regions to collaborate on analysis Enable physicists in all world regions to collaborate on analysis Building high speed intercontinental networks to support the above requirements Building high speed intercontinental networks to support the above requirements Fazhi Qi/IHEPCC

CSTNet Chinese Science and Technology Network Chinese Science and Technology Network One core center and twelve regional centers, covering over 30 provinces, 370 Institutes in China One core center and twelve regional centers, covering over 30 provinces, 370 Institutes in China Fazhi Qi/IHEPCC

CERNet Chinese Education and Research Network Chinese Education and Research Network Connects more than 200 cities and provides connectivity to about 2000 colleges and institutes in China Connects more than 200 cities and provides connectivity to about 2000 colleges and institutes in China Fazhi Qi/IHEPCC

IPv6 Network in China CNGI : Chinese Next Generation Internet CNGI : Chinese Next Generation Internet Backbone: 2.5~100Gbps Backbone: 2.5~100Gbps More than 25 core exchange nodes and over 600 universities and other organizations connected More than 25 core exchange nodes and over 600 universities and other organizations connected Fazhi Qi/IHEPCC

Current Network Status for IHEP IPv4 IPv4 Good connection for international cooperationGood connection for international cooperation A little poor for the domestic communicationA little poor for the domestic communication  Universities… IPv6 IPv6 Available bandwidth is goodAvailable bandwidth is good But unstable,But unstable,  related with different time Fazhi Qi/IHEPCC

Chinese HEP Communities Status IHEP is the leadership members in Chinese HEP Communities IHEP is the leadership members in Chinese HEP Communities Most of the HEP Experiments in China are hosted by IHEP Most of the HEP Experiments in China are hosted by IHEP Lack of sufficient IPv4 network bandwidth……but IPv6 resources is pretty good! Lack of sufficient IPv4 network bandwidth……but IPv6 resources is pretty good! Lack of network engineers in other HEP experiment members in China(manpower) Lack of network engineers in other HEP experiment members in China(manpower) Some times, For IHEP network team, our users is not only the IHEP users, also the other Chinese HEP community members are included …… Some times, For IHEP network team, our users is not only the IHEP users, also the other Chinese HEP community members are included …… Fazhi Qi/IHEPCC

How to have a good network for HEP Good reliability and availability Good reliability and availability Good performance Good performance Monitor and measure the status and performance of network Monitor and measure the status and performance of network Find the best network resources Find the best network resources Optimize network link configuration Optimize network link configuration … Fazhi Qi/IHEPCC

What shall we do for this? Provide a simple, flexible, robust, high performance and central － controlled network environment for HEP members in China Provide a simple, flexible, robust, high performance and central － controlled network environment for HEP members in China SDN is one of the Choices… SDN is one of the Choices… Fazhi Qi/IHEPCC

Agenda About IHEP & IHEPCC About IHEP & IHEPCC Why Why How How Progress & Plan Progress & Plan Summary Summary Fazhi Qi/IHEPCC

The key features for SDN Three level architecture: Applications-Controller-SwitchThree level architecture: Applications-Controller-Switch Central control ： simple for end usersCentral control ： simple for end users Opening for the network capability/functionOpening for the network capability/function  build the network environment on the applications needs ;  A programmatic interface between application and the network, Tight integration; direct feedback loop -> reactive system Overlay and VirtualizationOverlay and Virtualization Fazhi Qi/IHEPCC

It is much easier for the SDN within a data center It is much easier for the SDN within a data center We can control all the network infrastructure in the data centerWe can control all the network infrastructure in the data center  Links, Devices, Applications…… Data Data Center But the big problem we are facing is the Internet performance among the HEP members in China But the big problem we are facing is the Internet performance among the HEP members in China a private virtual network across Chinese HEP members based on SDN a private virtual network across Chinese HEP members based on SDN architecture  Intelligent data transmission network path selection algorithm  Use IPv6 network links around China (CNGI) IHEP Fazhi Qi/IHEPCC

DataCenter Science DMZ Science DMZ R&D for Cloud Computing R&D for Cloud Computing Fazhi Qi/IHEPCC

Science DMZ Fazhi Qi/IHEPCC

Science DMZ Fazhi Qi/IHEPCC Policies: S-IP S-Port D-IP D-Port Policies For Science Data-Exchange Current Based on ACL SDN Switches are under consideration for this

for Cloud Computing Provided by Shan Zeng Cloud computing becomes more and more popular Cloud computing becomes more and more popular IHEP has built its own cloud platform based on OpenStack IHEP has built its own cloud platform based on OpenStack More and more applications have moved to this platform More and more applications have moved to this platform Compute and storage resources becomes flexible, however, under the traditional network architecture, network capability becomes the bottleneck of restricting the flexible application of cloud computing. Compute and storage resources becomes flexible, however, under the traditional network architecture, network capability becomes the bottleneck of restricting the flexible application of cloud computing. So we introduce a SDN So we introduce a SDN Fazhi Qi/IHEPCC

Goals Bandwidth control (apply/release/adjustment) based on SDN Bandwidth control (apply/release/adjustment) based on SDN Integrated with OpenStack Integrated with OpenStack Efficient flow table installation: Multi-controller supported Efficient flow table installation: Multi-controller supported Fazhi Qi/IHEPCC

Testbed SDN controller: ODL Helium SDN controller: ODL Helium OpenStack Neutron plugin: OpenStack Juno with ML2 plugin OpenStack Neutron plugin: OpenStack Juno with ML2 plugin Northbound API: REST Northbound API: REST Southbound protocols:OF1.3 Southbound protocols:OF1.3 Network element: OVS Network element: OVS Fazhi Qi/IHEPCC

Multi-controller Survey Survey HyperFlowHyperFlow ONIXONIX PalettePalette The above three all focus on the information synchronization while increase the installation time of the flow tableThe above three all focus on the information synchronization while increase the installation time of the flow table Hierarchy management: Hierarchy management: Top level controller: Core switchTop level controller: Core switch Second level controller: TOR switchSecond level controller: TOR switch Fazhi Qi/IHEPCC

What we have done & future work Have done Have done ODL integrated with OpenStack in backendODL integrated with OpenStack in backend Designed the multi-controller solution, but not yet deployed successfullyDesigned the multi-controller solution, but not yet deployed successfully Future Work Future Work Modify OpenStack to present ODL control web pageModify OpenStack to present ODL control web page Deploy the multi-controller solution successfully and efficientDeploy the multi-controller solution successfully and efficient Load balance between ODL and switchesLoad balance between ODL and switches Fazhi Qi/IHEPCC

WAN A private virtual network across Chinese HEP members based on SDN architecture A private virtual network across Chinese HEP members based on SDN architecture Fazhi Qi/IHEPCC

Goals & Thoughts Goals & Thoughts Goals Goals Improve the data exchange performance, based on the currentImprove the data exchange performance, based on the current  Network infrastructure  Applications Thoughts Thoughts Overlay: use IPv4 & IPv6 network link/SOverlay: use IPv4 & IPv6 network link/Separately or Aggregately Automatically and Dynamically network path choosing based on the application requirements and network performance status,Automatically and Dynamically network path choosing based on the application requirements and network performance status, Software Defined/Controlled (SDN)Software Defined/Controlled (SDN) Fazhi Qi/IHEPCC

Application Scenarios SiteA SiteB SiteC Host A Host B Host C ✗ Fazhi Qi/IHEPCC

Route Switching SiteA SiteB SiteC Host A Host B Host C Fazhi Qi/IHEPCC

Route Switching SiteA SiteB SiteC Host A Host B Host C Fazhi Qi/IHEPCC

Bandwidth Merging (Aggregating paths) SiteA SiteB SiteC Host A Host B Host C Fazhi Qi/IHEPCC

Overlay Overlay IPv4OverIPv6IPv4OverIPv6 VPN ： GRE tunnel,VPN ： GRE tunnel, Efficiency: >95% Efficiency: >95% SDN Controller SDN Controller Based on OpenDaylightBased on OpenDaylight North-API: RESTAPINorth-API: RESTAPI South-API: OpenFlow, SNMPSouth-API: OpenFlow, SNMP The Key Technologies SDN Boundary Switch SDN Boundary Switch OpenFlow 1.3OpenFlow 1.3 VPN functionVPN function Link performance probingLink performance probing Fazhi Qi/IHEPCC

The Key Technologies  Link Quality Detection Packet loss Packet loss Latency Latency Link Bandwidth Link Bandwidth Available bandwidth Available bandwidth Historical data statistics Historical data statistics  Dynamic path selection algorithm 3 tuple 3 tuple S-IP,D-IP,Protocol S-IP,D-IP,Protocol 5 tuple 5 tuple S-IP,s-port,D-IP,d-port,protocol S-IP,s-port,D-IP,d-port,protocol Fazhi Qi/IHEPCC pathPacket loss latencyAvailable bandwidth Site A->site B(ipv4) Site A->site B(ipv6) Site A->site C->site B (ipv4) Site A->site C->site B (ipv6) Site A->(ipv4)site C-> (ipv6)site B Site A->(ipv6)site C-> (ipv4)site B

Routing Switching algorithm Fazhi Qi/IHEPCC

Current Progress Phase I Phase I Aug. 2014Aug SJTU,SDU,IHEPSJTU,SDU,IHEP Only IPv6 link was usingOnly IPv6 link was using Manual path selectionManual path selection ProblemsProblems  IPv6 link performance was unstable Phase II Phase II Oct.2015Oct.2015 SJTU,SDU,IHEP,CCNUSJTU,SDU,IHEP,CCNU IPv6 & IPv4IPv6 & IPv4 Automatic path selectionAutomatic path selection Performance improvementPerformance improvement  testing Fazhi Qi/IHEPCC

IHEPCCNU SDU SJTU Network topology Network topology SJTU (Shanghai Jiao Tong University), SDU (Shandong University), CCNU (Central China Normal University), IHEPSJTU (Shanghai Jiao Tong University), SDU (Shandong University), CCNU (Central China Normal University), IHEP IPv4 & IPv6IPv4 & IPv6 Fazhi Qi/IHEPCC Deployment status

Controller dashboard SDN Switch info info Flow table info Fazhi Qi/IHEPCC

Controller dashboard – link performance statistics Availablebandwidth(ipv4) Availablebandwidth(ipv6) latency(ipv6) Fazhi Qi/IHEPCC

Controller dashboard- Path selection parametersetting pathSelectionresult Fazhi Qi/IHEPCC

IHEP SDU Results (Girdftp) IHEP<->SDU:IPv4 IHEP<->SDU:IPv6  IPv6 is much better than IPv4, 10~100 times increased Fazhi Qi/IHEPCC

IHEP SJTU Results(Girdftp) IHEP<->SJTU:IPv4 IHEP<->SJTU:IPv6  IPv6 is 10 times of IPv4 Fazhi Qi/IHEPCC

Test Results Path Traditional method （ Mbps ） SDN method （ Mbps ） Best path IHEP->SJTU IHEP->(IPv6)SJTU SJTU->IHEP SJTU->(IPv6)IHEP IHEP->SDU IHEP->(IPv6)SDU SDU->IHEP SDU->(IPv6)SJTU- >(IPv6) IHEP SDU->SJTU SDU->(IPv6)SJTU SJTU->SDU SJTU->(IPv6)SDU SDU->IHEP shortest path is not very good / Packet loss SDU->IHEP shortest path is not very good / Packet loss SDN Transfer performance has been improved a lot ( Even using the long path ) SDN Transfer performance has been improved a lot ( Even using the long path ) Fazhi Qi/IHEPCC

Future plan: SDN  CHEPDTN Fazhi Qi/IHEPCC

HEPiX Spring Talk given by Shawn McKee Future plan (2):

Summary  Center research is starting ……  is in production  The testbed is a centrally-managed virtual private network linking the participating sites in order to use the available IPv4 and IPv6 network links to exchange physics data of those Universities and IHEP  Decisions on how to route traffic are taken by a controller application based on data collected on the observed activity of the links (via iperf, SNMP, network flow data, activity logs, etc.).  An initial version of a web-based interface for administering the system was developed for this testbed.  The preliminary results of this experimentation look promising and the team intends to continue working on this subject to improve the performance and the efficiency…… Comments & suggestions ： Comments & suggestions ： Fazhi Qi/IHEPCC

Thanks for your attention ! MERCI ! 谢谢！ Fazhi Qi/IHEPCC