1. The Status and Plans for SDN Projects @ IHEP Fazhi Qi CC,IHEP04-25-2016 Fazhi Qi/IHEPCC 1

2. Agenda About IHEP & IHEPCC About IHEP & IHEPCC Why SDN @IHEP? Why SDN @IHEP? How SDN @IHEP? How SDN @IHEP? Progress & Plan Progress & Plan Summary Summary Fazhi Qi/IHEPCC

3. Agenda About IHEP & IHEPCC About IHEP & IHEPCC Why SDN @IHEP? Why SDN @IHEP? How SDN @IHEP? How SDN @IHEP? Progress & Plan Progress & Plan Summary Summary Fazhi Qi/IHEPCC

4. IHEP at a Glance ~1500 staffs, with ~1200 scientists and engineers ~1500 staffs, with ~1200 scientists and engineers Four(Six) sites currently Four(Six) sites currently Beijing, Dongguang(CSNS), Shenzhen (dayabay), Tibet (Yangbajing), Jiangmen (JUNO), Chengdu (LHASSO)Beijing, Dongguang(CSNS), Shenzhen (dayabay), Tibet (Yangbajing), Jiangmen (JUNO), Chengdu (LHASSO) The largest fundamental research center in China with following research fields: The largest fundamental research center in China with following research fields: Experimental Particle PhysicsExperimental Particle Physics Theoretical Particle PhysicsTheoretical Particle Physics Astrophysics and cosmic-raysAstrophysics and cosmic-rays Accelerator Technology and applicationsAccelerator Technology and applications Synchrotron radiation and applicationsSynchrotron radiation and applications Nuclear analysis techniqueNuclear analysis technique Computing and Network applicationsComputing and Network applications …

5. Fazhi Qi/IHEPCC Major Projects BEPCII/BESIII BEPCII/BESIII 36 Institutes from China, US, Germany, Russian, Japan,…36 Institutes from China, US, Germany, Russian, Japan,… 5PB data in 5 years 5PB data in 5 years Daya Bay Neutrino experiment Daya Bay Neutrino experiment 39 Institutes from China, US, …39 Institutes from China, US, … 400TB/year data collected400TB/year data collected Yangbajing in Tibet Yangbajing in Tibet Cosmic-ray observatory,Cosmic-ray observatory, Collaborations of China, Italy, JapanCollaborations of China, Italy, Japan ~200TB raw data per year.~200TB raw data per year. JUNO JUNO Jiangmen Underground Neutrino ObservatoryJiangmen Underground Neutrino Observatory ~ 1PB Raw date per year~ 1PB Raw date per year LHAASO LHAASO the Large High Altitude Air Shower Observatorythe Large High Altitude Air Shower Observatory ~ 2PB Raw data pear year~ 2PB Raw data pear year CSNS CSNS Chinese Spallation Neutrons SourceChinese Spallation Neutrons Source LHC LHC Members of ATLAS and CMSMembers of ATLAS and CMS WLCG Tire-2 at IHEPWLCG Tire-2 at IHEP AMS (Alpha Magnetic Spectrometer) AMS (Alpha Magnetic Spectrometer)

6. Other Projects hosted by Chinese universities and institutes CDEX, China Dark-matter EXperiment CDEX, China Dark-matter EXperiment Direct Detection of cold dark matter with ultra-low energy thresholdDirect Detection of cold dark matter with ultra-low energy threshold SRF, Shanghai SRF, Shanghai SRF, USTC, Hefei SRF, USTC, Hefei … Fazhi Qi/IHEPCC

7. * Computing Center, IHEP Computing Center, IHEP 36 + 5 Staffs, 20 project staffs ， 15 Students 36 + 5 Staffs, 20 project staffs ， 15 Students Serve for the HEP Experiments Serve for the HEP Experiments Infrastructure Operation Operation Network and Security Network and Security Computing & Storage Computing & Storage Basic IT services Basic IT services Database Database Applications Development Applications Development …… …… IHEP CC Fazhi Qi/IHEPCC 7

8. Agenda About IHEP & IHEPCC About IHEP & IHEPCC Why SDN @IHEP? Why SDN @IHEP? How SDN @IHEP? How SDN @IHEP? Progress & Plan Progress & Plan Summary Summary Fazhi Qi/IHEPCC

9. HEP Computing Challenges Rapid access to data stores & efficient data exchange Rapid access to data stores & efficient data exchange Secure, efficient, managed access to worldwide computing resources Secure, efficient, managed access to worldwide computing resources Tracking state and usage patterns Tracking state and usage patterns Matching resources to demands Matching resources to demands Enable physicists in all world regions to collaborate on analysis Enable physicists in all world regions to collaborate on analysis Building high speed intercontinental networks to support the above requirements Building high speed intercontinental networks to support the above requirements Fazhi Qi/IHEPCC

10. CSTNet Chinese Science and Technology Network Chinese Science and Technology Network One core center and twelve regional centers, covering over 30 provinces, 370 Institutes in China One core center and twelve regional centers, covering over 30 provinces, 370 Institutes in China Fazhi Qi/IHEPCC

11. CERNet Chinese Education and Research Network Chinese Education and Research Network Connects more than 200 cities and provides connectivity to about 2000 colleges and institutes in China Connects more than 200 cities and provides connectivity to about 2000 colleges and institutes in China Fazhi Qi/IHEPCC

12. IPv6 Network in China CNGI : Chinese Next Generation Internet CNGI : Chinese Next Generation Internet Backbone: 2.5~100Gbps Backbone: 2.5~100Gbps More than 25 core exchange nodes and over 600 universities and other organizations connected More than 25 core exchange nodes and over 600 universities and other organizations connected Fazhi Qi/IHEPCC

13. Current Network Status for IHEP IPv4 IPv4 Good connection for international cooperationGood connection for international cooperation A little poor for the domestic communicationA little poor for the domestic communication  Universities… IPv6 IPv6 Available bandwidth is goodAvailable bandwidth is good But unstable,But unstable,  related with different time Fazhi Qi/IHEPCC

14. Chinese HEP Communities Status IHEP is the leadership members in Chinese HEP Communities IHEP is the leadership members in Chinese HEP Communities Most of the HEP Experiments in China are hosted by IHEP Most of the HEP Experiments in China are hosted by IHEP Lack of sufficient IPv4 network bandwidth……but IPv6 resources is pretty good! Lack of sufficient IPv4 network bandwidth……but IPv6 resources is pretty good! Lack of network engineers in other HEP experiment members in China(manpower) Lack of network engineers in other HEP experiment members in China(manpower) Some times, For IHEP network team, our users is not only the IHEP users, also the other Chinese HEP community members are included …… Some times, For IHEP network team, our users is not only the IHEP users, also the other Chinese HEP community members are included …… Fazhi Qi/IHEPCC

15. How to have a good network for HEP Good reliability and availability Good reliability and availability Good performance Good performance Monitor and measure the status and performance of network Monitor and measure the status and performance of network Find the best network resources Find the best network resources Optimize network link configuration Optimize network link configuration … Fazhi Qi/IHEPCC

16. What shall we do for this? Provide a simple, flexible, robust, high performance and central － controlled network environment for HEP members in China Provide a simple, flexible, robust, high performance and central － controlled network environment for HEP members in China SDN is one of the Choices… SDN is one of the Choices… Fazhi Qi/IHEPCC

17. Agenda About IHEP & IHEPCC About IHEP & IHEPCC Why SDN @IHEP? Why SDN @IHEP? How SDN @IHEP? How SDN @IHEP? Progress & Plan Progress & Plan Summary Summary Fazhi Qi/IHEPCC

18. The key features for SDN Three level architecture: Applications-Controller-SwitchThree level architecture: Applications-Controller-Switch Central control ： simple for end usersCentral control ： simple for end users Opening for the network capability/functionOpening for the network capability/function  build the network environment on the applications needs ;  A programmatic interface between application and the network, Tight integration; direct feedback loop -> reactive system Overlay and VirtualizationOverlay and Virtualization Fazhi Qi/IHEPCC

19. It is much easier for the SDN within a data center It is much easier for the SDN within a data center We can control all the network infrastructure in the data centerWe can control all the network infrastructure in the data center  Links, Devices, Applications…… SDN @ Data CenterSDN @ Data Center But the big problem we are facing is the Internet performance among the HEP members in China But the big problem we are facing is the Internet performance among the HEP members in China SDN @WAN: a private virtual network across Chinese HEP members based on SDN architectureSDN @WAN: a private virtual network across Chinese HEP members based on SDN architecture  Intelligent data transmission network path selection algorithm  Use IPv6 network links around China (CNGI) SDN @ IHEP Fazhi Qi/IHEPCC

20. SDN @ DataCenter Science DMZ Science DMZ R&D for Cloud Computing R&D for Cloud Computing Fazhi Qi/IHEPCC

21. Science DMZ Fazhi Qi/IHEPCC

22. Science DMZ Fazhi Qi/IHEPCC Policies: S-IP S-Port D-IP D-Port Policies For Science Data-Exchange Current Based on ACL SDN Switches are under consideration for this

23. SDN @DataCenter for Cloud Computing Provided by Shan Zeng Cloud computing becomes more and more popular Cloud computing becomes more and more popular IHEP has built its own cloud platform based on OpenStack IHEP has built its own cloud platform based on OpenStack More and more applications have moved to this platform More and more applications have moved to this platform Compute and storage resources becomes flexible, however, under the traditional network architecture, network capability becomes the bottleneck of restricting the flexible application of cloud computing. Compute and storage resources becomes flexible, however, under the traditional network architecture, network capability becomes the bottleneck of restricting the flexible application of cloud computing. So we introduce a SDN solution @datacenter So we introduce a SDN solution @datacenter Fazhi Qi/IHEPCC

24. Goals Bandwidth control (apply/release/adjustment) based on SDN Bandwidth control (apply/release/adjustment) based on SDN Integrated with OpenStack Integrated with OpenStack Efficient flow table installation: Multi-controller supported Efficient flow table installation: Multi-controller supported Fazhi Qi/IHEPCC

25. Testbed SDN controller: ODL Helium SDN controller: ODL Helium OpenStack Neutron plugin: OpenStack Juno with ML2 plugin OpenStack Neutron plugin: OpenStack Juno with ML2 plugin Northbound API: REST Northbound API: REST Southbound protocols:OF1.3 Southbound protocols:OF1.3 Network element: OVS Network element: OVS Fazhi Qi/IHEPCC

26. Multi-controller Survey Survey HyperFlowHyperFlow ONIXONIX PalettePalette The above three all focus on the information synchronization while increase the installation time of the flow tableThe above three all focus on the information synchronization while increase the installation time of the flow table Hierarchy management: Hierarchy management: Top level controller: Core switchTop level controller: Core switch Second level controller: TOR switchSecond level controller: TOR switch Fazhi Qi/IHEPCC

27. What we have done & future work Have done Have done ODL integrated with OpenStack in backendODL integrated with OpenStack in backend Designed the multi-controller solution, but not yet deployed successfullyDesigned the multi-controller solution, but not yet deployed successfully Future Work Future Work Modify OpenStack to present ODL control web pageModify OpenStack to present ODL control web page Deploy the multi-controller solution successfully and efficientDeploy the multi-controller solution successfully and efficient Load balance between ODL and switchesLoad balance between ODL and switches Fazhi Qi/IHEPCC

28. SDN @ WAN A private virtual network across Chinese HEP members based on SDN architecture A private virtual network across Chinese HEP members based on SDN architecture Fazhi Qi/IHEPCC

29. Goals & Thoughts Goals & Thoughts Goals Goals Improve the data exchange performance, based on the currentImprove the data exchange performance, based on the current  Network infrastructure  Applications Thoughts Thoughts Overlay: use IPv4 & IPv6 network link/SOverlay: use IPv4 & IPv6 network link/Separately or Aggregately Automatically and Dynamically network path choosing based on the application requirements and network performance status,Automatically and Dynamically network path choosing based on the application requirements and network performance status, Software Defined/Controlled (SDN)Software Defined/Controlled (SDN) Fazhi Qi/IHEPCC

30. Application Scenarios SiteA SiteB SiteC Host A Host B Host C ✗ Fazhi Qi/IHEPCC

31. Route Switching SiteA SiteB SiteC Host A Host B Host C Fazhi Qi/IHEPCC

32. Route Switching SiteA SiteB SiteC Host A Host B Host C Fazhi Qi/IHEPCC

33. Bandwidth Merging (Aggregating paths) SiteA SiteB SiteC Host A Host B Host C Fazhi Qi/IHEPCC

34. Overlay Overlay IPv4OverIPv6IPv4OverIPv6 VPN ： GRE tunnel,VPN ： GRE tunnel, Efficiency: >95% Efficiency: >95% SDN Controller SDN Controller Based on OpenDaylightBased on OpenDaylight North-API: RESTAPINorth-API: RESTAPI South-API: OpenFlow, SNMPSouth-API: OpenFlow, SNMP The Key Technologies SDN Boundary Switch SDN Boundary Switch OpenFlow 1.3OpenFlow 1.3 VPN functionVPN function Link performance probingLink performance probing Fazhi Qi/IHEPCC

35. The Key Technologies  Link Quality Detection Packet loss Packet loss Latency Latency Link Bandwidth Link Bandwidth Available bandwidth Available bandwidth Historical data statistics Historical data statistics  Dynamic path selection algorithm 3 tuple 3 tuple S-IP,D-IP,Protocol S-IP,D-IP,Protocol 5 tuple 5 tuple S-IP,s-port,D-IP,d-port,protocol S-IP,s-port,D-IP,d-port,protocol Fazhi Qi/IHEPCC pathPacket loss latencyAvailable bandwidth Site A->site B(ipv4) Site A->site B(ipv6) Site A->site C->site B (ipv4) Site A->site C->site B (ipv6) Site A->(ipv4)site C-> (ipv6)site B Site A->(ipv6)site C-> (ipv4)site B

36. Routing Switching algorithm Fazhi Qi/IHEPCC

37. Current Progress Phase I Phase I Aug. 2014Aug. 2014 SJTU,SDU,IHEPSJTU,SDU,IHEP Only IPv6 link was usingOnly IPv6 link was using Manual path selectionManual path selection ProblemsProblems  IPv6 link performance was unstable Phase II Phase II Oct.2015Oct.2015 SJTU,SDU,IHEP,CCNUSJTU,SDU,IHEP,CCNU IPv6 & IPv4IPv6 & IPv4 Automatic path selectionAutomatic path selection Performance improvementPerformance improvement  testing Fazhi Qi/IHEPCC

38. IHEPCCNU SDU SJTU Network topology Network topology SJTU (Shanghai Jiao Tong University), SDU (Shandong University), CCNU (Central China Normal University), IHEPSJTU (Shanghai Jiao Tong University), SDU (Shandong University), CCNU (Central China Normal University), IHEP IPv4 & IPv6IPv4 & IPv6 Fazhi Qi/IHEPCC Deployment status

39. Controller dashboard SDN Switch info info Flow table info Fazhi Qi/IHEPCC

40. Controller dashboard – link performance statistics Availablebandwidth(ipv4) Availablebandwidth(ipv6) latency(ipv6) Fazhi Qi/IHEPCC

41. Controller dashboard- Path selection parametersetting pathSelectionresult Fazhi Qi/IHEPCC

42. IHEP SDU Results (Girdftp) IHEP<->SDU:IPv4 IHEP<->SDU:IPv6  IPv6 is much better than IPv4, 10~100 times increased Fazhi Qi/IHEPCC

43. IHEP SJTU Results(Girdftp) IHEP<->SJTU:IPv4 IHEP<->SJTU:IPv6  IPv6 is 10 times of IPv4 Fazhi Qi/IHEPCC

44. Test Results Path Traditional method （ Mbps ） SDN method （ Mbps ） Best path IHEP->SJTU 29.7 337IHEP->(IPv6)SJTU SJTU->IHEP 23.1 245 SJTU->(IPv6)IHEP IHEP->SDU 4.76 401 IHEP->(IPv6)SDU SDU->IHEP 4.46 189 SDU->(IPv6)SJTU- >(IPv6) IHEP SDU->SJTU 4.84 238 SDU->(IPv6)SJTU SJTU->SDU 4.89 215 SJTU->(IPv6)SDU SDU->IHEP shortest path is not very good / Packet loss SDU->IHEP shortest path is not very good / Packet loss SDN Transfer performance has been improved a lot ( Even using the long path ) SDN Transfer performance has been improved a lot ( Even using the long path ) Fazhi Qi/IHEPCC

45. Future plan: SDN  CHEPDTN Fazhi Qi/IHEPCC

46. HEPiX Spring Talk given by Shawn McKee http://indico.cern.ch/event/466991/contributions/1143612/attachments/1260140/1862168/HEPiX_WLCG-neworking.pdf Future plan (2):

47. Summary  SDN @Data Center research is starting ……  SDN @WAN is in production  The testbed is a centrally-managed virtual private network linking the participating sites in order to use the available IPv4 and IPv6 network links to exchange physics data of those Universities and IHEP  Decisions on how to route traffic are taken by a controller application based on data collected on the observed activity of the links (via iperf, SNMP, network flow data, activity logs, etc.).  An initial version of a web-based interface for administering the system was developed for this testbed.  The preliminary results of this experimentation look promising and the team intends to continue working on this subject to improve the performance and the efficiency…… Comments & suggestions ： sdn-group@maillist.ihep.ac.cn Comments & suggestions ： sdn-group@maillist.ihep.ac.cn Fazhi Qi/IHEPCC

48. Thanks for your attention ! MERCI ! 谢谢！ Fazhi Qi/IHEPCC