1. International Comparisons of Network Time Protocol Servers Michael A. Lombardi, NIST, USA, lombardi@nist.gov Judah Levine, NIST, USA J. Mauricio Lopez and Francisco Jiménez, CENAM, Mexico John Bernard and Marina Gertsvolf, NRC, Canada Harold Sanchez and Oscar G. Fallas, ICE, Costa Rica Liz Catherine Hernández Forero, INM, Colombia Ricardo José de Carvalho and Mario N. Fittipaldi, ONRJ, Brazil Raul F. Solis, CENAMEP, Panama Franklin Espejo, IBMETRO, Bolivia PTTI 2014, 12/03/2014, Boston

2. The use of NTP servers at national timing labs in the SIM region Description of NTP measurement system Direct comparisons of NTP servers to UTC(NIST) Common-view comparisons of NTP servers Causes and effects of network asymmetry Summary Outline

3. The Interamerican Metrology System (SIM) region covers North, Central, and South America, and the Caribbean Islands

4. Introduction: The Use of NTP Servers in the SIM Region The Network Time Protocol (NTP) synchronizes clocks via the public Internet. Many national timing laboratories now utilize NTP as the dominant or only method for distributing the national time. NTP is usually the first time service offered by a new national timing lab because The Internet has become ubiquitous for all types of communication, including time transfer NTP servers are much more useful than telephone time systems and much cheaper to acquire and maintain than radio time systems All official providers of the national time should verify the accuracy of the time they disseminate via NTP. This requirement led to the international comparison system described in this presentation.

5. Current Participants in SIM NTP Comparisons LaboratoryCountrySynchronization Source CENAMMexico 1 pps from local time scale CENAMEPPanama 1 pps from local time scale IBMETROBolivia 1 pps from oscillator disciplined to SIM Time Scale ICECosta Rica 1 pps from local time scale INMColombia 1 pps from local time scale NISTUnited States 1 pps from local time scale or NIST Automated Computer Time Service NRCCanada 1 pps from local time scale ONRJBrazil 1 pps from local time scale

6. Description of NTP Measurement System

7. NTP Measurement System The system went on-line on May 31, 2014 at NIST in Boulder, Colorado and has run continuously since that date. Can currently measure up to 20 servers and can be expanded as necessary Eight countries are currently participating in NTP comparisons, with each country providing access to either one or two servers. The reference for the measurements is a clock board synchronized with a 1 pulse per second (pps) signal from UTC(NIST) The clock board has 100 ns resolution The cable from UTC(NIST) to the clock board has a compensated delay of 480 ns The system sends an NTP request to every server every 10 s. Measurements are recorded in 10-minute segments using two different methods and then uploaded to the SIM web site. The two methods are: AVG method – the system stores the average time difference between the server clock and UTC(NIST) based on 60 requests MIN method – the system only stores the time difference when the round trip delay between the server and NIST was at a minimum (59 of the 60 NTP requests are discarded)

9. NTP Measurements

10. All data in this presentation, and live NTP measurements in progress, can be viewed at: tf.nist.gov/sim

12. Direct Comparisons to UTC(NIST)

13. All server clocks are synchronized to UTC, at least to within ± 1 ms Most of the measured time error is not due to the server clocks, but instead is due to network asymmetry The effects of network asymmetry are nearly always reduced by the MIN method The effects of network asymmetry are not necessarily a function of the distance travelled by the NTP packets. For example, servers located in Brazil (~9500 km from Boulder), can be measured with less uncertainty than other servers located much closer to Boulder.

14. 100-day comparison (ending 10/27/14) of NTP Servers to UTC(NIST)

15. ONRJ-1 – UTC(NIST) over 100-day period ending 10/27/14

16. Common-view NTP Measurements

17. Common-View Comparisons between Servers The common-view method is implemented by directly comparing two server clocks to UTC(NIST) and then subtracting the results of the two measurements. This technique is a convenient way to estimate the time difference between two server clocks. The apparent common-view time differences between two server clocks are dominated by network asymmetries, which usually do not cancel when the measurements are subtracted. This is especially true in the case of servers located in different countries that are being measured via very different network paths. When two servers that are synchronized to the same source and connected to the same network are compared to each other, then most of the network asymmetry does cancel. For example, the average time difference between two servers located at ONRJ in Brazil can be determined to within tens of microseconds by implementing the common-view method from Boulder.

18. Common-view NTP Measurements

19. Common-view comparison of ONRJ (Brazil) servers from Boulder, Colorado for 100-day period ending 10/27/2014

20. Common-View Measurement Uncertainty

22. Causes and Effects of Network Asymmetry

23. Inconsistent routing of the NTP packets Based on link availability and routing tables, the NTP packets may be sent using different hubs at different times and when travelling in opposite directions. Thus, the delay from server-to-client can differ significantly from the delay from client-to-server. Network operators control the routing tables and end users are not allowed to request specific routes for applications such as NTP. Network congestion Occurs when the amount of traffic carried by the network exceeds the amount of available bandwidth (bandwidth saturation). Periods when network congestion is worse than usual are usually identified by an increase in the round trip delay, because packets are being buffered or rerouted. Sources of Network Asymmetry

24. Round trip delays between servers under test and NIST during September-October 2014 (61 days) ServerLocationRT Delay (ms) AverageMinimumAvg – Min NIST-BColorado, USA3.22.11.1 NIST-GMaryland, USA52.151.11.0 NRCCanada47.946.91.0 CHUCanada77.263.913.3 ICE-1Costa Rica133.5126.17.4 ICE-2Costa Rica132.0122.49.6 INM-1Colombia162.5150.212.3 ONRJ-1Brazil188.7187.41.3 ONRJ-2Brazil189.8188.31.5 CENAMMexico106.072.733.3 CNM-2Mexico107.374.732.6 IBMETBolivia410.0228.5181.5 CNMEPPanama114.898.516.3

25. Inconsistent Routing of NTP Packets The number of network “hops” and the type of hops, can differ in both directions and at different times of day. They can also change as hardware and software changes are made to the network. The network routing may not change very often, but it is always subject to change. The table shows how the round trip delay between NIST and NRC was different in July (81 ms) and October (53 ms). The number of hops was reduced from 13 to 10 and the shaded area marks the hops that were different on the two occasions. July 2014October 2014 hop Min RT Delay (ms) node IPhop Min RT Delay (ms) node IP 1 <1132.246.52.11<1132.246.52.1 2 <1132.246.0.532<1132.246.0.53 3 <1132.246.0.2531 4 5206.130.255.1345 5 36205.189.32.180516205.189.32.117 6 44205.189.32.178 7 53205.189.32.182 8 64205.189.32.174 9 88207.231.245.129634198.71.45.2 10 92137.164.25.49731198.71.45.8 11 80137.164.26.2854192.43.217.223 12 81128.117.243.11954128.117.243.11 13 81140.172.2.261053140.172.2.26

26. Time difference between NRC server in Canada and UTC(NIST) for September-October 2014 (61 days)

27. Round trip delay between NRC (Canada) and Boulder, Colorado for September-October 2014 (61 days)

28. Network Congestion – Case 1 (Mexico) Internet access on the campus of the Centro Nacional de Metrología (CENAM) is obtained by a wireless microwave link from the nearby city of Querétaro. Network traffic exceeds the available bandwidth during the normal working hours at CENAM. This leads to excessive buffering of the NTP packets, causing both the network delays and the uncertainty of the time received by clients to substantially increase. The time uncertainty is much lower on the weekdays than on the weekends.

29. Time difference between CENAM server in Mexico and UTC(NIST) for 12-day period ending 10/17/2014

30. Round trip delay between CENAM (Mexico) and Boulder, Colorado for 12-day period ending 10/17/2014

31. Network Congestion – Case 2 (Bolivia) The most extreme example of network congestion recorded by the SIM system. The difference between the round trip delay, as estimated with the AVG and MIN methods, was 181.5 ms during September/October 2014, much larger than the differences recorded for the other servers. The IBMETRO server was connected to an asymmetric digital subscriber line (ADSL) provided by a local telecommunications provider. ADSL is inherently asymmetric because its downstream rate is faster than its upstream rate, thus it is not well suited for NTP time transfer. The ADSL data transfer rate in Bolivia was not particularly fast, limited to 2.5 Mb/s downstream and 1 Mb/s upstream, so the bandwidth could easily be saturated.

32. Time difference between IBMETRO server in Bolivia and UTC(NIST) for 12-day period ending 9/26/2014

33. Round trip delay between IBMETRO (Bolivia) and Boulder, Colorado for 12-day period ending 9/26/2014

34. Summary Due to its cost effectiveness and the ubiquitous nature of the Internet, NTP has become the dominant or sole method for distributing the national time in many SIM nations. We have developed a measurement system that verifies the accuracy of NTP servers that distribute the official time of SIM nations. The system makes the measurement results instantly accessible to all users. In addition to serving as a time verification tool, the system allows us to study the delay asymmetry of the public Internet. Future work could include: Expanding the system to include more servers, from both within and outside of the SIM region. Installing NTP client measurement system at multiple SIM laboratories. With a periodic data exchange, this could allow the unambiguous measurement of server clocks by fully compensating for network asymmetries.