1. Road Sensor Data Marco Puts
THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

2. Road sensors Road sensor data
Passing vehicle counts for each minute (24/7) at about sensors in the Netherlands
Types of sensors:
Induction loop
Camera
Bluetooth
Length categories (e.g. small, medium, long vehicles)
Large volume: approx. 230 mln records/day

3. Dutch Highways

4. Sensors

5. Statistical Process 5 Transform & Select Calibration Clean Estimate
Big Data processing
Transform & Select 1 a
Calibration
Big Data processing 2
Clean 3 4 5 b
Estimate c 6 5

6. Data Size 6 Examples of Big Data processing Transform + Select
Cleaning
Estimation
Raw Data
4 TB
Transformed
Data
10 GB
Clean Data
500 MB
Traffic Index
6 KB
Examples of Big Data processing 6

7. Processing Traffic loop data
3 steps
1) Read files
CSV, 1 million records, 48 variables
2) Transform and Reduce dataset
Delete Data error =1, select “all_vehicles”, split dates to year and day, calculate #vehicles per day
Select only 12 variables
3) Save results

8. Process Road Sensor data

9. File size vs processing time

10. Quality of the Data (single sensor)

11. Cleaning the Data Hidden Markov ModelX1 X2 X3 X4 Y1 Y2 Y3 Y4
observation
state
𝑥 𝑘 = 𝑥 𝑘−1 + 𝜀 𝑝
𝑦 𝑘 =𝑃𝑜𝑖𝑠𝑠( 𝑥 𝑘 )

12. Cleaning the Data Recursive Bayesian Estimation

13. Cleaning the Data Recursive Bayesian Estimation

14. Cleaning the Data Recursive Bayesian Estimation

15. Cleaning the Data Recursive Bayesian Estimation

16. Results of the filter

17. Monitoring Quality
Number of minutes for which data is available varies per day per sensor
Filter fills in blocks of missing values. For large blocks, the estimation of missing values is less accurate.
Minimal deviation between original non-missing values and resulting signal.
Smoothness of resulting signal

18. Resulting Indicators Number of Measurements Block indicator𝑀
Block indicator
For each block:
Difference between data and signal
Smoothness of the signal
𝑁(𝑁+1) 2
𝐷 = 𝑘∈𝑀 𝑥 𝑘 𝑘∈𝑀 𝑦 𝑘 −1
𝑆= 1 𝐾 𝑘=1 𝐾 𝑦 𝑘 − 𝑦 𝑘− 𝑦 𝑘 + 𝑦 𝑘−1 2

19. A closer look…

20. A closer look…

21. Sensor data Data from a single sensor: Raw data Filtered data
Vehicle count
Time [minutes]

22. Calibration method
Survey statistics Target population (unit=person) Sample of respondents Questionnaire Based on demographics
Traffic statistics
Roads (unit=km)
Road sensors
Sensor data
Based on location …
Frame
Selection
Measurements
Weights

23. Road selection
Dutch Highways
Main routes only

24. Metadata input shape file of Dutch roads Road sensor metadata Road
Direction
Type
Lat
Long
A79
West
Main
5.7502
5.7625
5.7737
5.8082
5.8650 …

25. Map projection Amersfoort: projection centre
Dutch National Grid (Rijksdriehoekstelsel)
Preserves real-world distances

26. Main routes
Simplify
Raw shape
Main routes

27. Projections Project road sensors on main routes
Determine points of bifurcation for all entrance and exit ramps
Exit ramp
Entrance
ramp

28. Visual inspection tool

29. Automatic sensor metadata editing
Check and (if necessary) correct traffic flow direction
Projection of road sensors on roads
Group sensors with unique location
Remove outliers
Raw sensor metadata
Edited sensor metadata

30. Calibration of the road sensors
Main route
Exit ramp
Entrance ramp
3000
2000
2500
1000
500
Road segments (=weights)

31. Calibration of the road sensors (2)
Traffic flow
3000
2000
2500
1000
500
Count
3000
3000
2000
2000
2500 20
Length 5 5 10 7
5* * * * *2500 = 107,500 vehicle-km

32. Dimension reduction day level
Daynumber (Since )

33. Dimension reduction PCA
2D dataset
Intrinsic: 1D
Decorrelate data by rotation

34. Principal Component analysis (see Jolliffe, 2002)
Normalized dataset 𝐴 :
Mean = 0
SD = 1
𝐴 𝑇 𝐴 is the covariance matrix
Find eigenvalues 𝚲 and eigenvectors 𝑿 such that:
𝑨 𝑻 𝑨𝑿=𝑿𝚲
Order eigenvector from high to low eigenvalues
The components of A will be:
𝐶= 𝑋 𝑇 𝐴

35. Three components
Component 1 (70%)
Daynumber (Since )

36. Three components
Component 2 (5%)
Carnaval
Heavy snow
Heavy snow

37. Three components
Question:
Why does
Component 3 (5%)
Heavy snow

38. Three components day level
The cleaned data can be seen as a linear combination of the three components a +b +c

39. Three components minute levela +b +c

40. A79 South Limburg

41. References
Jolliffe, I (2002). Principal Component Analysis (2nd edition). Berlin: Springer.
Puts, M.J.H., Tennekes, M., Daas, P.J.H., de Blois, C. (2016). Using Huge Amounts of Road Sensor Data for Official Statistics. European Conference on Quality in Official Statistics (Q2016).