Screening for Abnormal Values in AirBase Datasets Oliver Kracht and Michel Gerboles European Commission - Joint Research Centre I – 21026 Ispra (VA) www.jrc.ec.europa.eu 18th EIONET Workshop on Air Quality Assessment and Management 24th and 25th October 2013 Dublin - Ireland

“Smooth Spatial Attribute Method” Objectives: Present a prototyped screening tool for abnormal values and uncertain classifications of ambient air quality monitoring stations  Methodology: “Smooth Spatial Attribute Method” (first developed for traffic sensors by Lu et al. 2003 & Shekhar et al. 2003)  Applications: AirBase records of daily PM10 values 22 February 2019

Data availability in Airbase: Public air quality database system of the European Environment Agency (EEA) Monitoring data submitted by about 35 participating countries throughout Europe 140 pollutants, more than 6000 stations and 25000 time series with hourly and daily data of more than 30 years 22 February 2019

Focus of this Exercise: records with varying time- extend from AirBase versions 4 and 7 daily PM10 values station type “Background” all area types (urban, suburban and rural – to be discussed) 22 February 2019

“Smooth Spatial Attribute Method” Proposed for traffic sensors by Lu et al. 2003 & Shekhar et al. 2003 1st quantify how the measurement value of a station deviates from the corresponding values observed within its spatio-temporal neighbourhood (the ‘Sx value’) 2nd compare this Sx-deviation to the corresponding Sx-deviations observed for the station’s neighbours Lu, CH.-T., D. Chen & Y. Kou, 2003: Detecting Spatial Outliers with Multiple Attributes. ICTAI'03, IEEE 2003. Shekhar, S., CH.-T. Lu & P. Zhang, 2003: A Unified Approach to Detecting Spatial Outliers. GeoInformatica, 7(2), 139-166. 22 February 2019

Definition of Neigbourhood in 3 Dimensions: spatial domain limited to +/- 1 spherical degrees temporal domain limited to +/- 2 days temporal domain is automatically expanded if initial neighbourhood is too little 22 February 2019

“Smooth Spatial Attribute Method” Calculation of Sx-values (for each individual neighbourhood) z-transformation of Sx using the mean and std of Sx (Sxn and sSxn) within a neigbourhood Define a reference basis θ (e.g., applying a KZ-filter to the individual zi timeseries). Test statistics for abnormal values screening (e.g., threshold value chosen to as 1.96) 22 February 2019

Example for spatio-temporal outlier screening: 1st step: log transformation of non-Gaussian data remark: AirBase v.4 nomenclature: AT0227A AirBase v.7 nomenclature: AT30104 22 February 2019

Calculate neighbourhood mean. 2nd step: Calculate neighbourhood mean. (weighted mean using inverse squared normalized Euclidian distance) 22 February 2019

Calculate Sx within individual neighbourhoods. 3rd step: Calculate Sx within individual neighbourhoods. 22 February 2019

4th step: For each station, calculate the weighted mean and weighted standard deviation of Sx values within its neighbourhood. (Sxn and sSxn) 22 February 2019

5th step: Sx values of the central station are Z-normalised (using the Sxn and sSxn of each neighbourhood). 22 February 2019

(e.g., θ +/- a predefined threshold of 1.96) 6th step: Test statistics for abnormal values searches for zi values exceeding the upper/lower limits chosen as a reference. (e.g., θ +/- a predefined threshold of 1.96) 22 February 2019

Threshold criteria applied in the outlier screening: Threshold reference Ө obtained from low pass filtering of individual stations zi time series. |zi| exceeding 1.96 not taken into account for computing Ө. Minimum number of data points required within a spatio-temporal neighbourhood (e.g., 20 neighbourhood points). Minimum number of data points required within a rolling window. Use a Kolmogorov-Zurbenko filter (with m = 5, k = 3) to obtain a smooth reference Ө. 22 February 2019

Threshold criteria applied in the outlier screening: …  Kolmogorov-Zurbenko filter (with m = 5, k = 3) to obtain a smooth reference Ө. Removes signal components with a periodicity of less than ca 8.7 days. 22 February 2019

Final Example Outcome 22 February 2019 remark: AirBase v.4 nomenclature: AT0227A AirBase v.7 nomenclature: AT30104 22 February 2019

Threshold criteria applied in the outlier screening: Threshold reference Ө obtained from low pass filtering of individual stations zi time series. |zi| exceeding 1.96 not taken into account for computing Ө. Minimum number of data points required within a spatio-temporal neighbourhood (e.g., 20 neighbourhood points). Minimum number of data points required within a rolling window. Use a Kolmogorov-Zurbenko filter (with m = 5, k = 3) to obtain a smooth reference Ө. non verifiable 22 February 2019

Systematic deviation from neighbourhood

Automated Data Processing All codes prototyped in the R environment Directly coupled to postgreSQL database 22 February 2019

Inherent challenges in the method: 22 February 2019

Limited availability of neighbourhood information Examples: availability of background station records Example 1: reasonably distributed spatial neighbourhood 22 February 2019

Limited availability of neighbourhood information Examples: availability of background station records Example 2: “asymmetric” spatial neighbourhood 22 February 2019

Limited availability of neighbourhood information Examples: availability of background station records Example 2: “asymmetric” spatial neighbourhood Consider investigating transboundary datasets. 22 February 2019

Example 3: changing neighbourhood over time 22 February 2019

Changing neighbourhood needs to be dynamically accounted for in the automated data processing (-> done). Maybe it is useful to flag a significant change of the group of stations within a neighbourhood which would explain sudden inset of abnormal station values (-> to do). 22 February 2019

Summary of Outcomes 2006 / 2007 records of AirBase v.4 22 February 2019

Some more aspects: Reprocessing with longer time series (AirBase v.7)

Influence of station-area type selections used urban, suburban and rural used urban and suburban only Sx mean and std of neighbourhood are changing, causing a change in the normalization and in the reference.

Preliminary Results and Conclusions Processed 2006 / 2007 AirBase records of daily PM10 values for a selection of 8 countries (AT, CZ, DE, ED, FR, GB, IT and NL). Content of identified abnormal datapoints typically ranges between 4% and 10% of the records within each individual country dataset. Number of non-verifiable records typically ranges between 1% and 15% per individual country (limitations of network design, e.g. to few neighbours). Figures about abnormal datapoints content are dependent on the parameter values chosen in the screening method. An absolute definition for abnormal records is not feasible, but depends on the intended objectives for using the method.

Preliminary Results and Conclusions Demonstrated extension to longer time series with AirBase v.7. We anticipate that the screening tools can be a useful AirBase post-processing tool for Modellers Preparation of data summaries Spatial and temporal trend analysis Statistical evaluations May also support QA/QC with a short feedback cycle for network operators when implemented in real or near to real time mode

Open Questions Is there a need to derive a harmonized set of screening tools parameters through collaboration? Or better leave this open to the end-user's choice? Adjustable parameter settings: spatial domain: +/- 1 spherical degrees temporal domain: +/- 2 days temporal domain automatically expanded if neighbourhood is too little test statistics: θ +/- predefined threshold of 1.96 thresholding reference Ө obtained from low pass filtering of zi time series: |zi| exceeding 1.96 not taken into account for computing Ө. Minimum number of data points within a spatio-temporal neighbourhood: 20 Minimum number of data points required within a rolling window. KZ filter with m = 5, k = 3

Open Questions Identify the circle of interested end-users (EOINET community, modelling community (FAIRMODE), EEA …). Is there a need to derive a harmonized set of screening tools parameters through collaboration? Or better leave this open to the user's choice? How to report results (graphs / tables / quantitative point per point information / simple flagging / aggregated statistics)? Structure for future implementations?

Thank you for your attention! 22 February 2019

Comparison with a conventional outlier screening approach Step 1: calculate neighbourhood standard deviation 22 February 2019

“Classical approach” Step 2: use the Z-score of log-transformed measurement values as an outlier criterion? 22 February 2019

Use the Z-score of log- transformed values as an outlier criterion? Z-score of log-transformed values does not provide a conclusive outlier criterion for this application. Spatial and temporal trend cannot be considered in this way. 22 February 2019