EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 1 Wolfgang Garche Saxony-Anhalt Environmental Protection Agency Department Air Quality Monitoring, Information, Assessment Estimation of Measurement Uncertainties

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 2 Measurement error Systematic error Random error Known systematic error Unknown systematic error CorrectionResidual error Measurement result Measurement uncertainty (Source: EUROLAB Technical Report 1/2006 „Guide to the Evaluation of Measurement Uncertanty for Qualitative Test Results“) Types of measurement error and their consideration in determining the result of a measurement and the associated uncertainty

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 3 Measured values for simultaneous occurring of random and systematic errors (Source: EUROLAB Technical Report 1/2006 „Guide to the Evaluation of Measurement Uncertanty for Qualitative Test Results“)

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 4 Uncertainty of measurement is a parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measurand. Uncertainty of the result Estimated quantity intended to characterise a range of values which contains the reference value. Definitions Standard uncertainty (u) Uncertainty of the result of a measurement expressed as a standard deviation. Combined standard uncertainty (u c ) Standard uncertainty of the result of a measurement when that result is obtained from the values of a number of other quantities, equal to the positive square root of a sum of terms, the terms being the variances or covariances of these other quantities weighted according to how the measurement result varies with changing these quantities.

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 5 Expanded uncertainty (U p ) Quantity defining an interval about the result of a measurement that may be expected to encompass a large fraction of the distribution of values that could reasonably be attributed to the measurand. Coverage factor (k) Numerical factor used as a multiplier of the (combined) standard uncertainty in order to obtain an expanded uncertainty. Accuracy The closeness of agreement between a test result and the accepted reference value. Trueness The closeness of agreement between the average value obtained from a large series of test results and an accepted reference value. Precision The closeness of agreement between independent test results obtained under stipulated conditions.

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 6 EN ISO 20988:2007 „Air quality – Guidelines for estimating measurement uncertainty“ A five-step procedure for uncertainty estimation is described: 1.Problem specification 2.Statistical analysis 3.Estimation of variances and covariances 4.Evaluation of uncertainty statements 5.Reporting

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 7 Objective is to specify -the measurement -the wanted uncertainty statement -the experimental data -effects not described by experimental data Problem specification ElementDirect approachIndirect approach experimental designone designmore than one design experimental datay(j) with j = 1 to N y R (reference value) x 1 (j) with j = 1 to N, x 1R x 2 (j) with j = 1 to N, x 2R …. Additional deviation, if appropriate YY YY

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 8 Statistical analysis ElementDirect approachIndirect approach Method model equationnot requiredy= f(x 1, x 2,…) Statistical model equation Y=y +  YY= f(x 1, x 2,…)+  Y Variance budget equation var(Y)=var(y)+var(  Y) var(Y)=c 1 ² var(x 1 )+ c 2 ² var(x 2 )+…. +2c 1 c 2 cov(x 1,x 2 )+ … var(  Y) A statistical model equation shall be established to describe the relationship between the statistical population of possible results of measurement Y and the input data.

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 9 Direct approach A direct approach provides a series of results of measurement observed in a specified experimental design and, if appropriate, expert judgement of additional deviations  Y i caused by effects not described by the series of observations variance budget equation: where: Y a possible result of measurement y a realized result of measurement (input data) dYi an additional deviation of result of measurement y not described by the experimental data (can be neglected, if the corresponding variance contributes less than 5 % to the variance estimate var(Y) used in the uncertainty estimation) statistical model equation: where: var(Y)estimate of the variance of possible results of measurement Y var(y)estimate of the variance of a series of results of measurement y var(  Y i )estimate of the variance of additional deviation  Y i, obtained by a type B evaluation

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 10 Indirect approach provides for each input quantity x i of a known method model equation y = f(x 1,..,x K ) either a series of experimental data xi(j) collected in a specified experimental design, or a variance estimate var(xi). If appropriate, additional deviations δYj, which are not described appropriately by the experimental data to be evaluated, are assessed by expert judgement. statistical model equation: where Y possible result of measurement; x i input quantity of the method model equation y = f (x 1,.., x K ) dYi additional deviation of result of measurement y not described by the experimental data variance budget equation: where c i sensitivity coefficient with respect to variations of input quantity x i cov(x i, x j ) estimate of covariance between input quantities x i and x j

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 11 Sensitivity coefficients The sensitivity coefficient c i is the partial derivative of the method model function y = f (x 1,.., x K ) c i -can be calculated numerical from the partial derivative c i -as mean value of the ratio of observed changes of the results of measurement  y(j) to the the changes of the input data  xi(j) Example: y= mass concentration of particles

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 12 Estimation of variances and covariances Type A:Evaluation using statistical analysis of measurement series The calculation method is depending on the experimental design used to collect the input data. Type B:Evaluation using means other than statistical analysis of measurement series (expert judgement) If information are available on the expected range of variation [min(δY j ) < δY j < max(δY j )] of the deviation δY j on the expected type of the statistical population of δY j Statistical population Range max(δY j ) = – min(δY j ) Estimated variance var(δY j ) Rectangularaa²/3 Triangularaa²/6

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 13 Covariances The covariance associated with the values x i and x k assigned to two input quantities of the applicable method model equation shall be zero if: x i and x k have not been observed repeatedly in the same experimental design either x i or x k was kept constant, when providing repeated observations of the other quantity repeatedly. Calculation of covariances can be avoided often by appropriate choice of experimental designs!

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 14 Evaluation of uncertainty statements Combined standard uncertainty: where var(Y) is the estimate of the variance of the population of possible results of measurement Y Relative standard uncertainty: Expanded Uncertainty:

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 15 Expanded uncertainty – coverage faktor k coverage factor k and coverage probability p shall be stated when an expanded uncertainty U p (y) is reported relationship between k and p: 1.y is a mean value (N>1) of independent observations with the same measuring system 2.y is obtained by single application of a measuring method, distribution of possible results approximately is Gaussian 3.y is obtained by single application of a measuring method, distribution of possible results are not described by a Gaussian distribution

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 16 Case 1. and 2.:k = t(p,  (direct approach) where: t(p  is the (1-p)-quantile of Students t-distribution of ν degrees of freedom p is the coverage probability of interval [–t(p,ν); +t(p,ν)] by Students t-distribution with ν degrees of freedom is the number of degrees of freedom; = N – 1 Uncertainty interval for a coverage probability: Indirect approach: Welch-Satterthwaite-equation:

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 17 A report on execution of a specified task of uncertainty estimation shall include (at least) the following items: Problem specification including -method of measurement -wanted uncertainty statement -statistical population of possible results of measurement considered -experimental data and experimental design -effects not described by experimental data statistical analysis, describing the applied statistical model equation and the variance (budget) equation evaluation methods, describing the applied evaluation methods numerical value of the wanted uncertainty statement and its range of application Reporting

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 18 Uncertainty estimation in practise What is required ?? Analysers have to fulfil the data quality objectives of the Directive 2008/50/EC “on ambient air quality and cleaner air for Europe”. (Uncertainty and minimum data capture) Analysers have to fulfil the relevant performance characteristics and criteria of the EN standards. (Type approval test) Test reports of the type approval should contain all needed uncertainty information for a specific type of analyser. But the user has to show that an analyser fulfil the requirements on measurement uncertainty also under the specific conditions on the monitoring site.  suitability evaluation

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 19 Type approval of analysers 1.the value of each individual performance characteristic tested in the laboratory shall fulfil the requirements 2.the expanded uncertainty calculated from the standard uncertainties due to the values of the specific performance characteristics obtained in the laboratory tests fulfils the requirements 3.the value of each of the individual performance characteristics tested in the field shall fulfil the requirements 4.the expanded uncertainty calculated from the standard uncertainties due to the values of the specific performance characteristics obtained in the laboratory and field tests fulfils the requirements

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 20  When a type approved analyser has been chosen for a particular measuring task, then the suitability of this analyser shall be evaluated at a specific measuring location.  The analyser at the specific site has been judged to conform with the EU data quality objectives  An expanded uncertainty calculation for the type approved analyser shall be made according to the specific circumstances at the monitoring station or site.  If the site-specific conditions are outside the conditions for which the analyser is type approved, then the analyser shall be retested under these site-specific conditions and a revised type approval will be issued.  If the analyser complies with the requirements, then that particular analyser may be installed and used at that monitoring station. General Requirements:

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 21 ParametersRemarks Sample pressure variationThe sample gas pressure variation during a whole period of a year shall be estimated. Sample gas temperature variation The sample gas temperature variation during a whole period of a year shall be estimated. Sample gas temperature may be controlled by heating or thermostating. Surrounding air temperature variation The temperature fluctuation shall be within the range specified in the type approval test. Temperature may be controlled thermostatically. Voltage variationThe voltage fluctuation shall be within the range in the type approval test. Voltage fluctuations may be controlled by means of voltage stabilizers. H 2 O concentration rangeThe H2O concentration range during a whole period of a year shall be estimated. H 2 S, NH 3, NO, NO 2 and m-xylene concentration ranges The concentration range of each compound during a whole period of a year shall be estimated. Expanded uncertainty of the calibration gas The expanded uncertainty of the calibration gas shall be included. This implies the uncertainty of the calibration gas itself as well as the uncertainty of any dilution system (where applicable) Calibration frequencyThe intended calibration frequency shall be used for the calculation of the influence of the drift.

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 22 Choosing of the needed performance characteristics from the type approval test report Evaluation or estimation of the site-specific conditions and of the uncertainty of the span gas used for calibration Calculation of the combined expanded uncertainty with inclusion of the site-specific conditions Comparison with the uncertainty requirements Operating Sequence:

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 23 Example: Uncertainty of the certification of a transfer standards Measurand y: NO concentration of the test gas Method:analyzer according to EN Input data: daily measurements of a certified NO test gas uncertainty of the test gas concentration Model equation: with Variance equation: Residual standard deviation: Standard uncertainty:

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 24 Calculated: mean value132,65 ppb residual standard dev. u(e)= 0,538 ppb Datey(j)y(j)-yR ,30, ,5-0, ,00, ,00, ,10, ,20, ,7-0, ,7-0, ,00, ,10, ,00, ,7-0, ,80, ,7-1, ,90, ,10, ,20, ,30, ,00, ,7-0, ,7-0, ,7-1, ,6-1, ,2-0, ,8-1, ,2-0, ,80, ,4-0, ,7-0, ,5-0, ,5-1,3 certified NO concentration y R = 132,8 ppb stated expanded uncertainty U(y R ) = 2% with k=2  standard uncertainty u(y R )=1,328 ppb Expanded uncertainty:

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 25 Ozone measurements: (example C.2 from EN ISO 20988) Method: automatically measurements according EN (UV-absorption) function control every 25 h with zero and span gas daily correction of zero offsets Measurand: 1-h mean value of the ozone concentration in ambient air Described effects:variations of surrounding air temperature and pressure Method model equation:y = x – e(j) x is the observed value without correction e(j) is the offset for day j Wanted uncertainty statement:standard uncertainty u(y), expanded uncertainty U 0,95 (y)

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 26 Experiment:1.every 25 hours providing of zero gas and determination of the zero offset e(j) = x 0 (j) 2.every 25 hours providing of span gas and evaluation of the span factor ß(j) = x s (j)/y s Input data:series of offset corrections (N=20) series of span factors (N=20) Reference values:Zero gas y 0 = 0 µg/m³ Span gas y s = 280 µg/m³u(y s ) = 2,8 µg/m³ Effects not described: Influence of sampling device (Influence of humidity and other compounds in the ambient air) Variance budget equation: var(y) = u²(x) + u²(e) + 2cov(x,e) cov(x,e) = 0

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 27 Index jObserved measuring value for zero gas e/j) Span factor ß(j) 1-0,71,00 2-0,90,96 3-1,40,98 4-0,90,99 5-1,11,04 6-0,31,05 7-0,81,04 8-0,81,03 9-1,01, ,01, ,91, ,81, ,11, ,81, ,81, ,61, ,51, ,01, ,71, ,00,97 Standard uncertainty of zero offset e: (includes the bias u B (e)) Model equation for x(j): Variance equation: (cov(ß,y S ) = 0) Standard uncertainty of ß: (includes the bias u B (ß) )

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 28 Standard uncertainty of y: Degrees of freedom:  = 20  coverage factor: k = t(0,05,20) = 2,1 Uncertainty depends on the corrected measured value! Now it is possible to calculate an uncertainty statement for the 1- hour mean values measured in the observed period depending on the corrected measured value.

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 29 Helpful documents and programs: EN ISO 20988“Air quality – Guidelines for estimating measurement uncertainty” Eurolab Technical Report 1/2006“Guide to the Evaluation of Measurement Uncertainty for Quantitative Test Results” Eurolab Technical Report 1/2007“Measurement Uncertainty Revisited: Alternative Approaches to Uncertainty Evaluation” Excel-Program Nordtest TR 537“Measurement Uncertainty Estimation”

EU-Twinning Project RO 2006 IB EN 09 Bucharest, Saxony-Anhalt Wolfgang Garche State Environmental Protection Agency 30 Many Thanks for your Attention!