WLTP CoP Procedure for CO2/FC Analysis of CoP data and JRC Proposal assessment B. Ciuffo, A. Marotta, M. Ktistakis WLTP CoP TF telco June 6, 2019

Conformity of production (CoP) CoP is a means of evidencing the ability to produce a series of products that comply with the specifications, performance and marking requirements demonstrated and outlined in the type approval documentation. CoP should thus be sufficiently robust to ensure the conformity of what is produced with what has been demonstrated during type- approval Given the complexity of the “vehicle” product, CoP is not a trivial task as there are many uncertainties In the production process In the testing method

Conformity of production (CoP) Managing the existing uncertainty is a task that should not be underestimated It is necessary to ensure sufficient evidence to base the conformity upon It is important to follow sound statistical procedures in order to have sufficient confidence on the results of the assessment In particular, what we want to achieve at the end of the CoP is sufficient confidence that there is no significant difference between the emissions and CO2/FC measured during the type- approval process (thus representative of the entire population) and what is measured from the population sample

Conformity of production (CoP). Conditions Population. During the emission type approval, CO2 results are compared with a declared value (DV). DV is accepted if the resulting CO2 is lower than DV When it comes to CoP, there are 3 reasons why DV should be higher than the population mean: Customer protection (the CO2/FC declared on the CoC should be higher than the individual vehicle CO2/FC value under WLTP, for the majority of the consumers); Imposing a margin between DV and population mean creates an incentive at production level to improve the quality and decrease the CO2/FC spread; Assuming that DV could be equal to the population mean would penalise manufacturers with higher production quality standards. Confidence Interval. In the hypothesis that the different CoP samples are uncorrelated and that the production process does not change over time, confidence intervals can be the right tool to assess whether the population is in line with the expectation with a certain confidence Confidence interval around the mean with 90% or 95% confidence can be used

Conformity of production (CoP). Approach In order to use a pragmatic approach, 16 can still be the maximum sample size The boundaries of the confidence interval are calculated using a Student distribution with (N-1) degrees of freedom and the standard deviation of the sample DV is assumed to be the upper bound of the acceptance region in case of 16 tests The assumption on DV leads to the identification of the theoretical mean of the CO2 distribution. This depends on the confidence level and the standard deviation of the distribution (either the theoretical value or the sample value)

Conformity of production (CoP). Acceptance-Rejection Region (s=3%) m=DV-(t90,(16-1)*s/√16) DV m

Assessment of the method (1) The method was tested using a simulation approach. In particular 100.000 combinations of 16 randomly generated results of CoP tests have been derived from a population following a Normal distribution having mean (m) standard deviation (s) Simulation results derived in terms of Probability for a vehicle to have CO2 higher than the declared value (Pr{x<DV}): Defective rate Overall Pass/Fail probability Pass/Fail rate at the end of each CoP step

Assessment of the method (2) The method was also tested using the CoP data provided by France, Germany and The Netherlands. In particular 100.000 combinations of 16 results of CoP tests have been randomly sampled from the available data Simulation results derived in terms of Overall Pass/Fail probability Pass/Fail rate at the end of each CoP step Before applying the method, the three datasets were analysed and centred around a common mean value to remove the variability due to the selection of the declared value for the specific family

French dataset. Preprocessing Raw data Data centered around 100% min 1st Qu median mean 3rd Qu max sd 83.40 92.50 94.70 94.60 96.80 102.50 3.10 min 1st Qu median mean 3rd Qu max sd 88.34 98.66 100.07 100.00 101.48 107.28 2.31

French dataset. Assessment of JRC method Raw data N. tests Pass Rate (%) Fail Rate 3 46.514 0.002 4 20.828 0.000 5 13.387 6 8.308 7 4.977 8 2.779 9 1.508 10 0.833 11 0.435 12 0.219 13 0.107 14 0.054 15 0.025 16 0.024 Total 99.998 Data centered around 100% N. tests Pass Rate (%) Fail Rate 3 2.50 8.87 4 0.81 7.21 5 0.40 6.70 6 0.24 6.01 7 0.12 5.30 8 0.09 4.76 9 0.05 4.22 10 0.04 3.80 11 0.03 3.41 12 0.02 3.19 13 2.89 14 0.01 2.67 15 2.54 16 31.68 2.41 Total 36.03 63.97 Data centered around 99% N. tests Pass Rate (%) Fail Rate 3 8.32 1.97 4 3.72 1.39 5 2.40 1.17 6 1.88 0.98 7 1.48 0.87 8 0.77 9 1.02 0.68 10 0.63 11 0.79 0.54 12 0.69 0.50 13 0.47 14 0.56 0.40 15 0.39 16 64.79 Total 88.87 11.13 Data centered around 98% N. tests Pass Rate (%) Fail Rate 3 19.96 0.26 4 9.84 0.11 5 7.07 0.06 6 5.81 0.05 7 4.99 0.03 8 4.22 9 3.82 10 3.16 0.02 11 2.85 0.01 12 2.65 13 2.36 0.00 14 2.28 15 1.98 16 28.44 Total 99.41 0.59

German dataset. Preprocessing Raw data Data centered around 100% min 1st Qu median mean 3rd Qu max sd 85.47 92.58 94.86 95.01 97.27 105.26 3.44 min 1st Qu median mean 3rd Qu max sd 93.55 99.02 100.03 100.00 100.99 106.22 1.70

German dataset. Assessment of JRC method Raw data N. tests Pass Rate (%) Fail Rate 3 38.83 0.03 4 16.64 0.01 5 10.76 0.00 6 8.23 7 6.41 8 4.73 9 3.65 10 2.66 11 2.04 12 1.55 13 1.19 14 0.88 15 0.63 16 1.76 Total 99.96 0.04 Data centered around 100% N. tests Pass Rate (%) Fail Rate 3 2.48 8.12 4 0.84 7.02 5 0.43 6.61 6 0.25 5.69 7 0.14 5.10 8 0.10 4.76 9 0.08 4.21 10 0.05 3.87 11 0.02 3.56 12 0.03 3.18 13 2.94 14 0.01 2.72 15 2.64 16 32.73 2.36 Total 37.22 62.78 Data centered around 99% N. tests Pass Rate (%) Fail Rate 3 11.99 0.89 4 5.86 0.58 5 4.23 0.48 6 3.21 0.41 7 2.61 0.34 8 2.34 0.29 9 1.96 0.24 10 1.82 0.22 11 1.64 0.18 12 1.51 0.16 13 1.35 0.14 14 1.22 0.12 15 1.19 16 54.79 0.10 Total 95.73 4.27 Data centered around 98% N. tests Pass Rate (%) Fail Rate 3 31.13 0.06 4 15.16 0.02 5 10.29 0.01 6 7.65 0.00 7 6.00 8 4.65 9 3.85 10 3.22 11 2.63 12 2.25 13 1.93 14 1.9 15 1.38 16 8.17 Total 99.90 0.10

Dutch dataset. Preprocessing Raw data Data centered around 100% min 1st Qu median mean 3rd Qu max sd 84.03 92.45 94.73 94.70 96.85 102.46 2.75 min 1st Qu median mean 3rd Qu max sd 90.88 98.88 99.98 100.00 101.34 105.40 1.85

Dutch dataset. Assessment of JRC method Raw data N. tests Pass Rate (%) Fail Rate 3 50.698 0.001 4 23.394 0.000 5 13.457 6 6.491 7 3.089 8 1.575 9 0.683 10 0.330 11 0.156 12 0.074 13 0.029 14 0.015 15 0.006 16 0.002 Total 99.999 Data centered around 100% N. tests Pass Rate (%) Fail Rate 3 2.73 8.21 4 0.94 7.15 5 0.47 6.68 6 0.27 5.95 7 0.18 5.28 8 0.11 4.72 9 0.07 4.27 10 0.04 3.96 11 3.48 12 0.03 3.19 13 0.01 2.86 14 0.02 2.72 15 2.53 16 31.72 2.38 Total 36.64 63.36 Data centered around 99% N. tests Pass Rate (%) Fail Rate 3 11.46 1.78 4 4.89 0.77 5 3.06 0.68 6 2.26 0.57 7 1.94 0.45 8 1.69 0.40 9 1.49 0.34 10 1.39 0.25 11 1.19 0.23 12 1.13 0.22 13 1.05 0.20 14 0.95 0.16 15 0.87 1.16 16 60.29 0.12 Total 93.65 6.35 Data centered around 98% N. tests Pass Rate (%) Fail Rate 3 25.51 0.10 4 11.34 0.04 5 9.82 0.02 6 8.04 0.00 7 6.48 0.01 8 5.46 9 4.63 10 3.79 11 3.16 12 2.79 13 2.47 14 2.15 15 1.84 16 12.34 Total 99.82 0.18

Summary of the results achieved French German Dutch JRC simulation Mean (Raw data) 94.60% 95.01% 94.70% -- Sd (Raw data) 3.10% 3.44% 2.75% Sd (Centered data) 2.31% 1.70% 1.85% 2% Total pass rate (Raw data) 99.99% 99.96% Total pass rate (Centered data 100%) 36.03% 37.22% 36.64% 38% Total pass rate (Centered data 99%) 88.87% 95.73% 93.65% 91.4% Total pass rate (Centered data 98%) 99.41% 99.90% 99.82%

Histograms of data cantered around 100% Overview of CoP data distributions Histograms of raw data Histograms of data cantered around 100%

Conclusions The presentation provides an analysis of the CoP data submitted by France, Germany and The Netherlands and an assessment of the JRC proposal for the CoP of CO2/FC. Main results: The three datasets present similar characteristics (Mean around 95% and Std around 2%) and show that CO2 is normally distributed The assessment of JRC proposal for CoP confirms the results obtained by the previous simulations in terms of overall pass/fail and impact of the standard deviation of the sample In addition it shows that with respect to the current CoP approach, in order to have the same pass probability, OEMs can use a declared value which is significantly closer to the test results (98% instead of 95%) although increasing the number of tests required The JRC is also available to use the same approach to assess the other CoP methods proposed by the task-force

Thanks Questions? You can find me at Biagio.CIUFFO@ec.europa.eu