Review of Trigger Effectiveness How Much is Enough? Rob Klausmeier, de la Torre Klausmeier Consulting, Inc. IM Solutions April 29,

2 IM Solutions Background “Clean-Scanning” is a concern with OBDII inspections. Many triggers are available to identify clean scanning: o Real-time: Real-Time Triggers use information collected during an OBDII inspection to flag potential cases of fraud as it’s occurring.  When this happens, the inspection is stopped until it’s determined that a valid inspection is being done.  Some new programs initiate video conferencing as part of determining if a valid inspection is being done. o Batch: Triggers are run on a daily, weekly, or monthly basis to identify stations/inspectors that performed questionable inspections. 2

3 IM Solutions Presentation Organization This paper evaluates triggers using data from well enforced I/M programs. o Centralized (One program) o Decentralized (Average of two programs) Evaluation Parameters: o Accuracy – How many tests are falsely triggered? o Value – How many additional cases of clean scanning are identified? 3

4 IM Solutions Possible Triggers The following triggers have been developed to identify clean scanning. Most of these triggers are applicable for real-time and batch triggers: Mismatches between OBD-VIN and VIN entered for the vehicle under test (VUT) Mismatches between expected test parameters and parameters recorded for VUT. Possible parameters: o Communication protocol and supported readiness monitors:  Expected parameters on initial test can be based on look-up table of known values or previous inspections.  Retest values can be based initial tests. o PID Count, PCM ID, CAL ID, CVN o OBD VIN on initial test but not retests o Short time between initial failing tests and passing retests 4

5 IM Solutions OBD VIN Mismatches Even though they were not mandated until the 2005 model year, OBD VINs have been available from some vehicles since 2000 through Mode $09 commands: A significant percentage of OBD VINs returned from pre vehicles have errors. For example, some VINs are recorded as: Still, with screening for VIN errors, OBD VINs from pre vehicles are very valuable. Without screening for VIN errors, over 98% of the VIN mismatches are false. SE34L8YHA05959 WBAAA00000TD65133 KMHDxxxDx1Uxxxxxx 5

6 IM Solutions OBD VIN Mismatches Percent of Tests Reporting OBD VINs – Centralized 6

7 IM Solutions OBD VIN Mismatches Impact of screening for invalid VINs: Centralized Program Without OBD VIN validity checks: 13% of the records with OBD VINs mismatched. 98% of these mismatches were invalid. With OBD VIN validity checks: 0.2% of the records mismatched. In only 0.07% of the tests did the last 6 digits mismatch. 0.11% were errors in the VIN entered for the vehicle under test. In a well enforced program, few tests will be interrupted if real-time triggers were applied to valid OBD VINs and valid VINs for the VUT. 7

8 IM Solutions OBD VIN Mismatches How many OBD VIN mismatches are due to “clean scanning” More than 50% of the valid VIN mismatches appear to be due to PCM changes. The initial VIN digits are similar. Percent of the valid VIN mismatches that had mismatches in the 1 st 2 VIN characters: Centralized Program: 38% (0.03% of the tests) Decentralized Program: 15% (0.02% of the tests) I/M program managers should consider only interrupting cases where the 1 st digits mismatch. The rest of the valid mismatches could be investigated in data audits. 8

9 IM Solutions Summary of OBD VIN Mismatch 9

10 IM Solutions Other OBD Parameter Mismatches If OBD VIN is not available, other OBD parameters can be used to identify possible clean scanning. Parameters of interest: Communication (COM) protocol. Monitors supported profile: vehicles have different supported monitors. Others? PCM ID, PID Count, CAL ID, CVN Types of OBD parameter mismatches: Mismatches between current test parameters and parameters for the same vehicle type based on a look-up table. Mismatches between initial test parameters and retest parameters. Mismatches between current test parameters and parameters for the same vehicle when tested in previous inspection cycle e.g., MA). 10

11 IM Solutions COM Protocol and Monitors Supported How effective is COM protocol and monitors supported as a fraud deterrence? Reviewed OBDII inspection data from centralized sample and determined discrete combinations of COM protocol and monitors supported. Overall, there were 35 discrete combinations. The top combinations are listed below (all combinations had Cat and O2 Sensors supported): 16%: ISO / Evap + O2 Sensor Heater + EGR 15%: ISO / Evap + O2 Sensor Heater 14%: VPW / Evap + O2 Sensor Heater + EGR From this we concluded that someone would have less than a 1 in 6 chance of substituting a vehicle with the correct profile, without being detected. If the inspector is cheating, you’ll catch him. 11

12 IM Solutions COM Protocol The most reliable fingerprint outside of OBD VIN is COM Protocol. If COM Protocol changes from the initial test or a previous test, something’s up. It’s straightforward to develop a look-up table of expected COM protocols. It’s necessary to screen out known problem vehicles where COM protocol changes due to communication with body computer instead of PCM. Problem occurs primarily in 2001 to 2003 Chrysler products. Percent of tests with mismatches based on look-up table: o Centralized: 0.025% (0.010% increment over VIN mismatch) o Decentralized: 0.020% (0.019% increment over VIN mismatch) 12

13 IM Solutions COM Protocol Effectiveness of Protocol Mismatch Trigger: o Some of the mismatches had OBD VINs recorded. Following is the % of the supported monitor mismatches where OBD VINs mismatched VINs for the VUT.  Centralized sample: 100%  Decentralized sample: 95% o In decentralized sample, most of the COM protocol mismatches occurred in vehicles that did not report OBD VINs. This trigger doubled the number of suspect tests. Many of the suspect tests were 2005 and newer models that should have reported OBD VINs, but didn’t. o Over half of the COM protocol mismatches in the decentralized sample were from stations with more than one mismatch case. Some stations had mismatch rates >40%. The vast majority (>99.5%) of stations had no mismatches. 13

14 IM Solutions COM Protocol Mismatches 14 Model YearMakeStation ID OBD VIN Available Expected ProtocolRecorded Protocol 2008CHEVR A CANVPW 2004BMW BISOKWP 2004BMW CISOKWP 1998DODGE DISOKWP 2002SUBAR EISOKWP 2003LINCO FPWMVPW 2006BMW GX MismatchISOCAN 2004FORD H PWMISO 2003VOLKS H KWPISO 2004BMW JISOKWP 2005KIA K KWPISO 2000CHEVR LX MismatchISOPWM 2000TOYOT L ISOPWM 1999FORD MPWMISO 2000FORD N PWMISO 2002FORD NPWMISO 2006ME/BE N CANISO 2008VOLKS N CANISO 2001FORD OPWMVPW 2006ME/BE P CANISO 2006BMW P KWPISO 2000SUBAR QISOPWM 2001FORD R PWMISO 2001FORD RPWMISO 2002FORD RPWMISO 1998FORD RPWMISO 1997FORD RPWMISO 2003FORD R PWMISO

15 IM Solutions Supported Readiness Monitors Supported readiness monitors is another reliable signature. It’s straightforward to develop a look-up table of expected readiness monitors supported. Percent of tests with mismatches in supported readiness monitors based on look-up table: Centralized: 0.04% (0.03% increment over VIN mismatch). Decentralized: 0.05% (0.04% increment over VIN mismatch). 15

16 IM Solutions Supported Readiness Monitors Effectiveness of Monitor Mismatch Trigger: o % of the supported monitor mismatches where OBD VINs mismatched VINs for the VUT.  Centralized: 69%  Decentralized: 48% o In the decentralized sample, most of the monitor mismatches occurred in vehicles that did not report OBD VINs. o A handful of stations in the decentralized sample had greater than 50% mismatch rates. The vast majority of stations (>99.5%) had no mismatches. 16

17 IM Solutions Summary of Mismatch Triggers 17

18 IM Solutions Conclusions concerning COM and Monitor Triggers Triggers that look for mismatches in communication protocol and supported readiness monitors reliably identify many cases of clean scanning in vehicles without OBD VINs. The above results are based on a look-up table. o Mismatches based on previous inspection results from either an earlier test cycle or an initial failed test will be even more reliable. 18

19 IM Solutions Other Possible Trigger Parameters What about PID Count and PCM ID? These were the first fingerprint parameters to identify clean scanning. These parameters are most feasible if only one equipment vendor is used. Even then, inconsistencies are reported for the same vehicle, especially for PID Count. Surprisingly, PCM ID does not help as much as you may think. Based on data from the centralized sample, adding PCM ID to the signature slightly improves the uniqueness of the signature. The top combinations of PCM ID/COM Protocol/ and Monitor Supported are listed below: 13.6%: PCM ID 16 / VPW/ Evap + O2 Sensor Heater + EGR 12.4%: PCM ID 16 / PWM/ Evap + O2 Sensor Heater + EGR 11.3%: PCM ID 16 / VPW/ Evap + O2 Sensor Heater 19

20 IM Solutions Other Possible Trigger Parameters Review of Vehicle Fingerprint Tools – Highest fraction of fleet with parameter combination 20

21 IM Solutions Other Possible Trigger Parameters What about CAL-ID and CVN? These parameters have been proposed as additional tools to identify clean scanning. Most vehicles that provide CAL-ID or CVN also provide OBD VINs: 94% of the vehicles reporting CAL-ID also reported OBD VINs. 96% of the vehicles reporting CVN also reported OBD VINs. CAL-ID and CVN may prove to be useful for investigations, but they are not productive triggers, especially for real-time applications. 21

22 IM Solutions Other Possible Triggers vehicles without OBD VINs o This trigger would be useful in investigations, particularly cases where initial failing tests had OBD VINs while passing retests did not have them. o If this criteria were used as a real-time trigger, the number of interrupted tests would more than double. o May be a useful batch trigger. Stations with high percentages of vehicles without OBD VINs could be investigated. o Monitor mismatch and/or profile mismatch triggers will identify many of the cases where vehicles do not provide OBD VINs 22

23 IM Solutions Other Possible Triggers Short-Time Between Failing and Passing Tests o Vehicles that pass OBDII tests shortly after failing them are suspect:  If DTCs are cleared as part of the repair, most vehicles must be driven a while to set readiness monitors.  If the technician does not clear DTCs, most vehicles must be driven a while to turn off the MIL. o A review of data from the centralized program found that 4 passing tests (out of 26,000 tests) occurred within ½ hour of the failing test:  All four tests involved vehicles that exceeded the not ready limit by one monitor. 23

24 IM Solutions Conclusions The most reliable triggers are: o Mismatches between OBD VIN and VIN for vehicle under test (VUT). In-valid OBD VINs and entered VINs must be screened-out; otherwise, 98% of the OBD VIN mismatches will be false. o COM Protocol and/or Monitors Supported mismatch values for previous inspections, retests or look-up tables.  PID-Count and PCM-ID provide little additional benefit over COM Protocol and/or Monitors Supported. o The above triggers can be used as real-time triggers, with appropriate screening. For example, states should consider basing real-time triggers for OBD VIN mismatches on the first 2 VIN digits. CAL-ID and CVN mismatches provide minimal additional benefits over OBD VIN mismatches. Looking for tests on vehicles that do not have OBD VINs could be an effective batch trigger. Flagging passing OBD retests that occur shortly after the failing test is likely to be an effective trigger. 24