1. Real World Test Drive – OICA views
Updated presentation based on the paper shown at the Den Haag meeting

2. Introduction/basis for discussion
The next slides are based on the document “Real world test drive” (TFAV- SG ) that OICA provided to the TF AutoVeh meeting in Den Haag.
The intention of this presentation is to start the discussion and explain a proposal on how a real world test drive can fit into the overall concept for the certification of AVs developed by OICA.
Several conceptual issues that were raised during the meeting. OICA was asked to further develop / clarify these items.
This updated presentation includes these further explanations to the original document. New sections appear in blue font.

3. Road Test for AVs: Understanding its Role in the Certification Process
What is the road test supposed to demonstrate? What is its role in the entire certification process?
What is the suggested content?
Which assessment approach is considered?
How could the road test look like from a procedural and timing perspective?

4. What is the road test supposed to demonstrate
What is the road test supposed to demonstrate? What is its role in the entire certification process (1/2)?
Hypothesis:
The road test is going to demonstrate the capability of the vehicle to adhere to traffic rules [and maneuvers according to the general expectations of other road users].
This capability is brought to the driving task currently by the experienced / approved driver.

5. What is the road test supposed to demonstrate
What is the road test supposed to demonstrate? What is its role in the entire certification process (2/2)?
The road test is an integral building block in the assessment and certification of automated vehicles. That said it is not suggested that this is the one and only deciding criteria for certification.
The road test is going to address typical / normal traffic scenarios that a human driver is exposed to on a regular basis.
After this road test the generic „competence“ of the vehicle is documented to adhere to traffic rules and the assessor has the ability to declare if it moves in traffic without becoming an obstacle.

6. Coverage of Scenarios - to be addressed according to the use case -
„Typical“ Driving –
Real World Test Drive
Demanding Traffic Scenarios –
Physical Tests
Edge Cases –
Simulation
With the approach suggested by OICA all traffic scenarios can be addressed appropriately

7. Definition of “realistic / Typical / Normal” traffic conditions
> 90 % of all road trips are „un-eventful“ because the driver does not have to deal with challenging scenarios or edge cases
During these trips the adherence to traffic rules, showcasing a behavior that is understood by other road users and participating in the traffic without being an obstacle to other road users is the prime role of the driver, i.e. the automated system in the future.
Therefore, traffic scenarios as suggested in the „checklists“ – see below – fullfil this criteria

8. What is the suggested content?
Hypothesis:
Automated/ autonomous vehicle will not operate at the beginning under all conditions and on all roads. The initial focus will be on the use cases called „highway“ and „urban“ driving.
Consequently, the content of the road test will have to be adjusted to these use casses (i.e. test scenarios of traffic situations).
Note: the minutes of the SG2 session state that the group should „start with urban situations, while ACSF continues with highway situations.“

9. What is the suggested content?
The selected scenarios will have to be derived after assessment from various sources. Ultimate goals is to generate a data base filled with traffic scenarios with which the statistical relevance of scenarios can be assessed and changes to traffic cenarios can be document.
A vehicle can – based on the input of the vehicle manufacturer – be nominated for one or more use case related road tests.
Limitations of the automated / automonous system will be reflected, assessed and documented based on the input provided by the vehicle manufacturer. This includes weather conditions, speed restrictions, non supported roads (e.g. tunnels). For identified limitations, the HMI approach needs to be assessed during the real world test drive to ensure that an appropriate hand-over is initiated by the system and that the system can recognise the limitations.

10. Which assessment approach is considered?
Hypothesis:
Based on a checklist the assessor exposes the vehicle to a pre-defined number of mandatory scenarios to maintain objectivity and comparability between road tests. Additional scenarios (supplementary ones) can be tested as well according to availability.
Comments should be provided on the checklist after a scenario has been completed indicating whether it was successful or not. Additional comments – if necessary – can be provided as well.

11. OICA proposal for checklists as integral part of the road test
Brief description of test route/location
Date/time of test drive
Item #
Situation
Pass
Comments (must be filled out in case of “no/unclear”)
Yes
No/
unclear
Part A: mandatory
All lines in Part A have to be evaluated during the test drive
HA.1
Entering the highway
HA.2
Following other vehicle in same lane
HA.3
Passing a slower vehicle: lane change/Passing/merging back in previous lane
HA.4
Adapting to changing speed limits
HA.5
Merging from an ending lane
HA.6
Exiting the highway
HA.7
HA.8
HA.9
HA.10
Suggests splitting into a mandatory and a supplementary section
All mandatory aspects need to be covered while supplementary aspects can help to refine the understanding of the vehicle performance in real traffic
Part B: supplementary
If any of the following situations is encountered during the test drive this shall be noted in the respective line.
Additional lines may be added for situations not listed which were observed.
HB.1
Situation involving an emergency vehicle (police, ambulance, fire brigade)
HB.2
Policeman or roadman directing traffic
HB.3
Objects/obstacles on the road (e.g. lost cargo)
HB.4
Driving through construction site (if possible with modified lane markings)
HB.5
Driving through area with no/bad lane markings
HB.6
Safely approaching end of traffic jam
HB.7
Driving in traffic jam
HB.8
Driving through area with bad road surface conditions
HB.9
HB.10
Additional considerations:
Across the markets (e.g. the EU) similar but not same traffic rules and expected behaviors apply (example: how to approach a pedestrian crossing and when to stop)
OICA suggests to not make this part of the road test but consider this for the „Audit“ pillar

12. Examples for a Checklist – Highway driving (1/2)
Brief description of test route/location
Date/time of test drive
Item #
Situation
Pass
Comments (must be filled out in case of “no/unclear”)
Yes
No/
unclear
Part A: mandatory
All lines in Part A have to be evaluated during the test drive
HA.1
Entering the highway
HA.2
Following other vehicle in same lane
HA.3
Passing a slower vehicle: lane change/Passing/merging back in previous lane
HA.4
Adapting to changing speed limits
HA.5
Merging from an ending lane
HA.6
Exiting the highway
HA.7
HA.8
HA.9
HA.10

13. Examples for a Checklist – Highway driving (2/2)
Part B: supplementary
If any of the following situations is encountered during the test drive this shall be noted in the respective line.
Additional lines may be added for situations not listed which were observed.
HB.1
Situation involving an emergency vehicle (police, ambulance, fire brigade)
HB.2
Policeman or roadman directing traffic
HB.3
Objects/obstacles on the road (e.g. lost cargo)
HB.4
Driving through construction site (if possible with modified lane markings)
HB.5
Driving through area with no/bad lane markings
HB.6
Safely approaching end of traffic jam
HB.7
Driving in traffic jam
HB.8
Driving through area with bad road surface conditions
HB.9
HB.10

14. Examples for a Checklist – Urban Driving (1/2)
Brief description of test route/location
Date/time of test drive
Item #
Situation
Pass
Comments (must be filled out in case of “no/unclear”)
Yes
No/
unclear
Part A: mandatory
All lines in Part A have to be evaluated during the test drive
UA.1
Wake/initial start of journey (with objects in close-proximity of the vehicle)
UA.2
Pass intersection regulated by traffic light
UA.3
Pass intersection regulated by signs
UA.4
Pass intersection without explicit regulation concerning right of way
UA.5
Merge lane (two flows of traffic become one)
UA.6
Make a left turn from a priority road (in case of right hand traffic)
UA.7
Make a turn which requires previous lane change
UA.8
Make a turn which crosses a bicycle path / pedestrian walkway
UA.9
Pass a roundabout
UA.10
Pass a pedestrian walkway (with pedestrian present)
UA.11
Park vehicle at destination
UA.12
Adherence to speed limits
UA.13
Adherence to stop sign
UA.14
Adherence to other road signs

15. Examples for a Checklist – Urban Driving (1/2)
Part B: supplementary
If any of the following situations is encountered during the test drive this shall be noted in the respective line.
Additional lines may be added for situations not listed which were observed.
UB.1
Situation involving an emergency vehicle (police, ambulance, fire brigade)
UB.2
Policeman or roadman directing traffic
UB.3
Objects/obstacles on the road (e.g. lost cargo)
UB.4
UB.5
UB.6
UB.7
UB.8
UB.9
UB.10

16. How could the road test look like from a procedural and time perspective?
Hypothesis:
The road test should be aligned with the existing driving test in terms of duration, acceptance and general conditions.

17. How could the road test look like from a procedural and timing perspective?
Process:
Duration per “use case”: Minutes in a realistic traffic environement, i.e. not in the middle of the night or during rush hour.
The assessor identifies the route to be taken and programs the route for the use case to be tested in to the navigation system.
During the road test the scenarios are being checked (not necessarily in the listed sequence) and assessed. This can include the HMI related questions in case certain limitations of the system have been declared by the OEM.
At the end an overall assessment is provided (successful: yes / no) and potentially additional comments created and recorded.

18. Other Aspects raised at the Den Haag Meeting
Country specific software / applications or application changes at crossing borders  should be considered as part of the audit process
Self-declaration on compliance with traffic codes (was seen as a big point to be discussed in WP29)  adherence to traffic codes should be considered as part of the audit process
"safe driving" and adherence to speed limits  Can be assessed via a scheme developed from the one currently used for human drivers (see back-up)
Driving cultures and environments (e.g. night time in Finland)  driving culture has many times nothing to do with traffic rules; Country specific aspects to be assessed as part of the audit process
When is maturity good enough for a real life test?  the OEM will have conducted extensive testing upfront that can be used as an initial validation of the robustness of the system to be approved