New Assessment & Test Methods VMAD-02-09 New Assessment & Test Methods Clarification of (new) tasks assigned to VMAD & way forward

Framework Document on AVs The “Framework Document on Automated/Autonomous Vehicles” (WP29-177- 19) introduced the work priorities for a new Informal Working Group named “New assessment / Test methods” According to the Document, the group responsibilities are: Assessment of Functional Safety Audit Simulation Electronic System Compliance Test Track Real World Driving Evaluation Digital Identity (not defined) MULTI-PILLAR APPROACH as presented by OICA (WP.29/2019/2)

Framework Document on AVs The “Framework Document on Automated/Autonomous Vehicles” (WP29-177- 19) introduced the work priorities for a new Informal Working Group named “New assessment / Test methods” Group deadlines/deliverables: [March 2020] – NOT ENDORSED General Deliverables Specific Deliverables Review of the existing and upcoming methods and a proposed way forward for the assessment for Automated / Autonomous Driving The test and assessment method for Lane Keeping Systems of SAE levels 3/4 as New UN Regulation for Contracting Parties to the 1958 Agreement

Industry Proposal (Assessment) A group led by the United Kingdom already developed amendments to the “CEL Annex” intended for ALKS with support from Germany, France, The Netherlands, Japan and OICA/CLEPA The amendments proposed by the UK at GRVA 2nd session (GRVA/2019/4, as amended by GRVA-02-19), are deemed by industry as sufficient for the new UN Regulation on ALKS If a Contracting Party to the 1958 Agreement would like to develop it further, given such short-term deadlines, Industry believes the most efficient way would be to rely on the existing UK Technical Group

Industry Proposal (Test Methods) The same rationale applies to the development of test methods for ALKS, that should be dealt within the existing ACSF Informal Working Group A complete new group would need some time to learn the detailed technical background of the ALKS requirements (not yet finalised), and the required “learning curve” would delay progress Until GRVA organization is adopted and implemented, industry believes the “New assessment / Test methods” group should focus on the new certification concepts (e.g. the multi-pillars approach) These proposals are guided by the willingness to secure the delivery of the ALKS new regulation in due time

Way Forward for the VMAD Informal Working Group

+ The analysis should help to derive dedicated tasks for VMAD Discussion Paper Can Item x,y,z be reliably assessed through a “Classical Certification Approach”? Safety Principles (Framework Document) Yes Classical Certification Tests Methods No Partially Transition Phase (e.g. ALKS) Conventional Track Testing + CEL Step 2 IWG ACSF CEL UK Increasing Level of Complexity Higher Automation New Assessment & Test Methods Multi-Pillar Approach (Simulations; Audit; Track Testing; Real World Driving) VMAD The analysis should help to derive dedicated tasks for VMAD

Conclusions New Assessment & Test Methods (NATM) is a priority because conventional compliance test methods cannot capture the full range of automated driving challenges NATM does not replace conventional testing; instead it complements testing such as done under classical type approval NATM seeks methods that can be applied under the 1958 and 1998 Agreements VMAD’s role should be to submit Technical Proposals for New Assessment & Test Methods to GRVA for further consideration

Backup Background Information

Draft Agenda GRVA-03 (03-04 June 2019) For the special GRVA session in June the following input connected to VMAD is expected according to the draft agenda: 5. Automated Lane Keeping System (ALKS) (a) … (b) New assessment / test method GRVA may wish to receive information on the work done by the Validation Method for Automated Driving (VMAD) group that may be relevant specifically for ALKS. (c) …. (d) Functional safety, Complex Electronic (CEL) control system GRVA agreed at its second session on the need to develop functional safety and Complex Electronic (CEL) control system provisions suitable for ALKS. GRVA may wish to review any proposal available on this matter, if available. Documentation: (ECE/TRANS/WP.29/GRVA/2019/4 Informal document GRVA-02-19)

Overview of Amendments of “CEL Annex” for ALKS Outcome of UK ad-hoc group. Ref: ECE/TRANS/WP.29/ GRVA/2019/4, as amended by GRVA-02-19 Description and verification of the “control strategies”, which ensure “robust and safe operation of the function(s) of "The System" in response to a specific set of ambient and/or operating conditions…”). Declaration of enabled and disabled safety related functions when the hardware and software are present in the vehicle at the time of production. Functional safety, misuse and unauthorized modifications “The Technical Service shall perform an assessment of the application of the analytical approach(es). The assessment shall include: (a) Inspection of the safety approach at the concept (vehicle) level with confirmation that it includes consideration of: interactions with other vehicle systems; Malfunctions of the electronic control system […]; Situations when a system free from faults may create safety critical risks (e.g. due to a lack of or wrong comprehension of the vehicle environment); Potential misuse or simple unauthorised modification of the system. This approach shall be based on a Hazard / Risk analysis […] (b) Inspection of the safety approach at the system level (FMEA, fault tree) […] (c) Inspection of the validation plans and results […]”