1. OpenADE – ESPI Testing and Certification Considerations Phil Beecher phil@beecher.co.uk

2. Testing Methods - Conformance Conformance Testing – Testing that a product is conformant to a specification – Implementation Under Test (IUT) is exercised by a Test Harness to verify its behavior to the specification – The Test Harness can behave in a way conformant to the specification, – It may also behave in a non-conformant way, allowing negative test cases to be executed.

3. Testing Methods - Interoperability Interoperability Testing – Testing that a product will interoperate with one or more other implementations. – Implementation Under Test (IUT) is tested together with another (conformant) implementation to verify their combined behavior is interoperable. – Interoperability testing is ideally against a “Golden Unit”, i.e. an implementation which has been tested to be conformant to the specification. – The Golden Unit does not behave in a non- conformant way, as it is a conformant implementation, so negative testing is not possible.

4. Test for Compliance Compliance Testing – May comprise Conformance Testing, Interoperability Testing or both. – Phil’s preferred method is both Test for conformance using a test harness which can provide good coverage of both positive and negative test cases. Test for interoperability using a known mature, conformant implementation.

5. Compliance Testing Steps for Conformance Testing – Develop a pro-forma PICS – Develop a test plan – Develop test harness – who and how? – Verify test harness – Decide if test lab or labs are necessary Assess lab capability and appoint lab(s).

6. Options to Consider (1) OpenADE develops Test Plan Who develops (and maintains) Test Harness? – (ZigBee Alliance – each test lab develops its own test harness) – Wi-Fi Alliance – lead lab (Wi-Fi staff) develop test harness and supply to labs. Do we use Conformance testing, Interop testing or both to assess product? – ZigBee uses a bit of a hybrid – Wi-Fi also uses hybrid – Bluetooth uses both: conformance followed by Interop with popular shipping devices

7. Options to Consider (2) How many test labs and do we restrict the number? – ZigBee has 3 in 2 countries – Wi-Fi has 15 in 8 countries – UCAIug 61850 appoints any who meet criteria – Is there sufficient business for more than 1? – Do we need any, or can we use an accredited test harness, with self-test? Who is the certification body? – Does UCA do it? – Does ISO 65 accredited body (e.g. NEMA) do it?

8. Things to decide Number of labs? Do we need a conformance test harness and interop test harnesses (Golden platforms)? How do we decide who develops test harness? How do we ensure availability of test tools / low bar for entry?