Discussions for Testing Group Name: WG-6 Source: Yoshinori Goto, NTT, Shingo Fujimoto, FUJITSU, Nick Yamasaki, KDDI, Meeting Date:

1. Introduction On the Test-WG ML, there was a discussion on the testing. An idea was proposed that M2M devices (ASN, ADN etc) are subject for conformance testing. Comments were raised; – General overview of testing should be clarified. (currently missing) – A variety of implementation for M2M devices (ASN, ADN etc.) is possible and this impacts to the testing methodology. The testing of M2M devices is not like the testing of mobile handset (3GPP). – Assuming that the test is carried out by a test lab, attention to the difficulties of testing at a test lab should be paid. © 2014 oneM2M Partners 2

2. Object(Product) under Test Product based testing – Actual products Car Meter Consumer Electronics product etc. Module based testing – Hardware module. – Software module. – Hardware and software module. © 2014 oneM2M Partners 3u

2.1 Product test Product – Subject is actual devices/products (like car, meter, CE product etc.) – Commercial product is subject for frequent update/model change to respond market needs. Do we need to test all the updates/model changes even if the updated part is not related with oneM2M specifications? (If so, what is the criteria for re- testing?) – Testing environment needs to ensure proper function of the product under test. (Is this always feasible?) © 2014 oneM2M Partners 4u

2. 2 Module test Module – Most of the devices/products are assumed to implement the oneM2M functionality as a hardware/software module. – But it is possible that devices/products are designed in such a way that the oneM2M functionality is not able to be separated from main functions of devices/products. – In case of hardware module, it is possible that the hardware module is combined with underlying network function (NSE). Is it feasible to prepare all the network technologies used for M2M bearing in mind variety of applicable areas of M2M. – Similarly, for software module, the test environment should ensure proper function of software module under testing. This requires deep knowledge of software technologies used in the vertical sectors (probably many differences from ICT industry can be seen). Probably, module based testing is possible for many cases, but there are some cases that are difficult for module based testing at a test lab. © 2014 oneM2M Partners 5u

3. Approach for Testing Testing at test lab(s) – Test is carried out at test lab(s) – Test lab is designated/qualified. Testing by test equipment(s) – Test is carried out by anyone who has test equipment(s). – The test equipment could be subject to qualification. Testing by manufacturer(s) using their own equipment – Test is carried out by those manufacturer(s) who wants to test their products. © 2014 oneM2M Partners 6u

3.1 Testing at Test lab(s) Need to designate/qualify test lab(s). – How to designate/qualify is a SC issue. Probably, good for conformance testing rather than interoperability. The manufacturers need to bring their products to the test lab. – Not always feasible due to logistical problems, e.g. transportation of the product, particularly big device. – Good for software/hardware module. But a whole device may be difficult for testing. The test lab needs to provide necessary environment to function the M2M device/module properly. – Not only physical environment but also underlying networks and other aspects. The test lab should be well prepared for the testing, even if the test case is very rare implementation. (Otherwise, the test lab excludes a certain type of M2M devices from testing.) The test lab is subject to periodical review for designation/qualification. – Not only for transparency, but also for anticipation for update of oneM2M specifications and market needs. © 2014 oneM2M Partners 7

3.2 Testing by test equipment(s) The testing can be carried out by anyone at anywhere. – Usually, by those manufacturers who develop products. No transportation. Flexible testing. – A M2M device difficult to transport (e.g. too big machine) is testable. – Testing can be done whenever the manufacturers want to do so. (e.g. in the middle of the process of product development) A test equipment should be publically available. © 2014 oneM2M Partners 8 M2M device under test Qualified test equipment

3.3 Testing by the manufacturer(s) This makes sense for an unimaginable use case/implementation which is difficult for testing. – Recall the scope of M2M is large. We should not exclude any use case/implementation from testing. The last resort when any alternative is not available. The quality of test result is not guaranteed, but we can expect the manufacturer’s technical knowledge, reputation and goodwill. © 2014 oneM2M Partners 9

4. Conclusion/proposal TST-WG should focus on general overview of testing. – Making a new TS/TR is a good approach for this subject. The general overview should include all the available approaches for testing – Not only test lab, but also other approaches as long as agreeable. It should be highlighted that a variety of implementations is possible for M2M devices. © 2014 oneM2M Partners 10

[Appendix] Conformance / Interoperability Conformance Testing – Is testing to determine whether a product or system complies with the requirements of a specification, contract or regulation. (from wikipedia) Interoperability testing – Is the activity of providing that end-to-end functionality between (at least) two communicating systems is as required by those systems' base standards. (from ETSI) Conformance testing is not equal to interoperability testing in terms of purpose, methodology etc. © 2014 oneM2M Partners 11u