1. COEN/ELEC 390 1

2. I Performance Testing slides 1-12 II User Documentation slides 13 - end 2

3.  For Performance Testing ◦ Provide empirical evidence that system behaves in line with requirements, and does so.. ◦ Under all operating conditions, and.. ◦ Repeatedly (e.g. for N times).  What would you call a system that: A.Satisfies only part of the requirements? (incomplete) B.Functions only under a strict subset of operating conditions (not robust) C.Does A and B, a few times, before requiring maintenance (unreliable) In fact, there are other properties that can be tested, such as usability and extendibility 3

4. 4 Unreliable Product Great Looks, good portability but totally unreliable. It broke down in less then 2 months. I wish they had made it more durable.

5.  What matters- in your case: ◦ Effective: in getting from A to B - within given constraints ◦ Efficient: in terms of cost of parts used - though labour is the biggest cost  Other important concerns: ◦ Robust : so it does not fail on unexpected floors ◦ Reliable: so it is likely to perform on the “big day” Ya, sure, but how? 5

6. 6

7.  Testing Effectiveness ◦ Let the robot try to execute the task N times ◦ Count the number of times it succeeds ◦ Record a trace of the successful attempts ◦ Record a trace of the unsuccessful attempts ◦ Present an analysis of the successful attempts:  Sum image + execution time ◦ Present an analysis of the failed attempts:  Failure causes + improvement recommendations  Testing Efficiency ◦ How much it cost and  How much of the parts were actually used; also  How much could it have cost (e.g., w/ better design) 7

8. 8

9.  Remember the Robustness and Reliability issues? well, they can be tested, and this is done via: 1.By diversifying the conditions of testing 2.Repeating the tests a large number of times So, you decide what you want to measure, how and under what conditions, and for how many times, then collect and analyze data, to present conclusions and recommendations. 9

10. 10 How to analyse and present this “raw” data?

11.  This figure shows passengers on the Titanic, male and female, by class of ticket, and by survival, 11

12.  Again, given a design, you decide:  what you want to measure,  how you want to measure it,  under what conditions,  for how many times, then  collect raw data, and  analyze & present it, in order to  present conclusions & recommendations. 12

13. 13

14.  Because users (as my son says) are not you,  you need to direct them on: ◦ preparation ◦ use of existing functionalities ◦ extension of functionalities ◦ trouble-shooting ◦ maintenance 14

15. Products require: ◦ a list of precautions as to the handling of the robot or/and its/other parts ◦ a list of parts: how they appear and their names ◦ Instructions on how to assemble the robot (if necessary), utilizing clear and concise text and diagrams ◦ Configuration: what need to be connected (wires), removed (covers), installed (batteries) and so on ◦ Summary tables of operating conditions and technical specifications ◦ Instructions of how to switch on the robot (safely) 15

16. To use the robot, the user needs:  an introductory section on how the robot can be used (e.g. locate & seek object, follow line): its functionalities  Illustrated step-by-step tutorials on how to use every functionality, including: ◦ Switching on/off procedures ◦ Situation-specific adjustments ◦ Final checkup procedure ◦ Deployment procedure ◦ How to recover from minor operating errors 16

17.  Suggest how the robot’s functionalities can be extended, mainly through: A.More complex Decision Making: Different and larger programs (on larger chips) B.More Information/Abilities: Addition of more sensors and/or actuators C.Better Information/Abilities: Using different components (with same interface)  Instruct users on how to use a different chip and how to program chips; how to add and replace components with others from an approved list (and what adjustments are needed in every case)  Think! To enable such extendibility, what do you need to ensure about your current design? 17

18.  What have/could gone/go wrong and how to deal with it  Examples include: ◦ E1 Robot fails to move  Battery not installed or low: install a fresh battery  Robot not switched on: switch it on  Wire connecting A to B loose: re- connect it ◦ E2 Unusual smell coming from robot  Too much loading: lighten load 18

19.  What to do keep it operational? 1. How to keep it clean and usable (e.g. wiping, changing tires) 2. How to store it safely, in an operation-ready condition (with all necessary materials for operation) 3. How to test and replace power source (e.g. batteries) 19

20. Users expect plug-and-play products, but failing that, you must instruct the users using illustrated step-by-step clear and concise manuals on how to:  Prepare the robot for use  Use it, in a given environment for a given task  Extend its functionality, using H/W and S/W  Trouble-shoot minor problems  Maintain it in good working order 20