1. J. Alan Atherton Michael Goodrich Brigham Young University Department of Computer Science April 9, 2009 Funded in part by Idaho National Laboratory And Army Research Laboratory 1

2.  Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 2

3.  What is a remote manipulator?  Applications USAR EOD Planetary Exploration 3

4.  Remotely operating a robot is difficult “Soda straw” — Maintaining situation awareness Time delay Mental workload  Why is this a problem? Collisions Slow Stressful Foster-Miller Talon 4

5. All images adopted from Yanco, H. A.; Drury, J. L. & Scholtz, J. “Beyond usability evaluation: analysis of human- robot interaction at a major robotics competition” Hum.-Comput. Interact., L. Erlbaum Associates Inc., 2004, 19, 117-149 5

6.  Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 6

7. Idaho National LaboratoryUMass Lowell Bruemmer, D. J. et al. “Shared understanding for collaborative control.” IEEE Transactions on Systems, Man and Cybernetics, Part A, 2005, 35, 494-504 Yanco, H. A. et al. “Analysis of Human-Robot Interaction for Urban Search and Rescue.” Proceedings of the IEEE International Workshop on Safety, Security and Rescue Robotics, 2006 7

8. INL / BYU AV Interface Ferland et al. - Sherbrooke C. W. Nielsen, M. A. Goodrich, and B. Ricks. “Ecological Interfaces for Improving Mobile Robot Teleoperation.” IEEE Transactions on Robotics and Automation. Vol 23, No 5, pp. 927-941, October 2007. Ferland, F.; Pomerleau, F.; Dinh, C. T. L. & Michaud, F. “Egocentric and exocentric teleoperation interface using real-time, 3D video projection.” Proceedings of the 4th ACM/IEEE international conference on Human robot interaction, ACM, 2009, 37-44 8

9. NASA Viz Nguyen, L. A.; Bualat, M.; Edwards, L. J.; Flueckiger, L.; Neveu, C.; Schwehr, K...; Wagner, M. D. & Zbinden, E. “Virtual Reality Interfaces for visualization and control of remote vehicles” Autonomous Robots, 2001, 11, 59-68 9 Kelly, A.; Anderson, D.; Capstick, E.; Herman, H. & Rander, P. “Photogeometric Sensing for Mobile Robot Control and Visualisation Tasks” Proceedings of the AISB Symposium on New Frontiers in Human-Robot Interaction, 2009 CMU Robotics Institute

10.  Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 10

11.  Requirements Ecological Increase situation awareness Manage workload  Existing Interfaces Lack depth information No manipulation support Not designed for real-time operation 11

12. 12  Real-time remote manipulation

13. 13

14.  Robot Build from kit, modify Player driver Motion planning for arm Swiss Ranger driver  Communication Integrate with INL’s system Network data transfer  User Interface OpenGL display Experiment automation 14 Robot Controller User Interface

15.  Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 15

16. Variant 1Variant 2Variant 3 Variant 4Variant 5Variant 6 3D + Video End Effector Video3D Joint Robot Control Visualization  Task: collect yellow blocks  30 participants  Between-subject comparison 16

17.  Reduce memorization effects  Minimize damage to arm  Quick change 17

18. 18

19. 19

20.  Joint control  View-dependent end effector control 20

21. 21  Robot reaches for point  User moves point with joystick  Point movement depends on view orientation

22. 3D + Vid. End eff. 3D + Vid. Joint 3D End eff. 3D Joint Video End eff. Video Joint 22

23. 3D + Vid. End eff. 3D + Vid. Joint 3D End eff. 3D Joint Video End eff. Video Joint 3D + Vid. End eff. 3D + Vid. Joint 3D End eff. 3D Joint Video End eff. Video Joint 23 Collisions with posts, box, table Collisions with block in final adjustments

24. 24

25.  Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 25

26.  Problems Alignment Time lag Cluttered 3D scan model  Changes Stereo camera exterior orientation Interactive robot arm calibration Simple Quickening Scan Pruning 26

27.  Interactive stereo camera calibration  Live robot arm calibration 27

28. 28

29. 29

30. 30

31.  Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 31

32.  3D visualization supports SA  Video is faster  3D + video is a good tradeoff  3D + video might reduce workload 32

33. 33  Head tracking  Ecological camera video  Haptics