2. Technologies for Mobile Manufacturing Sanjiv Singh/Reid Simmons Robotics Institute Carnegie Mellon University February 2008

3. Outline  Motivation  Objectives  A Simple Example Autonomous Assembly Tele-operated Operation “Sliding” Autonomy  More complicated examples  Key Technologies Relative Position Estimation Coordinated Control of Mobile Manipulation Task Control Architecture

4. Terrestrial Construction Terrestrial Construction Many different tasks Complimentary entities Big plan that is constantly refined

5. Objectives  Enable heterogeneous multiple robots to coordinate complex assembly tasks Emphasis on tasks that can not be done by single robots  Enable flexible human-robot interaction during assembly Deal with unanticipated contingencies Reduce need for operator Candidate Tasks:  Assemble multi-element, compliant structure  Brace structure for strength  Cable structure

6. Previous Work: Distributed Architectures Executive Behavioral Control Planning Executive Behavioral Control Executive Behavioral Control Planning

7. Previous Work: Multi-Robot Synchronization  Enable agents to allocate and synchronize tasks; detect and handle each others exceptions Robot 1 Robot 2 Robot 3 Execute Sequentially Execute Concurrently Execute Sequentially Task A Task C Task B

8. Coordinated Assembly  Three heterogeneous robots  Crane has large workspace, high weight capacity  Manipulator has fine control  Roving eye provides high degree of resolution  Independent robot operation without accurate inter-robot calibration

9. Multi-Robot Testbed

10. A Simple Example: Fully Autonomous Dock a single beam into two upright connectors with mm tolerance

11. Combined State-Machine for Dual-End Dock Lower beam Align beam over far stanchion Turn far end into view Align beam over near stanchion Lower beam into far stanchion Watch crane Watch MM Move to far stanchion Watch dock Watch MM Watch crane Watch push Move away From MM Align MM at near stanchion Dock beam in near stanchion Grasp beam Dock beam Turn 180˚ Align MM at far stanchion Push beam over far stanchion Contact beam Push beam Stow arm CRANE MOBILE MANIPULATOR ROVING EYE *Sequential connections for watch tasks not shown for clarity.

12. First Results

13. Failures u First dock succeeds 70% of the time u Complete second dock succeed only 6% of the time u 20% of the time, initial conditions are not set u Common errors: n Mobile Manipulator might over or underturn n Beam gets caught on groove n Arm gets caught on beam n Fiducials are blocked n Actuator deadband causes infinite loop n Visual servoing fails n Crane actuator slip causes offset error

14. Mature Autonomous System  Setup Completely autonomous 50 trials  Typical Failures Electrical failure on MM Software crash Near collision due to errors in visual perception Obscured fiducial MM lost grip on beam Assembled portion broke apart  Speed µ=9.9min,  =1.6min

15. Teleoperated System  Setup 50 trials (total) with four robot- experienced users Several robot-specific GUIs Teleoperation using visual feedback from Roving Eye  Typical Failures Visual feedback created “tunnel” vision Stereo vision did not provide users with good depth perception Experienced one network and one electrical failure  Speed µ=12.5min,  =4.0min

16. Sliding Autonomy: Adding an Operator  Fully autonomous operation has many failure modes  Not enough bang for the buck to automate some operations  Would like seamless method to switch between operator and system  Operator should be able to take over either control or monitoring of task  Three modes of human interaction: Pre-assigned task Intervention Exception Handling

17. Sliding Tasks  Mobile Manipulator First dock Turn Second dock*  Roving Eye Visual search Turn  Crane Second dock*

18. Results w. Sliding Autonomy  Setup Several task-specific GUIs Limited adjustable tasks Feedback available from any autonomous task 50 trials  Results Discretionary-Intervention Successes Mandatory-Intervention Successes Failures due to damaged hardware & network failure Successes Completion Times MeanStd-dev Fully Autonomous64%9.9m1.6m Tele-operated96%12.5m4.0m Sliding Autonomy94%9.9m1.9m Discretionary Only68%9.5m1.5m Mandatory Only26%11.1m2.6m

19. Outline  Motivation  Objectives  A Simple Example Autonomous Assembly Tele-operated Operation “Sliding” Autonomy  More complicated examples  Key Technologies Relative Position Estimation Coordinated Control of Mobile Manipulation Task Control Architecture

20. More complicated example #1

21. Modes  Teleoperated: User controls each robot in turn through keyboard and mouse  Autonomous: Hit start and step back  System Initiative: System asks for help when needed  Mixed Initiative: System initiative + Operator intervention Completion TimeSuccess RateTLX Workload Teleop Autonomous System Initiative Mixed Initiative

22. Modes  Teleoperated: User controls each robot in turn through keyboard and mouse  Autonomous: Hit start and step back  System Initiative: System asks for help when needed  Mixed Initiative: System initiative + Operator intervention Completion Time [Std] Success Rate (Total Exp) TLX Workload [Std] Teleop732 [227] sec94% (16)52 [16] Autonomous516 [125] sec89% (35)0 System Initiative500 [182] sec100% (16)27 [21] Mixed Initiative529 [148] sec94% (16)29 [13]

23. Tele-Op Mixed System Initiative Autonomous 516[125] 89% (35) 0 500[182] 100% (16) 27 [21] 529 [148] 94% (16) 29 [13] 732 [227] 94% (16) 52 [16]

24. More complicated example # 2 Extended scenario involves planning because constraints make it difficult to script a plan Recovery from failure might require many steps

25. University of Maryland Space Systems Lab Neutral Buoyancy Tank EASE Truss Assembly Ranger Space Shuttle Cargo Bay

26. “Roving Eye” “Crane” “Mobile Manipulator” Trestle - U Maryland SSL Cooperation Operator at CMU Ranger Arms at U Maryland internet

27. Outline  Motivation  Objectives  A Simple Example Autonomous Assembly Tele-operated Operation “Sliding” Autonomy  More complicated examples  Key Technologies Relative Position Estimation Coordinated Control of Mobile Manipulation Task Control Architecture

28. Sensing Location  Need to localize parts wrt to robot during operation so robot can plan motion and adapt to any variations in starting conditions & performance  Complicated because the robot base is not stationary  Method 1: No global reference frame. Relative position (between parts & between robot and part) is determined via fiducials Advantages: flexible, low infrastructure Disadvantages: accuracy can be low unless high fiducials are sensed with high resolution, computationally expensive  Method 2: Establish global reference frame. Parts and Robots are all registered in common frame. Advantages: high accuracy, low computation requirements Disadvantages: high infrastructure costs, must guarantee line of sight from fixed infrastructure

29. Sensing Relative Position  Visual Fiducial allows determination of ID & 6 DOF displacement between camera and fiducial.  Fiducials have some redundancy, can work even if the fiducial is partly obscured.  Main computational expense is in detecting fiducial in the scene.  Accuracy increases as camera gets closer to fiducial

30. Tracking Fiducials (with occlusion)

31. Other kinds of Fiducials  Active Fiducials can be used.

32. Sensing in a Global Reference Frame  Transmitters fixed to infrastructure  Receivers on items that move  Requires synchronization between receivers http://www.indoorgps.com/

33. Mobile Manipulation  Want to place the robot end effector accurately in a large workspace. Could do this by coupling manipulator & mobile base.  Coordination of base and arm motions of Mobile manipulators is complicated because of redundant degrees of freedom degrees of freedom.  Further considerations: Want to keep the arm from getting close to singularities. Want to control end-effector but want to ensure that the base meets the constraints. Arm and base have very different response

34. Mobile Manipulation Resolved motion control with Arm motion only-- SMALLER WORKSPACE Resolved motion control with Coordinated Arm and Base-- LARGER WORKSPACE

35. Implementation on CMU MM

36. Projecting into the Null Space (Example1)

37. Projecting into the Null Space (Example2)

40. Offline Planning to decouple Base & Arm Motion Each grid cell gets a score based on how much of the path and how well the it can be covered with the base at that point.

41. Seam Following Motion sped up by 4x