Steps Toward Developing an Intelligent Robotics Course Kutztown University PACISE 2011 April 9, 2011 Oskars J. Rieksts Jeffrey W. Minton

2011Kutztown University2 Acknowledgments – Funding CS Department & LAS College CS Department & LAS College TYCO Electronics Foundation TYCO Electronics Foundation Kutztown University Foundation Kutztown University Foundation KU Faculty Professional Development Committee KU Faculty Professional Development Committee KU Undergraduate Research Committee KU Undergraduate Research Committee PASSHE Faculty Professional Development Council PASSHE Faculty Professional Development Council

2011Kutztown University3 Motivations Rodney Brooks Rodney Brooks DARPA Grand Challenge DARPA Grand Challenge Auburn Auburn Hardware-software synthesis Hardware-software synthesis  “Hardened” hardware  Central role of software

2011Kutztown University4 Motivations Applied A.I. Applied A.I. Applied cognition Applied cognition Mobility Mobility Assistive robotics Assistive robotics  Robotic wheelchairs

2011Kutztown University5 Outline Early efforts Early efforts  Hacking Roomba & other ventures  Research Experience for Undergraduates  Create and Mindstorms  Captain KURK  Trinity firefighting contest  Myro and Python  Reading-Berks Science Fair

2011Kutztown University6 Coming into Focus Emerging goals Emerging goals  Intelligence  Vision  Communcation  Philosophical issues

2011Kutztown University7 Point of convergence Course in intelligent robotics Course in intelligent robotics  Control  Vision  Communcation  Investigate cognitive issues

2011Kutztown University8 Course design objectives Robot is Robot is  Situated  Embodied Shared environment Shared environment  Human  Machine Shared communication Shared communication Framework for investigation Framework for investigation

2011Kutztown University9 Robot is situated Operates within an environment Operates within an environment  Embedded in the world  Chief knowledge source:  Data stream drawn from environment  “The world is its own best representation” – Brooks

2011Kutztown University10 Robot is embodied Entity 1 st, agent 2nd Entity 1 st, agent 2nd  Self reliant  Secretary of State model Self-seated conceptual framework Self-seated conceptual framework  World concepts grounded in sensor suite  Behavior set emanates from actuator suite  Decision apparatus grounded in sensor/actuator suites

2011Kutztown University11 Robot is embodied Eames: Design is a plan for arranging elements to accomplish a particular purpose Eames: Design is a plan for arranging elements to accomplish a particular purpose Elephants don’t play chess (Brooks) Elephants don’t play chess (Brooks)  “Mind” is to fit the body Design on need to basis Design on need to basis  Know  Think  Do

2011Kutztown University12 Shared environment Shared conceptual framework Shared conceptual framework  Robot’s  Derivative of human’s  Simplified in structure Shared factbase Shared factbase  Robot’s  Subset of human’s  Simplified ontology Vision based Vision based

2011Kutztown University13 Common communication framework Grounded in Grounded in  Shared  Environment  Conceptual framework  Ontology  Vision as main sensor Intersecting language constructs Intersecting language constructs

2011Kutztown University14 Projected course topics/activities Specialized robot control software Specialized robot control software Robot control architecture Robot control architecture Basics of image processing Basics of image processing Specialized image processing software Specialized image processing software Basics of communication theory Basics of communication theory Key issues of cognitive robotics Key issues of cognitive robotics

2011Kutztown University15 Slow, steady progress Browning: Browning:  Reach should exceed grasp Rapid prototyping Rapid prototyping Iterative development Iterative development

2011Kutztown University16 Structure of the class Four teams Four teams Roles Roles  Team leader  Designer  Coder  Document guru  Historian  Test designer  Test administrator  Hardware specialist

2011Kutztown University17 PACT demonstration Pennsylvania Association of Council of Trustees Pennsylvania Association of Council of Trustees Very early in learning curve Very early in learning curve Navigation within a “corral” Navigation within a “corral” Well received Well received  Gateway navigation  Bull fighting robot PR for PASSHE PR for PASSHE

2011Kutztown University18 Jeff – gateway navigation

2011Kutztown University19 Investigate cognitive issues Sight and touch Sight and touch  George Stratton  Spatial harmony of sight & touch Space Space  Benjamin Kuipers  Semantic spatial hierarchy  Metrical  Topological  Hybrid

2011Kutztown University20 Investigate cognitive issues Language and meaning Language and meaning  Stevan Harnad  Symbol grounding problem Language and shared space Language and shared space  Amichai Kronfeld  Shared referent problem

2011 Kutztown University 21 Investigate cognitive issues Embodied cognition Embodied cognition  Randall Beer  Situated, embodied, dynamical Principle of interactivism Principle of interactivism  Mark Bickhard  Emergence of representational content

2011 Kutztown University 22 Illustration – Shared Referent Problem From Kronfeld From Kronfeld  Steve & wife Bev are at a party  Steve: Bob & his wife are certainly enjoying the party  Bev: Sally is not his wife!  Note  Steve & Bev share a referent, Sally  Despite misidentification no communication problems arise!

2011Kutztown University23 The KLO Triad For communication involving referents For communication involving referents  For each communicant – this triad  Object (the referent)  KB – representation of object  Language – reference to object  For successful communication  Referents must match  For HMC (Human-machine communication)  Human & machine KBs do not match

2011Kutztown University24 Establishing co-refrence Per Frege’s distinction Per Frege’s distinction Extensional approach Extensional approach  Pointing or equivalent Intensional approach Intensional approach  Language alone HMC goal HMC goal  Minimize extensional  Maximize intensional

2011Kutztown University25 The Trinity Robot

2011Kutztown University26 The Trinity Robot VEX robotics kit for chasis VEX robotics kit for chasis VEX included motors for locomotion VEX included motors for locomotion Sonar rangefinders Sonar rangefinders Web-cam Web-cam Arduino to control motors and sensors Arduino to control motors and sensors Netbook to control Arduino and process images Netbook to control Arduino and process images

2011Kutztown University27 The Trinity Robot Inaccurate motor control Inaccurate motor control Sonar signals bounce inside corners Sonar signals bounce inside corners  Provide inaccurate measurements Viewing angle of web-cam too small Viewing angle of web-cam too small

2011Kutztown University28 Brobi

2011Kutztown University29 Brobi – hardware Create by iRobot Create by iRobot Platform built onto Create cargo bay to accommodate equipment Platform built onto Create cargo bay to accommodate equipment Web-cam with increased viewing angle Web-cam with increased viewing angle IR rangefinder IR rangefinder  IR light does not bounce like sound Arduino Arduino

2011Kutztown University30 Brobi – software OpenCV and Python OpenCV and Python Consultation – John Spletzer Consultation – John Spletzer  Lehigh  Little Ben in Urban Challenge MATLAB MATLAB  Image processing  Create API  A programming language

2011Kutztown University31 Image processing “A picture is worth a thousand words” “A picture is worth a thousand words” need to extract discrete objects from images to identify them need to extract discrete objects from images to identify them K-Means clustering K-Means clustering

2011Kutztown University32 Goal: identify green ball

2011Kutztown University33 K-Means Clustering Cluster sets of data Cluster sets of data  Into user-defined number of segments  Number of segments referred to as K  Clusters defined by MEAN of all values in cluster

K-Means Algorithm

K-Means Example K-Means using k = 6 K-Means using k = 5

2011Kutztown University36 Natural Language Processing “Go to the green ball” “Go to the green ball” The meanings behind words must be inferred The meanings behind words must be inferred “Go,” conceptually can represent many things “Go,” conceptually can represent many things  Take a turn in a game  The Chinese strategy game  Travel to a location Determine the concept being referred to Determine the concept being referred to Conceptual parsing Conceptual parsing

2011Kutztown University37 Conceptual Parsing Words are mapped to concepts Words are mapped to concepts Concepts Concepts  Rules define set of related concepts  One concept may have many separate rule sets

Concept Tree

2011Kutztown University39 Assessment – Platform Best hardware platform to date Best hardware platform to date Opens up many avenues of course development Opens up many avenues of course development  Software for robot control systems  Robot control architectures  Image processing  Communication  Cognitive robotics issues

2011Kutztown University40 Assessment – MATLAB Can do all 3 things Can do all 3 things  Sensor/actuator interface with Lehigh API  Image acquisition & processing  Robot control system programming Good IDE Good IDE Good documentation Good documentation  17 pdf files; 64 mB  Online documentation

2011Kutztown University41 Assessment – MATLAB Good IDE Good IDE Strong user community Strong user community  Blogs  Discussion boards  Good code segments Good programming language Good programming language  Matrix optimized  Advance features, e.g., lambda, apply

2011Kutztown University42 Assessment – Create Useful Useful Stable Stable Ubiquitous Ubiquitous Negatives Negatives  Battery short lasting  Does not travel in straight line

2011Kutztown University43 Assessment – Accessibility Cost not prohibitive Cost not prohibitive  Create – $130  $220 with battery and charger  Netbook ~ $330  Arduino ~ $30  MATLAB  $900 for 10 seat license  $130 for student version with image acquisition

2011Kutztown University44 Future directions Software repository Software repository Learning Learning  Bayesian  Genetic algorithms  Neural networks, etc. Control architecures Control architecures  Behavior-based  Hybrid  Blackboard?

2011Kutztown University45 Future directions Experiment with other sensors Experiment with other sensors  Touch {whiskers}  Odor  Sound {whistle, a la Sound of Music}  Heat  Odor  Light/brightness Wireless interface Wireless interface  “Watson, come here. I need you”

2011Kutztown University46 Future of robotics at KU Strong student interest Strong student interest Rejuvenated course Rejuvenated course  New syllabus  Approved  400 level – a mixed blessing  Not yet scheduled  In rotation?  Politics

2011Kutztown University47 Further Information Jeff’s graduate thesis Jeff’s graduate thesis  Do You See What I'm Saying: Relating Language and Vision to Create Interaction Between Humans and Robots  Delves further into concepts discussed here

2011Kutztown University48 Questions?

2011Kutztown University49 The End

2011Kutztown University50 Extra slides

2011Kutztown University51 Template Font 44 Font 44  font 40 Font 44 Font 44  font 40 Font 44 Font 44  font 40 Font 44 Font 44  font 40

2011Kutztown University52 Assessment Sensors can be added Sensors can be added Offers new areas of exploration Offers new areas of exploration  Heat sensors  Sound sensors  Odor sensors

2011Kutztown University53 Unsuccessful segmentation

2011Kutztown University54 Successful segmentation

2011Kutztown University55 Template Font 44 Font 44  font 40 Font 44 Font 44  font 40 Font 44 Font 44  font 40 Font 44 Font 44  font 40