Joe Cohen

Presentation Overview  Project definition and requirements  Solution process and explanation  Methodology  Demonstration  Development process  Final remarks

Project Definition  Mount a laptop and webcam atop a robot (iRobot Create)  Develop an application to control the Create via the laptop  Develop two separate applications: Server: Run on laptop, take user input and control Create, capture webcam video and publish to client Client: Run on separate computer, have full user interface to control the Create, allow for the viewing of video feed

Project Definition  Connect the two applications via sockets  Develop psuedo-language to communicate between the two applications  Run the two applications over the same (wireless) router

Solution Process Create Control  Develop a VB application to control the Create robot  The language built into Create takes a set of codes and parameters to execute commands  Develop user interface for client application

Solution Process Create Control OperationCode# of Bytes Parameter Meaning Drive1374Velocity (2) Radius (2) Store Song140xNumber, Length, note 1, note 1 length, etc… Play Song1411Song Number Read Sensors1421Packet Number Wait1551Time to wait in 1/10 second Start OI1280 Safe Mode1310

Solution Process Pseudo-Language  Develop a pseudo-language to communicate between server and client applications  Client application takes user input, encodes and sends to server. Server decodes and executes command.  Data/Control socket uses numerical codes

Solution Process Pseudo-Language ActionCode Drive straight forward at ## speed value1## Drive straight backward at ## speed value0## Change direction0## or 1## Stop200 or 201 Turn300, 301, 302, 310, 311, 312 Reset Robot320

Solution Process Socket Connections  Sockets used to communicate between two computer/programs  Server application establishes socket and waits for client to connect to it  Connection made by IP address and port number

Solution Process Client/Server Application  Separate current application into client and server Client: Keep user interface, instead of calling Create control functions, encode and sent command over socket Server: Decode code and execute Create commands

Solution Process Wireless Network  Allow the applications to run over a wireless network rather than the internet  Benefits: Faster transmission of commands Wireless movement of robot Wider range

Solution Process Video Capture  Use Intel’s OpenCV camera library to initialize camera and capture frames  Once frames are located in memory a bitmap file is created  Using a memory dump the bitmap is displayed in a PictureBox on the screen

Solution Process Video Feed  Once the bitmap structure is built, it is sent down a second socket to the client application  The server constantly captures and displays the video but only sends to client when it is requested

 Video demonstration

Solution Process Keyboard Control  Incorporate another control method for the user via the keyboard  Allows for basic functionality using the keyboard  Perhaps a better way to control the robot

Solution Process  Project easily broke into three parts  Develop separate solutions for each part and piece together to make final product  Robot control grouped into three functions: straight, stop, and turn

Solution Process Resources  iRobot Create Open Interface manual  Previous projects by John Moss, and Bryan Jorgensen & Adam Ziegler  Internet development forums  All CS faculty

Solution Process CS Courses & Concepts  Operating systems – COM Ports, sockets, numerical conversions  Databases – VB 6 user interface development  Event Programming – Structure and flow of windows applications

Solution Process Shortcomings  Populate COM ports on server connection form based on active COM ports  Allow for resizing of the video input window  Create respond to sensor events

Solution Process Future Extensions  Improve on shortcomings  Improve user interface and application feedback  Wireless camera and Create control  Incorporate video algorithms: motion tracking, object detection, etc.  Possibly integrate into.Net framework for further development

Solution Process Advice to Future Seniors  Work early, and work often  BACKUPS!  Divide and conquer approach, don’t get hung up on one part  Ask for help and meet with as many people as possible  Define project early and voice possible frustrations  Develop passion for project

Thank you…  All CS professors  Other CS seniors  My father and the other fine employees at Manitowoc Ice  Bryan Jorgensen & Adam Ziegler  Paul Deibele

 Live demo  Questions ?