GROUP SD1501 IOT VISION ADVISORS: BOB WEINMANN, SAMEE KHAN CLIENT: APPAREO Group Members Dale Bromenshenkel James Massey Bradley Hoffmann

INTRODUCTION Problem Problem How do you give a robot the ability to see? How do you give a robot the ability to see? Once the robot is able to ‘see’, how does it make decisions based on that? Once the robot is able to ‘see’, how does it make decisions based on that? How can you share that data with other people/devices? How can you share that data with other people/devices? Why is it important for a robot to see? Why is it important for a robot to see? Many tasks performed by humans are possible only through visual input. Many tasks performed by humans are possible only through visual input. Many monotonous tasks could be solved by a robot using computer vision. Many monotonous tasks could be solved by a robot using computer vision. Why is it important to share that data? Why is it important to share that data? Sharing the data allows it to be used by other applications. Sharing the data allows it to be used by other applications. Humans can use that data improve existing technology and develop new solutions. Humans can use that data improve existing technology and develop new solutions.

REQUIREMENTS Client Requirements Build something we are passionate about. IoT device capabilities Team Requirements Computer Vision Object Detection Image Color Segmentation Save coordinates of object to text file Robot Controlled by a Raspberry Pi Controlled wirelessly Integrate with other systems to be controlled by application Application Must be able to control outside scripts/applications Must have a physics system Must have the ability to make a GUI

TECHNICAL CONTENT Computer Vision

SIMPLECV Overview Overview SimpleCV is an open source computer vision library that can implement OpenCV without a high learning curve. SimpleCV is an open source computer vision library that can implement OpenCV without a high learning curve. Built in commands allow the use for display augmenting, image/video processing, object tracking, color mapping, color manipulation, data storage and extraction, etc. Built in commands allow the use for display augmenting, image/video processing, object tracking, color mapping, color manipulation, data storage and extraction, etc. We implement this library into python scripts. We implement this library into python scripts.

OBJECT TRACKING Combined object tracking techniques Combined object tracking techniques Blobs Blobs The blobs command is a predetermined script that allows for object detection. The blobs command is a predetermined script that allows for object detection. By implementing this command we do not have to re-invent the wheel. By implementing this command we do not have to re-invent the wheel. Ability to detect different geometric shapes, color, edges, data each blob contains. Ability to detect different geometric shapes, color, edges, data each blob contains. Camera/Display Camera/Display Allow the use to open and manipulate video through display Allow the use to open and manipulate video through display Image Processing Image Processing HSV and RGB color schemes HSV and RGB color schemes Histograms Histograms Image segmentation Image segmentation

OBJECT TRACKING

COLOR MAPPING/MANIPULATION RGB Color Scheme RGB Color Scheme Theory every image contains Red, Green and Blue fundamental colors. Theory every image contains Red, Green and Blue fundamental colors. Pixelated and mapped on screen with Red, Green and Blue pixels values Pixelated and mapped on screen with Red, Green and Blue pixels values RGB in SimpleCV RGB in SimpleCV By importing SimpleCV into our base code we can segment images into RGB components. By importing SimpleCV into our base code we can segment images into RGB components. Pushing back color in correlation with our object detection can allow us to segment our object and attain coordinates of the individual object minus its surrounding. Pushing back color in correlation with our object detection can allow us to segment our object and attain coordinates of the individual object minus its surrounding.

COLOR MAPPING/MANIPULATION HSV Color Scheme HSV Color Scheme Hue, Saturation, Value Hue, Saturation, Value Hue = Color in General Hue = Color in General Saturation = Presence of White Saturation = Presence of White Value = Brightness of Color Value = Brightness of Color HSV unlike RGB allows for a wider range of color and manipulation HSV unlike RGB allows for a wider range of color and manipulation No need to segment using Red, Green and Blue. We can use direct values of color. No need to segment using Red, Green and Blue. We can use direct values of color. HSV can be directly implemented and imported into python script using SimpleCV. HSV can be directly implemented and imported into python script using SimpleCV.

DESIGN AND TESTING OPTIONS Controlled conditions Controlled conditions Perfect light Perfect light Background Background Color of object Color of object Testing RGB vs. HSV Testing RGB vs. HSV Histograms = Map of image in pixel values Histograms = Map of image in pixel values Testing Object Tracking through the use of visual effects. Testing Object Tracking through the use of visual effects. Set color scheme to gray scale to perfect object detection. Set color scheme to gray scale to perfect object detection. Saving Coordinates of objects to text file Saving Coordinates of objects to text file

TECHNICAL CONTENT Raspberry Pi Robot

OVERVIEW OF ROBOT The robot has two powered wheels and a third to balance it. The robot has two powered wheels and a third to balance it. The two wheels will be powered by a DC motor. The two wheels will be powered by a DC motor. These DC motors will be connected to a couple H-bridges that will allow them to spin both forward and backwards. These DC motors will be connected to a couple H-bridges that will allow them to spin both forward and backwards. The movements will be executed by the Pi, with commands sent from a PC. The movements will be executed by the Pi, with commands sent from a PC. The robot will be completely wireless The robot will be completely wireless

SPECIFICS- MOTORS These DC motors are capable of 200mA for a 6v input These DC motors are capable of 200mA for a 6v input They can move.4 Kg/cm They can move.4 Kg/cm The wheels snap right on to the output The wheels snap right on to the output There is a 1:48 gear ratio There is a 1:48 gear ratio

SPECIFICS- RASPBERRY PI The robot will be controlled by a Raspberry Pi 2 The robot will be controlled by a Raspberry Pi 2 The GPIO pins of the Raspberry Pi will be the way the software will interact with the hardware The GPIO pins of the Raspberry Pi will be the way the software will interact with the hardware The Pi will be controlled wirelessly through SSH with a USB Wi-Fi dongle The Pi will be controlled wirelessly through SSH with a USB Wi-Fi dongle

RASPBERRY PI GPIO The Raspberry Pi is equipped with 40 pins 17 of which are GPIO The Raspberry Pi is equipped with 40 pins 17 of which are GPIO The motors will be controlled from GPIO pins 17, 18, 22, 23 The motors will be controlled from GPIO pins 17, 18, 22, 23 4 pins are needed because 17 and 18 control one motor forward and reverse and 22 and 23 for the other motor 4 pins are needed because 17 and 18 control one motor forward and reverse and 22 and 23 for the other motor The GPIO pins output 3.3v The GPIO pins output 3.3v

RYANTECK MOTOR CONTROL BOARD This is a pre-built board specifically for the Raspberry Pi This is a pre-built board specifically for the Raspberry Pi Its purpose is to control two motors and allow them to go forward and backwards Its purpose is to control two motors and allow them to go forward and backwards Allows for a separate source of power to drive the motors, because the Pi cannot supply enough current. Allows for a separate source of power to drive the motors, because the Pi cannot supply enough current. The board uses the TI chip SN754410NE, which is a quadruple high-current half-H driver The board uses the TI chip SN754410NE, which is a quadruple high-current half-H driver

BODY FOR THE ROBOT We will be designing the body of the robot to house the components. We will be designing the body of the robot to house the components. The body will be 3D printed. The body will be 3D printed. The top will have two distinguishable marks on it that the camera will be able to detect for location and direction. The top will have two distinguishable marks on it that the camera will be able to detect for location and direction.

POWER FOR THE ROBOT Raspberry Pi Power The Raspberry Pi requires 5 volts to run from a standard micro USB port For this we thought a portable USB charging pack would work the best These charging packs are commonly used for cell phones and output 5 volts I had one on hand that had a capacity of 2200mAh After testing it lasted over 2 hours before needing a charge Motor Power The motors operate at 6 volts so the USB power pack could have probably worked but we decided to go with batteries instead 4 AA batteries in series output 6 volts and are a perfect solution to power the motors

CODE FOR THE ROBOT The code that is on the Pi is written in python. The code that is on the Pi is written in python. The robot is coded to have 5 main actions: Forward, Reverse, Turn Left, Turn Right, and Stop. The robot is coded to have 5 main actions: Forward, Reverse, Turn Left, Turn Right, and Stop. We set it up to react to keyboard inputs in such a way that: W=forward, S= Reverse, A=Left, D=Right, Q= Stop. We set it up to react to keyboard inputs in such a way that: W=forward, S= Reverse, A=Left, D=Right, Q= Stop. GPIO pins 17 and 18 are for motor 1 and 22 and 23 are for motor 2. GPIO pins 17 and 18 are for motor 1 and 22 and 23 are for motor 2. ForwardReverseTurn LeftTurn RightStop GPIO GPIO GPIO GPIO

WIRELESS INTERACTION The Pi needs to be able to receive commands wirelessly. The Pi needs to be able to receive commands wirelessly. Considered using WebIOPi Considered using WebIOPi We researched many ways to do this and found a solution that works over the network that both the PC that is sending the commands and the Pi are connected to. We researched many ways to do this and found a solution that works over the network that both the PC that is sending the commands and the Pi are connected to. The way we are doing it is through an SSH connection using PuTTy and using the python module called SendKeys The way we are doing it is through an SSH connection using PuTTy and using the python module called SendKeys

TECHNICAL CONTENT Application

APPLICATION - OVERVIEW Base Requirements Recap Base Requirements Recap Control outside scripts, Customizable GUI, Physics Engine Control outside scripts, Customizable GUI, Physics Engine Solution Solution Unity3D Unity3D Python Scripts Python Scripts Putty Putty

APPLICATION – UNITY3D Supports a wide variety of platforms Supports a wide variety of platforms Windows Standalone, Webplayer, Android, iOS, Linux, etc. Windows Standalone, Webplayer, Android, iOS, Linux, etc. Ability to easily create three dimensional scenes Ability to easily create three dimensional scenes Can use built in objects or import custom objects Can use built in objects or import custom objects Comes with Monodevelop IDE Comes with Monodevelop IDE Has a customizable UI system Has a customizable UI system Buttons, sliders, text, images, etc. Buttons, sliders, text, images, etc. And much much more! And much much more!

UNITY - ENVIRONMENT

UNITY - MONODEVELOP Supported Languages Supported Languages C#, Javascript (Unityscript), Boo C#, Javascript (Unityscript), Boo F#, Visual Basic,.Net, C, C++,Vala F#, Visual Basic,.Net, C, C++,Vala Allows outside libraries to be imported. Allows outside libraries to be imported. Loads of other tools available Loads of other tools available Debugging, Source Control, Unit Testing, etc. Debugging, Source Control, Unit Testing, etc.

MONODEVELOP - IDE

APPLICATION - PYTHON Python Python Interpreted, high level object oriented, scripted language. Interpreted, high level object oriented, scripted language. Runs through console. Runs through console. Does not need to be compiled. Does not need to be compiled. Can integrate modules (libraries) Can integrate modules (libraries)

Client program for SSH, Telnet, and Rlogin network protocols Client program for SSH, Telnet, and Rlogin network protocols Can access devices over Wi-FI Can access devices over Wi-FI Serial Serial Used for sending commands to Raspberry Pi Used for sending commands to Raspberry Pi APPLICATION - PUTTY

PROJECT STATUS - OVERVIEW 3 different systems 3 different systems CV, Robot, Application CV, Robot, Application Work on bringing them together Work on bringing them together Testing Area Testing Area Scaled down Scaled down Github Repository Github Repository Code Code

PROJECT STATUS - TASKS REMAINING Computer Vision Implement Hue Color Control Implement Hue Color Control Recognize different colors Recognize different colors Recognize multiple objects Recognize multiple objects Work on collecting more data Work on collecting more data Statistical data Statistical data Robot Make a body Make a body 3D print / actobotics 3D print / actobotics Add ball catcher arm Add ball catcher arm Stop momentum of arm Stop momentum of arm Explore Wi-fi control Explore Wi-fi control Add the ability to program through Wi-Fi Add the ability to program through Wi-Fi Application Add GUI control methods Add GUI control methods Robot Robot Settings/Modes Settings/Modes Implement physics simulation Implement physics simulation Add ability to create more objects in scene Add ability to create more objects in scene Add additional platforms Add additional platforms Web, Android Web, Android

PROJECT STATUS - TIMELINE

BUDGET

SUMMARY SLIDE Accomplished We have created a system which can take in camera input, process that data, and then use that data in an application and robot. Future We hope to be able to simulate the environment more, and have the robot be even more autonomous in its decision making.

QUESTIONS?