1. Team Members: Jonathan Lobaugh Mohammed Hoque Troy Tancraitor Mentors: Dr. Ralph Ford & Ronald McCarty

2. Presentation overview Problem StatementProblem Statement NeedsNeeds RequirementsRequirements Architectural designArchitectural design Robotic layerRobotic layer Processing layersProcessing layers Communications layerCommunications layer

3. Problem statement Our objective is to develop a system that will allow the user to interact effortlessly, simplify user requests, and transfer information to and from a digital medium. It should allow for an improvement of the interface with a conventional machine in daily applications.Our objective is to develop a system that will allow the user to interact effortlessly, simplify user requests, and transfer information to and from a digital medium. It should allow for an improvement of the interface with a conventional machine in daily applications. Primary Objectives:Primary Objectives: Being able to detect the userBeing able to detect the user Being able to detect different human faces and remember themBeing able to detect different human faces and remember them Secondary Objectives:Secondary Objectives: Being able to understand what the users sayBeing able to understand what the users say Being able to detect different voicesBeing able to detect different voices Being able to have the robot moveBeing able to have the robot move according to the user’s gesture. according to the user’s gesture.

4. Needs Recognize faceRecognize face Recognize voiceRecognize voice Must be able to focus on userMust be able to focus on user User authenticationUser authentication Ease of useEase of use Aesthetically appealingAesthetically appealing Respond to voiceRespond to voice Low costLow cost Quick reactionsQuick reactions ReliableReliable AccurateAccurate Non-threateningNon-threatening Standardized adaptorsStandardized adaptors

5. Requirements Ease of UseEase of Use Standardized adaptorsStandardized adaptors Centralized controlCentralized control Minimize connections neededMinimize connections needed CostCost Less than $500 budgetLess than $500 budget ReliabilityReliability Detect a user visually with 90% success rateDetect a user visually with 90% success rate Detect a user’s voice with 80% success rateDetect a user’s voice with 80% success rate

6. Requirements continued AccuracyAccuracy Identify a specific user with 80% accuracyIdentify a specific user with 80% accuracy DurabilityDurability System should last for the duration of the projectSystem should last for the duration of the project SizeSize Occupy less than approximately 2 cubic feetOccupy less than approximately 2 cubic feet EfficiencyEfficiency Finish visual and audio computations within 0.5 secondsFinish visual and audio computations within 0.5 seconds

7. High-level Architectural design Robotic interface Audio processing Image processing Neural network Communications layer Processing Layer Robotic Layer

8. Motor controllerSensor controller Visual controller System controller Motor # LCD display Audio Input device # LED array Interface controller Visual input device Environ. Input device # USB 2.0 I/O Layer Controller Layer Interface Layer Robotic layer diagram

9. Robotic layer I/O controlI/O control Microcontroller based sub-systemsMicrocontroller based sub-systems Main “System” will oversee control of sub- systemsMain “System” will oversee control of sub- systems Three main sub-systemsThree main sub-systems SensorSensor MotorMotor VisualVisual

10. Robotic layer OutputOutput AudioAudio MotionMotion ImagingImaging InputInput CommunicationsCommunications ControlControl

11. Communication layer diagram USB 2.0 Real-time control system Protocol Lock & Key Buffers

12. Communication Layer Represents the link between the robotic and processing layerRepresents the link between the robotic and processing layer Coding will be done in C/C++Coding will be done in C/C++ We will be using a USB 2.0 connectionWe will be using a USB 2.0 connection Two forms of data being transferredTwo forms of data being transferred The audio will be in digital audio formThe audio will be in digital audio form The images will be a 512x512 bitmapThe images will be a 512x512 bitmap Two buffers to store the dataTwo buffers to store the data RTCS will send back commandsRTCS will send back commands

13. Processing layer diagram Interface Image correction Audio finger printing Neural networks Image Parsing USB 2.0 Real-time control system Image processing Audio processing

14. Image layer diagram Face Detection Image Extraction Image processing Real-time control system Neural Network

15. Image layer InputsInputs 512 x 512 bitmap image512 x 512 bitmap image TaskTask Detect and parse faceDetect and parse face OutputOutput 10-15 distinct features of the face10-15 distinct features of the face extracts and frames into a 100x100 formatextracts and frames into a 100x100 format Vectors containing distance values of distinctive featuresVectors containing distance values of distinctive features

16. Audio layer Uses a known database of users vocal informationUses a known database of users vocal information Compares this information to the inputted vocal information from the real-time systemCompares this information to the inputted vocal information from the real-time system Uses a system of longest common subsequences analysis of minor vocal waveform patternsUses a system of longest common subsequences analysis of minor vocal waveform patterns Computes likelihood of known-user ownership of voiceComputes likelihood of known-user ownership of voice Relays known user identity with percentage of certaintyRelays known user identity with percentage of certainty

17. Neural network layer Trained with inputs that are defined by the Image layerTrained with inputs that are defined by the Image layer Trained with 7 usersTrained with 7 users InputsInputs 100x100 images from the Image layer100x100 images from the Image layer Audio percent similarityAudio percent similarity OutputOutput Percent similarity of known usersPercent similarity of known users Returns the percentage to the Real-time control systemReturns the percentage to the Real-time control system

18. Real-time control system Controls the amount of time a process is allotted to runControls the amount of time a process is allotted to run Manages input and output of data to the interface layerManages input and output of data to the interface layer Determines robot’s actions during idle processing timeDetermines robot’s actions during idle processing time Allocates CPU time to priority processesAllocates CPU time to priority processes Controls programmed responses to robotic interface; audio, optical, and mechanicalControls programmed responses to robotic interface; audio, optical, and mechanical