AN INTELLIGENT ASSISTANT FOR NAVIGATION OF VISUALLY IMPAIRED PEOPLE N.G. Bourbakis*# and D. Kavraki # #AIIS Inc., Vestal, NY, *WSU, ITRI, Image-Video-Machine Vision Research Lab, Dayton OH.

Outline Introduction Tyflos System Human Tyflos Interaction Visual to Audio Communication Detection of Other Moving Objects Navigation with L-G Landmarks IMPLEMENTATION OF THE TYFLOS PROTOTYPE CONCLUSIONS

Introduction This paper presents the navigation methodology employed by an intelligent assistant (agent) for people with disabilities partial independence. Tyflos –Work in a 3-D dynamic environment –System carries two vision cameras and captures images from the surrounding 3-D environment. –These images into verbal descriptions for each image into a verbal communication.

Introduction In USA –there are more than 3 millions with low vision and millions more completely blind. A great challenge –robotic-like vision Be able to obtain 3-D representations of the surrounding world. provide verbal descriptions of it

Introduction Of particular importance –Unknown environment –High resolution –Analyze –Events based on simple queries –Vision system convert visual descriptions into verbal communication

Introduction Applied artificial intelligence and software technologies –QBIC Query By Image Content –EUROTRA Machine languages translation –STARS Newsletter –MEL An autonomous intelligent vehicle –Intelligent walking robots, 3-D space maps generation, autonomous vision systems, neuromorphic vision chips, the robot with feelings

Introduction verbally see –an intelligent system to make visually impaired people via a machine vision system equipped with speech and natural language understanding technologies. challenges and technological –the navigation methodology of an intelligent assistant for visually impaired people's partial independence.

TYFLOS SYSTEM System's Main Components –The Pair of Glasses Vision Cameras Laser Range Scanner Ear Speaker Microphone Portable Computer Communication System

TYFLOS SYSTEM The Pair of Glasses –The pair of glasses is a important part of the Tyflos system, and several system’s parts are mounted on it.

TYFLOS SYSTEM The Vision Cameras –The vision camera is mounted on the top of the glasses in order to provide a better view of the targeted scene. Camera –High resolution –Color –Small in size and weight.

TYFLOS SYSTEM The Laser Range Scanner –"scan“ the same view with the camera. –For determination of the distances of the existing objects –For 3-D view of a captured image The size of the laser scanner –resolution at least 1cm radius in a distance of 15 meters.

TYFLOS SYSTEM The Ear Speaker –high fidelity component has only one ear speaker through it the user receives the detailed description of the surrounding environment.

TYFLOS SYSTEM The Microphone –User communicates with computer and the appropriate Databases. –Filter To eliminate the noise from the user's environment.

TYFLOS SYSTEM The Portable Computer –It hosts all the important software tools that make the Tyflos system functional. –DBs Image Speech Natural Language –Software Interfaces –Knowledge Conversion Tools –Communication Tools

TYFLOS SYSTEM The Communication System –the user needs to establish communication with other resource portable or mobile information DBs

TYFLOS SYSTEM Source:

HUMAN TYFLOS INTERACTION Tyflos is able to assist the user are ： –where am I ? –describe the surrounding scene. –guide me to a particular point –where is a particular object?

HUMAN TYFLOS INTERACTION Independent –Human assistant by describing to the user the 3-D visual environment making him/her more independent. find an “object” → microphone → scanning the surrounding space → vision camera → analyzed and recognized → ear speaker What Object –describes to the user via the ear speaker what object made a certain sound.

Visual to Audio Communication Where am I ? –GPS (Global Positioning System) Tyflos –uses a Speech synthesizer –provides to the user the names of the streets and / or the names of certain buildings. –provides to the human user a verbal description of the surrounding 3-D scene.

Visual to Audio Communication Describe the Surrounding Scene –SPN graph Generation and Description of a 3-D Surrounding Unknown Environment Fusion of Images & range Data for 3-D Modeling of the Free Space

Visual to Audio Communication Generation and Description of a 3-D Surrounding Unknown Environment –stereo speaker –vision camera –laser range data Dr.Albus and Dr. Bourbakis –develop 3-D images via audio signals generated by two speakers during the motion of a sound generation equipment.

Visual to Audio Communication Fusion of Images & range Data for 3-D Modeling of the Free Space –Cameras Stabilization Issues synthesis of images –Generation & Representation of the Navigation Space Free Navigation Space (FNS) –orientation-angle –magnetic north –current position –inclination angle shape SH –” open-navigation corridors ”

Visual to Audio Communication Where is a Particular object? –particular object is converted into a hierarchical SPN graph –the SPN graph is compared to the system's KB –recognized or appropriately classified –speech DB and ear speakers

Visual to Audio Communication Guide me in a Particular Place –Collision Free Path Planning the system attempts to select the First Best Choice (FBC) open navigation corridors. –Detection of Other Moving Objects How Tyflos detects other moving objects in the same navigation space What is the perceived size of the moving object What is the direction and the velocity of the moving object How many different moving objects are in the same free space

Detection of Other Moving Objects How Tyflos detects other moving objects in the same navigation space –the space shape changing methodology shape SH & Dt do(Dt) = VRa*Dt → P' P' → SH' –IF SH" = SH' THEN there is no other moving object in the same navigation space –ELSE there is at least one moving object in the same free navigation space. –the differences detected from the comparison of two consecutive 2-D images. ”motion”

Detection of Other Moving Objects What is the perceived size of the moving object –When Tyflos, detects another moving object, Rb, in the same navigation area, then it attempts to define the size (dimensions) "perceived" from its current position P. Rb = |dk'(t(i+1)) - dk(t(i+1))| –Corresponding window 2-D image Knowledge Base

Navigation with L-G Landmarks L-G Landmarks –local-global graphs

Knowledge Base (KB) –successful and flexible generation of Space Maps is the development of a knowledge base able to store, process and access the Space Maps' forms. Communication and Mobile DBs Issues –to carry CDs with different DBs –its computer to be connected with a multimedia network in order to download the appropriate DB

IMPLEMENTATION OF THE TYFLOS PROTOTYPE Integration of AI Methodologies –The actual implementation of the Tyflos system combines the state-of-the-art in AI methodologies. The Prototype –AIIS (American Institute of Indian Studies) –Tyflos system consists of the pair of glasses, a Hitachi high resolution, color vision camera, a portable computer Mc book- note, an ear speaker, a voice synthesizer and a microphone.

CONCLUSIONS It will provide the ability to the user to understand better the surrounding environment, and give more independence in his/her motion. –Vision –Speech However, a significant progress has been done, and new methods in both vision and speech developed by other researchers will be appropriately used.

Thank you for your attention