1. ARTIFICIAL PASSENGER (A Sleep Prevention Dialogue Based Car System) Prepared By Jesalpura Riddhi 09-IT-14 Guided By Mansi Parmar

2. Index  Introduction  Why such system  Artificial passenger  What does it do  Devices  Working component  Advantages  Disadvantages  Application  Future enhancement  Conclusion  Bibliography

3. Introduction IBM (International business machines corporation, NY) has developed a software that holds a conversation with the driver to determine whether the driver can respond alertly enough, called “ Artificial Passenger ”. IBM (International business machines corporation, NY) has developed a software that holds a conversation with the driver to determine whether the driver can respond alertly enough, called “ Artificial Passenger ”. This was designed to make long solo journeys safer and long solo journeys safer and more bearable. more bearable.

4. Why Such System? According to a national survey in UK and USA, it is observed that the driver fatigue annually causes According to a national survey in UK and USA, it is observed that the driver fatigue annually causes 10000 crashes 10000 crashes 1500 deaths 1500 deaths 7100 injuries 7100 injuries  Majority of off-road accidents observed, were caused by eye closure of half and even 2-3 seconds, where the normal human eye blinks at 0.2-0.3 seconds.

5. What is an Artificial Passenger? Natural language e-companion. Natural language e-companion. Sleep preventive device in cars to overcome drowsiness. Sleep preventive device in cars to overcome drowsiness. Life safety system. Life safety system.

6. What Does It Do? Detects alarm conditions through sensors. Detects alarm conditions through sensors. Broadcasts pre-stored voice messages over the speakers. Broadcasts pre-stored voice messages over the speakers. Captures images of the driver. Captures images of the driver.

7. Condition Sensor Mobile Indicator Device

8. About Artificial Passenger The AP is an Artificial Intelligence based companion that will be resident in software and chips embedded in the automobile dashboard. The AP is an Artificial Intelligence based companion that will be resident in software and chips embedded in the automobile dashboard. The system has a conversation planner that holds a profile of you, including details of your interests and profession. The system has a conversation planner that holds a profile of you, including details of your interests and profession.

9. A microphone picks up your answer and breaks it down into separate words with speech-recognition software. A microphone picks up your answer and breaks it down into separate words with speech-recognition software. A camera built into the dashboard also tracks your lip movements to improve the accuracy of speech recognition. A camera built into the dashboard also tracks your lip movements to improve the accuracy of speech recognition.

10. A voice analyzer then looks for signs of tiredness by checking to see if the answer matches your profile. Slow responses and lack of attention are signs of fatigue. A voice analyzer then looks for signs of tiredness by checking to see if the answer matches your profile. Slow responses and lack of attention are signs of fatigue. If you reply quickly and clearly, the system judges you to be alert and tells the conversation planner to continue the line of questioning. If you reply quickly and clearly, the system judges you to be alert and tells the conversation planner to continue the line of questioning.

11. If your response is slow or doesn’t make sense, the voice analyzer assumes you are dropping off and acts to get your attention. If your response is slow or doesn’t make sense, the voice analyzer assumes you are dropping off and acts to get your attention. If driver displays signs of fatigue, the artificial passenger might be programmed to open all the windows, sound a buzzer, increase background music volume, or even spray the driver with ice water. If driver displays signs of fatigue, the artificial passenger might be programmed to open all the windows, sound a buzzer, increase background music volume, or even spray the driver with ice water.

12. Devices Used in Artificial Passenger Eye tracker Eye tracker Voice recognizer or speech recognizer Voice recognizer or speech recognizer Natural language processor Natural language processor Driver analyzer Driver analyzer Conversational planner Conversational planner Alarm Alarm Microphone Microphone Camera Camera

13. Working Components Eye-Tracker Collecting eye movement data requires both hardware and software. Collecting eye movement data requires both hardware and software. Hardware:: Head-Mounted systems or remote systems Both systems measure the corneal reflection of the infrared LED, which illuminates and generates a reflection off the surface of the eye. Both systems measure the corneal reflection of the infrared LED, which illuminates and generates a reflection off the surface of the eye. This action causes the pupil to appear as a bright disk in contrast to surrounding iris and creates a small glint underneath the pupil which is used for calibration and tracking. This action causes the pupil to appear as a bright disk in contrast to surrounding iris and creates a small glint underneath the pupil which is used for calibration and tracking. Head mounted systems use multiple data points to record eye movement and measures pupil glint from multiple angles. Head mounted systems use multiple data points to record eye movement and measures pupil glint from multiple angles. Remote systems,in contrast, measure the orientation of the eye relative to a fixed unit such as camera. Remote systems,in contrast, measure the orientation of the eye relative to a fixed unit such as camera.

14. Monitoring System

15. Software: Data collection and analysis is handled by eye-tracking software. Data collection and analysis is handled by eye-tracking software. Software catalogs eye-tracking in one of the two ways Software catalogs eye-tracking in one of the two ways 1. ERICA’s Eye Gaze software uses a small red x to represent eye movement. 2. In other, data is stored as a series of x/y coordinates related to specific grid points on the computer screen

16. Eye Tracker

17. Working Components( cont…) Automatic Speech Recognition There are two ASRs used in the system: There are two ASRs used in the system:  First one is “speaker independent” and used for decoding voice signals of the driver.  Second one operates with voice car media and decodes tapes, audio, telephones, mails etc.

18. Working Components( contd…) Natural Language Processor Processes the decoded signals of voice or text data provided from the ASR. Processes the decoded signals of voice or text data provided from the ASR.  Identifies related meanings from the contents of the decoded messages.  Produces variant of responses.  This output goes to the driver analyzer as an input.

19. Working Components( contd…) Driver Analyzer Receives the textual data and voice data. Receives the textual data and voice data. Measures the time response using a clock. Measures the time response using a clock. Time responses, conclusions about driver’s alertness will be passed on to the conversational planner. Time responses, conclusions about driver’s alertness will be passed on to the conversational planner.

20. Working Components( contd…) Conversational Planner It is the heart of the system. It is the heart of the system. Instructs the language generator to produce the response. Instructs the language generator to produce the response. If the driver is in perfect condition CP instructs the language generator to continue the conversation otherwise it will be instructed to change the conversation. If the driver is in perfect condition CP instructs the language generator to continue the conversation otherwise it will be instructed to change the conversation.

21. Working Components (contd…) Alarm If the CP (conversational planner) receives information that the driver is about to fall asleep an alarm system is activated. If the CP (conversational planner) receives information that the driver is about to fall asleep an alarm system is activated.Microphone For picking up words and separate For picking up words and separate them by some internally used software them by some internally used software for conversation. for conversation.

22. Working Components (contd…) Camera This will track the lip movements of the driver. This will track the lip movements of the driver. Used to improve the accuracy of the speech recognition. Used to improve the accuracy of the speech recognition.

23. . Showing the dashboard of the car where the whole artificial system is generally attached.

24. Steps Involved in Tracking Automatically initialize lips and eyes using color predicates and connected components. Automatically initialize lips and eyes using color predicates and connected components. Track lip corners using dark line between lips and color predicate even through large mouth movement like yawning. Track lip corners using dark line between lips and color predicate even through large mouth movement like yawning.

25. Construct a bounding box of the head. Construct a bounding box of the head. Determine rotation using distances between eye and lip feature points and sides of the face. Determine rotation using distances between eye and lip feature points and sides of the face. Determine eye blinking and eye closing using the number and intensity of pixels in the eye region. Determine eye blinking and eye closing using the number and intensity of pixels in the eye region. Determine driver vigilance level using all acquired information. Determine driver vigilance level using all acquired information.

26. Detecting Driver Vigilance Aiming a single camera at a head of the driver. Aiming a single camera at a head of the driver. Detecting frequency of up and down nodding and left to right rotations of the head within a selected time period with the camera. Detecting frequency of up and down nodding and left to right rotations of the head within a selected time period with the camera. Determining frequency of eye blinking and eye closing. Determining frequency of eye blinking and eye closing. Determining frequency of yawning of the driver within the selected time period with the camera Determining frequency of yawning of the driver within the selected time period with the camera Generating an alarm signal in real time if the frequency value of the up and down nodding, the left to right rotations, the eye blinking, the eye closings, the yawning exceeds a selected threshold value. Generating an alarm signal in real time if the frequency value of the up and down nodding, the left to right rotations, the eye blinking, the eye closings, the yawning exceeds a selected threshold value.

27. Advantages It is broadly used to prevent accident. It is broadly used to prevent accident. Artificial component establishes interface with other drivers very easily. Artificial component establishes interface with other drivers very easily. If the driver gets a heart attack or drunk, it will send signals to vehicles nearby about this so driver there become alert. If the driver gets a heart attack or drunk, it will send signals to vehicles nearby about this so driver there become alert.

28. Disadvantages It is costly. It can only be installed to high profiled

29. Applications Interface with Neighboring Cars - Determines if a driver presents a high safety risk for ex. falling asleep, tired, inexperienced or under the influence of alcohol and signals the cars nearby to be careful of the driver. Interface with Neighboring Cars - Determines if a driver presents a high safety risk for ex. falling asleep, tired, inexperienced or under the influence of alcohol and signals the cars nearby to be careful of the driver. Medical Application -The system can monitor a driver and detect if they are sick,for ex. having a stroke or heart attack. Medical Application -The system can monitor a driver and detect if they are sick,for ex. having a stroke or heart attack. In any problem it alerts the vehicles near by, so the driver there can become alert. In any problem it alerts the vehicles near by, so the driver there can become alert. Opens and closes the doors and windows of the car automatically. Opens and closes the doors and windows of the car automatically. It is also used for the entertainment. It is also used for the entertainment. Also used in cabins in airplanes, trains, boats etc Also used in cabins in airplanes, trains, boats etc

30. Future Implementation Will provide us with shortest time routing based on road conditions changing because of weather and traffic, information about the cars on the route, destination requirement (as-flight has been delayed etc) Will provide us with shortest time routing based on road conditions changing because of weather and traffic, information about the cars on the route, destination requirement (as-flight has been delayed etc)

31. Conclusion Method for monitoring driver alertness Method for monitoring driver alertness Sufficient time to avert an accident. Sufficient time to avert an accident. Successful implementation of Artificial passenger would allow use of various services in car like reading emails, navigation, downloading music files, voice games etc without compromising on driver safety Successful implementation of Artificial passenger would allow use of various services in car like reading emails, navigation, downloading music files, voice games etc without compromising on driver safety

32. Bibliography L R Bhal et al. “Performance of the IBM speech recognition system L R Bhal et al. “Performance of the IBM speech recognition system www.freepatentsonline.com/4682348.html www.freepatentsonline.com/4682348.html www.slideshare.com www.slideshare.com www.about.com www.about.com www.answers.com www.answers.com www.google.com www.google.com www.wikipedia.com www.wikipedia.com

33. Thank You… Thank You…