1. Advisor : Dr. Kai-Wei Ke Presenter : D. Jayasakthi Wireless and Broadband Networks Lab, Department of Electrical Engineering and Computer Science, National Taipei University of Technology, Taiwan-106.

2. Contents Introduction Objective Design and Analysis Implementation Experimental Results Conclusions 2

3. Introduction A common activity in most people’s life is driving; therefore, making driving safe is an important issue in everyday life. Even though the driver’s safety is improving in road and vehicle design, the total number of serious crashes is still increasing. Most of these crashes result from impairments of the driver’s attention. 3

4. Drowsy driving is a serious issue in our society not only because it affects those who are driving while drowsy, but also it puts all other road users in danger. Therefore, the use of assisting systems that monitor a driver’s level of vigilance is important to prevent road accidents. These systems should then alert the driver in the case of drowsiness or inattention. 4

5. Drowsiness detection can be done in various ways based on the results of different researchers. The most accurate technique towards driver fatigue detection is dependent on physiological phenomena like brain waves, heart rate etc. Also different techniques based on the behaviors can be used, which are natural and non-intrusive. These techniques focus on observable visual behaviors from changes in a human’s facial features like eyes, head and face. 5

6. Objective The aim of the thesis is develop a prototype for drowsiness detection system. The application is developed using the android SDK and it will detect the heart beat signals from a heart rate monitoring device. Also using this heart rate monitoring device, the ECG signals are obtained. As a result of the electrical stimulation a change in potential of the order of 1mV can be measured during the cardiac cycle. This signal is known as the Electrocardiogram (ECG). 6

7. ECG signal obtained from the sensor is analyzed in frequency domain. In frequency domain, the power spectral density (PSD) is found. From the PSD the Low Frequency(LF) to High Frequency(HF) ratio is estimated. It is found that the LF/HF ratio decreases as the person becomes drowsy. As a result the drowsiness of a person can be detected from this power ratio. 7

8. How it Works Autonomic Nervous System (ANS) activity presents alterations during stress, extreme fatigue and drowsiness. Wakefulness states are characterized by an increase of sympathetic activity and/or a decrease of parasympathetic activity. Extreme relaxation states are characterized by an increase of parasympathetic activity and/or a decrease of sympathetic activity. 8

9. The ANS activity can be measured non-invasively from the Heart Rate Variability (HRV) signal obtained from ECG. Power on low frequency (LF) band (0.04-0.15 Hz) is considered as a measure of sympathetic activity. Power on high frequency (HF) band (0.15-0.4 Hz) is considered of parasympathetic origin in classical HRV analysis. Balance between sympathetic and parasympathetic systems is measured by the LF/HF ratio. 9

10. Design and Analysis Various Signal Processing Methods needed to apply to the ECG signals are: Decimation Hamming Window Fast Fourier Transform Calculate the low to high frequency ratio 10

11. Decimation Consider a band-limited discrete-time signal x(n) with a base-band spectrum X(f). The sampling rate can be decreased by a factor of M through discarding of M–1 samples for every M samples of x(n). 11 h[k] M x[n] v[n] y[n] Filter Sampling Rate Compressor FsFs F s /M

12. Decimation by a factor of M can be achieved through a two-stage process of: (a) Low-pass filtering of the zero-inserted signal by a filter with a cutoff frequency of F s /M, where F s is the sampling rate. (b) Discarding of L–1 samples for every L samples The decimation factor is simply the ratio of the input rate to the output rate. It is usually symbolized by "M", so input rate / output rate=M. 12

13. The sampling frequency of the sensor was 250 Hz which means 250 samples per second. It was very high to process the ECG signals. So the sampling frequency was reduced by 50 Hz. The decimation was done using a low pass filter technique. 13

14. Hamming Window Technique Windowing functions, enhances the ability of an FFT to extract spectral data from signals. Windowing functions act on raw data to reduce the effects of the leakage that occurs during an FFT of the data. There are many window functions available. For an ECG signal the appropriate window function is the Hamming Window. The formula for Hamming window is w(n)=0.54−0.46cos(2πn/N−1). If x(n) is the signal,then we get the windowed signal by multiplying x(n) with the w(n). 14

15. Fast Fourier Transform(FFT) The FFT is a highly elegant and efficient algorithm, which is still one of the most used algorithms in speech processing, communications, frequency estimation, etc., Basic radix-2 algorithm is used which requires N to be a power of 2. FFT is applied to the windowed ECG signal. By applying FFT, the power spectrum is found. LF/HF ratio is calculated every 1 minute. If this ratio decreases then the person is becoming drowsy. 15

16. Implementation Various methods that has been implemented are: Bluetooth Module ECG for Drowsiness Detection Heart Beat Measurement Heart Rate Variability 16

17. I-Mami HRM2 and Android Phone 17 I-Mami HRM2 sensor from Microtime Computer Inc. Garmin Asus A50

18. Pairing Sensor with the Mobile First the device discovery is done in order to connect the sensor with the mobile. If a device is discoverable, it will respond to the discovery request by sharing some information, such as the device name and its unique MAC address. Once a connection is made with a remote device for the first time, a pairing request will be automatically presented to the user. The user must enter a 4 digit pin number for the device to be paired. 18

19. Scan for Bluetooth Devices Pairing Request Sensor has been paired with the mobile 19

20. Bluetooth Communication 20 Start Bluetooth Module Obtain bluetooth device object Use this object to acquire bluetooth socket Initialize bluetooth socket Perform lookup on remote device in order to match UUID Match UUID? Channels will not be opened Share RFCOMM Chanel Obtain signals from sensor No Yes UUID - Universally Unique Identifier

21. 1. Main Screen with all Modules 2.Bluetooth Module 3. List of Paired Devices 4. Sensor Connected to the Mobile 5. Device Not Connected Bluetooth Communication 21

22. ECG Signals Electrocardiography is the interpretation of the electrical activity of the heart over a period of time. The ECG detector works mostly by detecting and amplifying the tiny electrical changes on the skin that are caused during each heartbeat. The I-Mami HRM2 heart rate monitoring device is used to fetch the heart rate of a person and it is displayed in the android mobile with the help of programmable application, developed by using android SDK. 22

23. 1. Select Sensor from menu2. Displays paired devices3. Displays ECG Signals ECG Signals 23

24. Heart Rate Measurement The heart rate is the number of heart beats per minute. Normal heart rate of a human being depends on the age. For example, children will have higher heart rates comparing with the adults. This measurement can be done in various ways with respect to time. 60 seconds (no calculation needed) - most accurate 15 seconds (multiply by 4) 10 seconds (multiply by 6) Less than 10 seconds = less precise 24

25. 1.Heart Rate Measurement Module 2. Select a Device from Menu3. Lists the Paired Device4. Displays the Heart Rate and other values 25

26. Heart Rate Variability Heart rate variability (HRV), known as the variation of the period between consecutive heartbeats over time. In time domain analysis, based on beat to beat or NN intervals some variables are analyzed. They are SDNN: Standard Deviation of all normal to normal intervals index. Often calculated over a 24- hour period. SDANN, the standard deviation of the average NN intervals calculated over short periods, usually 5 minutes. SDANN is therefore a measure of changes in heart rate due to cycles longer than 5 minutes. NN50: Number of pairs of adjacent NN intervals differing by more than 50 ms in the entire recording pNN50: The proportion of NN50 divided by total number of NNs. AVNN: Average of all NN intervals. 26

27. 1. Select a Device from Menu 2. Select the Sensor 3.Displays the HRV 27

28. Implementation of Drowsiness Detection Technique 28 Obtain ECG signal from sensor Reduce the sampling rate to 50 Hz Apply Hamming Window Apply FFT Calculate LF/HF ratio Is Ratio Decreasing ? Person is not drowsy Person Becomes Drowsy Collect next 512 samples of data Start Drowsiness Detection Is max_count ? Alert the person No Yes No A A A A

29. Experimental Setup The testing was performed on two persons, one male and one female. Data was collected when the person’s were awake and asleep. Two hours of data was collected from each of them. The testing was repeated 10 times for both male and female and also when they were awake and asleep. 29

30. Experimental Results for Male, State: Awake 30 Time Ratio Time Ratio Time Ratio Time Ratio 10.2421310.2642610.2468910.2332 20.2388320.2425620.2638920.2055 30.2661330.2577630.2222930.1693 40.2537340.2153640.362940.2314 50.2305350.2087650.2022950.2001 60.2606360.1642660.2601960.2032 70.2823370.2302670.2147970.2403 80.2402380.1696680.2585980.2208 90.193390.2242690.243990.208 100.2287400.2226700.20571000.1926 110.1986410.2303710.19171010.235 120.2254420.2488720.21461020.2028 130.235430.2537730.24831030.2497 140.2333440.2143740.22731040.2472 150.244450.1815750.23851050.2351 160.2342460.2065760.25841060.269 170.2308470.2356770.20571070.2331 180.2424480.2066780.20591080.227 190.243490.2509790.22071090.2168 200.2487500.3267800.21461100.2563 210.2151510.1954810.18631110.2498 220.2485520.1994820.25151120.2436 230.3733530.2196830.20241130.258 240.2247540.2394840.19721140.2463 250.1823550.2682850.20341150.248 260.2164560.1931860.311160.2664 270.2235570.2305870.43131170.1511 280.2562580.2599880.24211180.2676 290.2353590.2595890.16871190.2349 300.2079600.2168900.2371200.2371

31. Time Ratio Time Ratio Time Ratio Time Ratio 10.2101310.2395610.4464910.2337 20.2054320.1931620.2369920.2459 30.2498330.2616630.2071930.2509 40.2184340.2512640.2206940.2789 50.2022350.2453650.2386950.2582 60.2252360.1892660.1996960.2584 70.3057370.2647670.1926970.2341 80.2415380.2533680.2202980.2132 90.2363390.1892690.2341990.2471 100.2142400.2386700.25511000.224 110.3224410.3341710.1941010.2278 120.2765420.2438720.22121020.2573 130.3478430.2074730.25121030.2221 140.2392440.2519740.21681040.2475 150.214450.1945750.23431050.2409 160.2252460.2723760.24941060.2105 170.1949470.2179770.2071070.1766 180.2536480.2199780.21991080.2454 190.2302490.2474790.24211090.2403 200.2191500.2229800.20991100.2519 210.2778510.2615810.27031110.2407 220.2513520.2286820.19931120.2164 230.2589530.2519830.22691130.2379 240.2352540.2884840.26751140.2229 250.2203550.2614850.25461150.3005 260.242560.2437860.21481160.2508 270.2518570.3833870.21341170.2846 280.1995580.2126880.24251180.2016 290.2414590.2533890.23921190.2406 300.3818600.2311900.1971200.2311 Experimental Results for Male, State: Awake 31

32. Experimental Results for Male, State: Asleep Time Ratio Time Ratio Time Ratio Time Ratio 10.2565310.1882610.1447910.1212 20.2506320.1889620.1555920.1293 30.2573330.1783630.1505930.1277 40.2033340.1788640.1487940.1144 50.1949350.1742650.1575950.1127 60.1995360.1837660.1515960.1177 70.1964370.1896670.1402970.1254 80.1902380.1811680.1443980.1922 90.2493390.1784690.1587990.1224 100.2393400.1774700.14931000.1473 110.2236410.1702710.15261010.1132 120.213420.1832720.15411020.1166 130.1981430.176730.14241030.1143 140.2287440.1805740.15061040.1139 150.1925450.1774750.13011050.1157 160.1881460.1506760.13341060.1019 170.1932470.1654770.13421070.1064 180.1936480.1468780.14831080.1052 190.1903490.1626790.13641090.1003 200.1953500.1531800.13491100.1075 210.1862510.1509810.14591110.1025 220.1844520.1686820.14931120.1041 230.1803530.1683830.13471130.1004 240.1862540.1638840.13611140.1032 250.1911550.1541850.12181150.1065 260.194560.1592860.12821160.1007 270.1815570.1501870.13541170.103 280.1859580.1532880.13661180.182 290.1871590.1426890.12031190.1142 300.1797600.1412900.12571200.1092 32

33. Experimental Results for Male, State: Asleep TimeRatioTimeRatioTimeRatioTimeRatio 10.2325310.1696610.1201910.1022 20.2216320.1702620.1291920.1178 30.2187330.1604630.1289930.1182 40.2122340.1576640.1241940.1126 50.2248350.1783650.1224950.1041 60.2483360.1545660.1273960.1073 70.2188370.1417670.112970.1095 80.1958380.1566680.1238980.1162 90.1727390.1678690.1125990.1192 100.1969400.1459700.12781000.1163 110.1575410.1542710.1291010.1121 120.1521420.152720.11981020.1033 130.1906430.1675730.10451030.1045 140.1853440.1742740.11671040.1075 150.1789450.1572750.10231050.1149 160.1594460.17760.10671060.1131 170.1957470.1591770.10341070.1173 180.1884480.1458780.10231080.1182 190.1822490.1556790.10931090.1267 200.1718500.1494800.10011100.114 210.1778510.1434810.1031110.1271 220.1886520.1533820.1021120.1143 230.1728530.1499830.10261130.1132 240.1781540.1407840.11891140.1184 250.1835550.1317850.11211150.1149 260.1659560.1264860.10561160.1163 270.1696570.1434870.10291170.1131 280.1875580.1366880.11341180.1047 290.1771590.1222890.11441190.1093 300.1615600.1212900.1781200.1074 33

34. Experimental Results for Female, State: Awake Time Ratio Time Ratio Time Ratio Time Ratio 10.3062310.4238610.2181910.2469 20.2559320.3814620.5302920.2013 30.2189330.1987630.2424930.246 40.3066340.5112640.3827940.2298 50.2503350.2059650.3045950.2276 60.3134360.2616660.2575960.2616 70.2286370.2665670.2192970.2382 80.255380.2669680.4336980.2388 90.3032390.2048690.5218990.2829 100.2448400.2394700.44831000.2018 110.2244410.2523710.35621010.2648 120.264420.2848720.22251020.238 130.2231430.308730.27221030.211 140.1928440.222740.22091040.2597 150.1939450.2982750.24831050.2362 160.3443460.3239760.25691060.2214 170.2239470.2601770.26341070.246 180.393480.2683780.2511080.248 190.5402490.241790.26761090.2581 200.2433500.2222800.19311100.2702 210.249510.2652810.26221110.6572 220.219520.2782820.20071120.3003 230.2463530.2574830.23571130.2403 240.2929540.4996840.27391140.2261 250.275550.2294850.27061150.273 260.6102560.3677860.25491160.235 270.1954570.636870.26421170.259 280.6796580.2986880.43031180.2495 290.3054590.3553890.23391190.2603 300.2364600.261900.21741200.2342 34

35. Experimental Results for Female, State: Awake Time Ratio Time Ratio Time Ratio Time Ratio 10.4251310.2311610.3823910.2681 20.3031320.2329620.2595920.2239 30.2604330.2434630.1994930.2324 40.2725340.2558640.2499940.2442 50.3506350.28650.2434950.2238 60.3885360.2521660.2428960.2547 70.2435370.2556670.2336970.23 80.2413380.2468680.2392980.2582 90.2435390.2382690.7867990.2346 100.2199400.2119700.20911000.2434 110.2596410.2347710.24261010.2349 120.4623420.2477720.23811020.263 130.2209430.2291730.24511030.408 140.2442440.2487740.23881040.2528 150.2463450.2462750.25661050.2335 160.2715460.1949760.22991060.254 170.252470.2348770.23261070.2513 180.2545480.2502780.23231080.2375 190.273490.2565790.39331090.2241 200.2423500.2327800.2571100.2356 210.212510.2817810.23661110.2347 220.2755520.2667820.2531120.1881 230.2692530.222830.22671130.2278 240.256540.36840.23551140.2328 250.2186550.2389850.21671150.2262 260.2419560.2326860.24691160.2824 270.1862570.2554870.25851170.2763 280.2272580.2509880.24251180.2341 290.2145590.2453890.2771190.3311 300.2594600.217900.2521200.2945 35

36. Experimental Results for Female, State: Asleep Time Ratio Time Ratio Time Ratio Time Ratio 10.3251310.1811610.1623910.1181 20.3031320.1829620.1595920.1139 30.2604330.1934630.1694930.1124 40.2725340.1858640.1599940.1242 50.2506350.1928650.1434950.1238 60.2885360.1921660.1428960.1247 70.2435370.1856670.1596970.123 80.2413380.1968680.1592980.1282 90.2435390.1882690.1567990.1246 100.2199400.1919700.16911000.1244 110.1996410.1847710.15261010.1149 120.1923420.1777720.14811020.1163 130.2209430.1891730.14511030.1108 140.2442440.1887740.13881040.1128 150.2463450.1762750.14661050.1035 160.2715460.1949760.14991060.1054 170.1952470.1848770.13261070.1013 180.1945480.1702780.13231080.1075 190.203490.1665790.12331090.1041 200.2423500.1727800.12571100.1056 210.212510.1817810.12661110.1047 220.2755520.1767820.12531120.1081 230.2692530.1622830.12671130.1078 240.256540.1636840.13551140.1028 250.2186550.1789850.11671150.1062 260.1919560.1626860.13691160.1124 270.1862570.1654870.12851170.1163 280.1972580.1509880.12251180.1152 290.1945590.1453890.12771190.1056 300.1994600.1517900.12521200.1032 36

37. Experimental Results for Female, State: Asleep Time Ratio Time Ratio Time Ratio Time Ratio 10.2487310.1934610.1479910.1084 20.2531320.1877620.1477920.1097 30.2663330.1786630.1506930.1045 40.2385340.1777640.1427940.1018 50.251350.1702650.1455950.1093 60.2474360.1588660.1329960.1087 70.233370.1618670.1499970.1086 80.2284380.1657680.1495980.1123 90.2392390.1725690.1564990.1182 100.2621400.1723700.15471000.1045 110.2381410.1712710.14741010.1189 120.2052420.1638720.13861020.1225 130.1914430.1651730.13021030.1131 140.2076440.1653740.12431040.1064 150.1893450.1581750.12881050.1012 160.1608460.1785760.13081060.1041 170.1925470.1605770.12981070.1053 180.1996480.1742780.13491080.1005 190.1992490.1542790.12061090.102 200.1983500.1537800.12281100.1009 210.1846510.1467810.10661110.1067 220.1992520.1542820.11011120.1056 230.1898530.1524830.11191130.1029 240.1924540.1431840.11851140.1034 250.1915550.1578850.11381150.1053 260.1968560.1632860.12021160.1155 270.1926570.1543870.10161170.1063 280.1825580.1408880.11521180.1002 290.1821590.1453890.10581190.1005 300.1865600.1431900.10071200.1114 37

38. Comparison of Power Ratio for Awake and Sleep States 38

39. Comparison of Power Ratio for Awake and Sleep States 39 0.119053

40. Conclusions In this research work, drowsiness detection has been analyzed based on the ECG signal obtained from the sensor. The application is successfully able to detect the heart beat accurately and it also displays the heart rate variability and the ECG signals in the android devices, respectively. By applying FFT to the obtained ECG signal, the power on the low and high frequency components were measured. Then the LF/HF ratio was calculated for every one minute. 40

41. From the graph it is clear that the HRV analysis on the two hour heart rate time series showed that LF/HF ratio had a decreasing trend when they were asleep or feeling drowsy. When a decreasing trend is identified below 0.17 and for a max_count value of 3, if the value continuously decreases, then an alarm will be invoked automatically in order to alert the driver. 41

42. My contributions for the thesis are Display the ECG signal successfully on the android mobile. Apply the signal processing techniques and process the obtained ECG signal. Find the power spectrum and calculate the LF/HF ratio and save in database for analysis purpose. Collect data when awake and asleep from both male and female for 2 hours. Analyze the power ratio values in both the cases. Detect the drowsy state from the collected data. Compare the power ratio values for both male and female. 42

43. Future Works The efficiency of this system could be further improved by employing the sensors on the seat belt to achieve better accuracy. In addition more data must be collected from human test in order to improve the accuracy of drowsiness judgement. Further with the help of the ECG, we can also analyze the person’s mood on a daily basis. 43

44. 44