1. Automated camera-based fall detection of elderly persons
Alex Edgcomb
Department of Computer Science and Engineering
University of California, Riverside
Copyright © 2014 Alex Edgcomb, UC Riverside.

2. Copyright © 2014 Alex Edgcomb, UC Riverside.
Outline
Elderly person falls and background work
SynchSM and moving-region-based fall detection
Other related work
Copyright © 2014 Alex Edgcomb, UC Riverside.

3. Falls in the elderly population need to be detected
Leading cause of injury-related hospitalization1 and death2
34% have fallen in the last year3
14% have fallen more than once3
Post-fall long lie correlated with passing-away4
50% who experience a long lie pass-away within 6 months4
In the elderly population, falls are the leading cause of injury-related hospitalization and death. 34% of elderly persons have fallen in the last year, while 14% have fallen more than once. Post-fall long lie (60+ minutes on the ground) is correlated with passing-away. 50% who experience a long lie pass-away within 6 months.
1Baker, S.P. and A.H. Harvey. Fall injuries in the elderly. Clinics in geriatric medicine, 1985.
2Hoyert, D.L., K.D. Kochanek, and S.L. Murphy. Deaths: final data for National vital statistics reports, 1999.
3Lord S.R., J.A. Ward, P. Williams, and K.J. Anstey. An epidemiological study of falls in older community-dwelling women. Australian journal of public health, 1993.
4Wild, D., U.S. Nayak, and B. Isaacs. How dangerous are falls in old people at home? British medical journal (Clinical research ed.), 1981.
Copyright © 2014 Alex Edgcomb, UC Riverside.

4. Falls need to be automatically detected. False alarm rates must be low.
In-home care
$3,400 – 5,800 / mo.5
Automated monitoring
Under $100 / mo.
Discontinued…
AmberSelect and Alert1
Discontinued b/c false alarms too high
lifeline-products/auto-alert
Falls need to be automatically detected, as automated monitoring is much less expensive than in-home care.
5Genworth 2013: Cost of care.
about-genworth/industry-expertise/cost-of-care.html
Copyright © 2014 Alex Edgcomb, UC Riverside.

5. Reasons for video-based assistive monitoring
Privacy
enhance-able7,8,9
Detect many
events and trends
Body-worn
Body-worn devices have many problems with patient adherence, whether intention or unintentional. Video can detect many events, such as falls , and trends, such as decreased energy expenditure. Also, video can be privacy-enhanced to obscure the monitored person, as privacy is critical for the adoption of assistive technology.
Pro: Anywhere
6Bergmann, J.H.M. and A.H. McGregor. Body-Worn Sensor Design: What Do Patients and Clinicians Want? Annals of Biomedical Engineering. Volume 39, pgs , 2011.
7Beach, S., R. Schulz, K. Seelman, R. Cooper and E. Teodorski. Trade-Offs and Tipping Points in the Acceptance of Quality of Life Technologies: Results from a Survey of Manual and Power Wheelchair Users. Intl. Symposium on Quality of Life Technology, 2011.
8Beach, S., R. Schulz, J. Downs, J. Mathews, B Barron and K. Seelman. Disability, Age, and Informational Privacy Attitudes in Quality of Life Technology Applications: Results from a National Web Survey. ACM Transactions on Accessible Computing, 2009.
9Demiris, G., M.J. Rantz, M.A. Aud, K. D. Marek, H.W. Tyrer and M. Skubic, A.A. Hussam. Older adults’ attitudes towards and perceptions of ‘smart home’ technologies: a pilot study. Medical Informatics and The Internet in Medicine, 2004.
Con: Not always worn6
Copyright © 2014 Alex Edgcomb, UC Riverside.

6. Approaches to camera-based fall detection
Minimum bounding rectangle (MBR)
Head tracking
3D projection10
laying
standing
There are three general approaches to camera-based fall detection in academic research, shown here in order of increasing computational complexity. The most common approach is using the MBR.
Image from paper by Auvinet20
Increasing order of computational complexity
10Auvinet, E., F. Multon, A. Saint-Arnaud, J. Rousseau, and J. Meunier. Fall detection with multiple cameras:
An occlusion-resistant method based on 3-d silhouette vertical distribution. Information Technology in
Biomedicine, IEEE Transactions on 15, no. 2 (2011):
Copyright © 2014 Alex Edgcomb, UC Riverside.

7. MBR-based fall detection
Hung – Occupied area and height11
Miaou – Height-to- width threshold12
Thome – Height and width probabilistic model13
Hung's algorithm combined two orthogonal cameras by multiplying the width of the MBR from each camera, calling the result the occupied area. The algorithm determined whether a person was standing, sitting, or laying using the height and occupied areas as features. If the person remained laying for an extended period of time, then a fall was said to have occurred.
Miaou's algorithm used a ceiling-mounted camera with a 360 degree view and extracted the person silhouette to produce an MBR. The height to width ratio of the MBR was a feature, and if that ratio exceeded a threshold, then a fall was said to have occurred.
Thome used camera placement and type similar to Hung’s. Thome's algorithm computed the angle between the height and width of the MBR then passed the angle to a layered hidden Markov model, which included a layer for the angle observations, the person’s poses, behavioral patterns, and finally fall likelihood.
11Hung, D.H. and H. Saito. The Estimation of Heights and Occupied Areas of Humans from Two Orthogonal Views for Fall Detection. IEEJ Trans. EIS 133, no. 1, 2013.
12Miaou, S.-G., P.-H. Sung and C.-Y. Huang. A Customized Human Fall Detection System Using Omni-Camera Images and Personal Information.
Proceedings of the 1st Distributed Diagnosis and Home Healthcare Conference, 2006.
13Thome, N., S. Miguet and S. Ambellouis. A Real-Time, Multiview Fall Detection System: A LHMM-Based Approach. IEEE Transactions on Circuits and Systems
for Video Technology, Vol. 18, No. 11, November 2008.
Copyright © 2014 Alex Edgcomb, UC Riverside.

8. Head tracking and 3D projection
Rougier – Head’s vertical velocity14
Anderson – Combine two silhouettes15
standing
laying
Auvinet – Person’s height volume16
Rougier developed a 3D head tracking algorithm using a single camera. If the head’s vertical velocity exceeded a threshold, then a fall was said to have occurred.
Anderson combined the silhouettes from two cameras then used fuzzy logic to determine whether the person was standing, sitting, or laying. If laying for 5 seconds, then a fall was said to have occurred.
Auvinet used 3 or more cameras to produce a person’s height volume. If the height volume was below a threshold for 5 seconds, then a fall was said to have occurred.
14Rougier, C., J. Meunier, A. St-Arnaud, and J. Rousseau. Monocular 3D head tracking to detect falls of elderly people. In Engineering in Medicine and Biology Society, EMBS'06. 28th Annual International Conference of the IEEE, pp IEEE, 2006.
15Anderson, D., et al. Linguistic summarization of video for fall detection using voxel person and fuzzy logic. Computer Vision and Image Understanding 113, 2009.
16Auvinet, E., F. Multon, A. Saint-Arnaud, J. Rousseau, and J. Meunier. Fall detection with multiple cameras: An occlusion-resistant method based on 3-d silhouette vertical distribution. Information Technology in Biomedicine, IEEE Transactions on 15, no. 2, 2011.
Copyright © 2014 Alex Edgcomb, UC Riverside.

9. Copyright © 2014 Alex Edgcomb, UC Riverside.
Outline
Elderly person falls and background work
Moving-region and synchSM-based fall detection
Other related work
Copyright © 2014 Alex Edgcomb, UC Riverside.

10. Person tracking with in-home video via foregrounding
GMM = Gaussian mixture model
Background model17,18
(Pixel-level GMM based on color)
Minimum bounding
rectangle (MBR)
Current
frame
Foreground
MBR builder (adjacent pixel groups merged)
MBR filters (dampen, smooth, & glitch-removal)
Person tracking is the first step in video-based fall detection. Each pixel in the current frame is subtracted from the respective pixel in the background image. If the difference is significant, then the pixel is colored black. Otherwise, the pixel is colored white. The resulting image is the foreground. Adjacent pixel groups are merged, starting from the largest groups, to produce a minimum bounding rectangle, which is usually around a person. A few filters are applied to the MBR to mitigate noise.
The background model is not aware of a person, just the pixel colors. So, an at rest person would be learned into the background within seconds.
(press next button to animation green box)
We stop learning the background if an insignificant amount of motion is observed. Also, a second frame with the MBR area replaced by the background is learned to slow the person from being learned.
(press next button to animate text box on right)
Person tracking may occur on the camera itself. Computer vision researchers have produced, what I assume to be, more accurate person trackers; however, those trackers tend to be 10x slower because those trackers compute 3D projections, as well as additional modeling. Since the tracking may occur on the camera itself, we use our faster person tracker. However, if resources allow, then use the more accurate person tracker.
Person tracking may occur on the camera itself
Computer vision person trackers tend to be 10x slower because of 3D projections and additional modeling
Stop learning background if insignificant amount of motion
Learn a second frame with MBR area replaced by background
17Zivkovic, Z. Improved adaptive Gaussian mixture model for background subtraction. In Pattern Recognition, ICPR Proceedings of the 17th International Conference on, vol. 2, pp IEEE, 2004.
18OpenCV. November 2013.
Copyright © 2014 Alex Edgcomb, UC Riverside.

11. Synchronous state machines: Good fit for fall detection
Normal behavior
Rapid descent
Extended lay
Normal behavior
Fall suspected
Fall detected
Rapid descent
Extended lay
Not laying
hadFall = 0
hadFall = 1
Descent velocity
A fall is characterized by a person beginning with normal behaviors, such as sweeping, followed by the person rapidly descending then laying on the ground for an extended period of time. A fall can be described as a time-ordered sequence of behaviors or events. State machines excel at describing time-ordered behaviors. A state machine consists of a set of states with actions, a set of transitions with conditions, and an initial state.
(press next to display inputs and outputs) State machines may also contain a set of input and outputs. Synchronous state machines support time-interval behavior, such as the time-interval that differentiates a suspected fall from a detected fall.
hadFall
Sitting, standing, or laying
Copyright © 2014 Alex Edgcomb, UC Riverside.

12. Synchronous state machine (SynchSM) fall detection (1 of 3)
SynchSMs promote capturing specific, modular behavior.
MBR tracker
SynchSMs promote the capturing of specific, modular behaviors. The suspected fall event state machine captures the rapid descent of the person. Person orientation determines whether a person is standing, sitting, or laying. OK-to-lay determines whether the person is laying in a pre-defined safe place, such as a couch. Fall sense captures the high-level logic and outputs a likelihood that a fall occurred.
Copyright © 2014 Alex Edgcomb, UC Riverside.

13. SynchSM fall detection (2 of 3)
The synchSMs together describe a fall detector. The fall detector converts an MBR into a fall likelihood score. A fall detector may be based on the height of the MBR, in particular the suspected fall event may be based on the height of the MBR.
Copyright © 2014 Alex Edgcomb, UC Riverside.

14. SynchSM fall detection (3 of 3)
We use two fall detectors per camera, one based on height and one based on width, and combine the fall likelihood scores into a single-camera fall score by averaging the fall likelihood scores.
Many single-camera fall scores are averaged to produce a multi-camera fall score. A camera may only contribute its single-camera fall score if a person is observed by that camera.
* Camera may only contribute single-camera fall score if a person is observed by that camera.
Copyright © 2014 Alex Edgcomb, UC Riverside.

15. SynchSM fall detection vs. state-of-the-art
MBR
MBR
3D proj.
Head
3D proj.
MBR
MBR
22 recordings from University of Montreal data set; each recording has multiple labels
Trained on 1 recording by selecting smallest threshold for sit-lay that got perfect accuracy
Tested with all combinations of remaining 21 videos
Did not use OK-to-lay SMs.
Sensitivity
# of cameras
SynchSM
Hung
Auvinet
Rougier
Anderson
Miaou
Thome 1
0.960 -
0.955
0.900
0.820 2
0.990
0.958
1.000
0.980 3
0.998
0.806 4
0.997 5
0.999 6 7 8
A dash (-) means unreported or not applicable, such as Hung’s algorithm that uses exactly two cameras.
Specificity
# of cameras
SynchSM
Hung
Auvinet
Rougier
Anderson
Miaou
Thome 1
0.995 -
0.964
0.860
0.980 2
1.000
0.938 3 4
0.998 5
0.993 6 7 8
I compared the synchSM fall detection approach to the state-of-the-art fall detectors. I compared using the commonly used data set from the University of Montreal, also used by Hung, Auvinet, and Rougier. Auvinet and Rougier’s lab developed that data set.
I trained on 1 recording by selecting the smallest threshold for sit-lay that got perfect accuracy. I tested with all combinations of the remaining 21 videos for the respective number of cameras. I did not use the OK-to-lay SM.
Fall detection accuracy is measured as two numbers: sensitivity and specificity. Sensitivity is the ratio of correct fall detections over actual falls. Specificity is the ratio of correct non-fall detections over actual non-falls.
The best performance per row is bold. Anderson has the highest sensitivity for 2 cameras, but at the trade-off of a very low specificity. Auvinet has the highest specificity for 4 and 5 cameras, but not significantly higher than the synchSM approach.
Copyright © 2014 Alex Edgcomb, UC Riverside.

16. Fall behavior coverage
MBR
MBR
3D proj.
Head
3D proj.
MBR
MBR
Fall behavior
SynchSM
Hung
Auvinet
Rougier
Anderson
Miaou
Thome
Suspected fall event Y
Person orientation
Fall sense
The accuracy of each algorithm may be due in part to that algorithm's coverage of a fall's typical time-ordered behaviors.
Hung, Auvinet, Anderson, and Thome did not consider a person’s downward movement.
Rougier and Miaou did not give the person time to get up, instead these algorithms immediately declare a fall if there is downward movement and the person is on the ground.
Did not consider sudden downward movement
Did not give time for person to get up
Copyright © 2014 Alex Edgcomb, UC Riverside.

17. Trade-off: Accuracy and efficiency (1 of 2)
Higher is better. Closest to top-right is best.
I conducted a trade-off analysis between the fall detection accuracy and computational efficiency.
A higher in each category is better, thus closest to top-right is best.
For accuracy, I used the combined accuracy score, which is the sensitivity times the specificity.
Combined accuracy score = sensitivity * specificity
Copyright © 2014 Alex Edgcomb, UC Riverside.

18. Trade-off: Accuracy and efficiency (2 of 2)
Higher is better. Closest to top-right is best.
Combined accuracy score = sensitivity * specificity
Copyright © 2014 Alex Edgcomb, UC Riverside.

19. SynchSM fall detection on other data sets
Ran synchSM fall detection on 55 of my own recordings (26 falls, 29 non-falls) using 1 and 2 cameras.
Perfect accuracy
Ran synchSM fall detection on hours of normal activity videos using 1 and 2 cameras.
I’ve also run the synchSM fall detector on my own recordings, including 26 falls and 29 non-falls. As well as, on the 22.5 hours of normal activity video.
Copyright © 2014 Alex Edgcomb, UC Riverside.

20. Copyright © 2014 Alex Edgcomb, UC Riverside.
With more resources, can synchSMs do better? Can head tracking improve fall detection?
With more resources, can synchSMs do better? Specifically, can head tracking improve fall detection?
Copyright © 2014 Alex Edgcomb, UC Riverside.

21. Fall detection accuracy: 2D/3D head tracking vs MBR tracking
87 video recordings, 1 min each19
69 non-confounding recordings (35 fall, 34 non-fall)
18 confounding recordings (5 fall, 13 non-fall)
Automated MBR: height, width, and top
Manual 2D head tracking by clicking on head
Manual 3D head tracking by estimating head height from ground
Same synchSMs. Suspected fall used feature.
Perfect accuracy with non-confounding scenarios
Head vertical position:
342 pixels
We compared 2D and 3D head-based fall detection to MBR-based fall detection.
We recorded 87 videos, including non-confounding and confounding recordings. An example of a confounding recording is looking under the couch for a dropped item or falling next to an upright vacuum that was just used.
We used the same automated MBR tracker and individually evaluated the MBR features of height, width, and top.
A research assistant performed manual 2D head tracking by click on the head, yielding the head vertical position feature.
Another research assistant performed manual 3D head tracking by estimating the head height from the ground.
The same synchSMs were used with the suspected fall state machine using a specified feature, such as the head vertical position or head height.
Each feature achieved perfect accuracy with the non-confounding scenarios.
Head height:
5.5 feet
19Edgcomb, A. and F. Vahid. Video-based fall detection dataset with 2D and 3D head tracking, and moving-region tracking. June 2014.
Copyright © 2014 Alex Edgcomb, UC Riverside.

22. Confounding recordings: Head tracking vs MBR tracking (1 of 2)
3D head
2D head
MBR top
MBR height
MBR width
Crouch with box Y
Kneel and move chair
Sit quickly
Sit then toss up item
Sit then hands to side
Hands up, down, then lay
Hands up, down, then sit 1
Hands up, down, then sit 2
Sit then hands up/down
Lay then toss up item
Hands to side then sit
Stand then toss up item
Set cushion on couch
Confused person orientation synchSM
Head tracking could tell that
head not near ground
Non-falls
The 3D head height yielded perfect accuracy for the non-fall confounding recordings. The 2D vertical head position misclassified two of the recordings, and the MBR features misclassified 5 recordings.
(Press next button)
Crouch with box and kneel and move chair confused the person orientation synchSMs.
The head tracking could tell that the head was not near the ground.
Copyright © 2014 Alex Edgcomb, UC Riverside.

23. Confounding recordings: Head tracking vs MBR tracking (2 of 2)
Falls
Summary
Confounding recordings
3D head
2D head
MBR top
MBR height
MBR width
Fall w/ vacuum 1 Y
Fall w/ vacuum 2
Put book in shelf
Look under couch
Take picture off wall
3D head
2D head
MBR top
MBR height
MBR width
Sensitivity
0.80
Specificity
1.00
0.85
0.54
Confused person orientation synchSM
Head tracking allowed rule that head had to be near the ground.
For the falls in the confounding recordings, each feature misclassified a fall with a vacuum in which the person fell toward the camera, causing a small MBR width, and the vacuum was just used, causing a large MBR height. So the person orientation synchSM thought the person was standing.
Overall, head tracking can improve fall detection by allowing a rule that the head had to be near the ground.
However, the MBR is suitable for a variety of scenarios. If confounding scenarios are likely, then head tracking may be justified.
MBR suitable for a variety of scenarios.
If confounding scenarios likely, then head tracking may be justified.
Copyright © 2014 Alex Edgcomb, UC Riverside.

24. Copyright © 2014 Alex Edgcomb, UC Riverside.
Outline
Elderly person falls and background work
SynchSM and moving-region-based fall detection
Other related work
Copyright © 2014 Alex Edgcomb, UC Riverside.

25. Assistive monitoring for the elderly
Assistive monitoring analyzes data from cameras and sensors for events and trends of interest.
I work in the area of assistive monitoring for the elderly. Assistive monitoring analyzes data from cameras and sensors for events of interest and trends of interest, and notifies the appropriate persons in response. An event of interest may be a person having fallen. A trend of interest may be a reduced amount of activity by a person.
Copyright © 2014 Alex Edgcomb, UC Riverside.

26. Commercial in-home assistive monitoring
In-home assistive monitoring companies typically monitor with anomaly detection. QuietCare uses motion sensors. BeClose also uses contact switches and light detectors, among others. Grandcare additionally provides if-then user programmability.
QuietCare (Intel and GE)
Motion sensor-based
anomaly detection
BeClose
Many sensor-based
anomaly detection
GrandCare
Many sensor-based anomaly detection
and if-then user programmability
Copyright © 2014 Alex Edgcomb, UC Riverside.

27. Master’s work – Monitoring and Notification Flow Language (MNFL)20,21
EasyNotify example
Spatial programming more intuitive than temporal. Under 8 mins to solve goal.
No compilation. Blocks are always executing, so users get instant feedback.
MNFL is a graphical language for lay-persons to describe an assistive monitoring system. A lay person is someone who can install a USB printer. On the left, the system sends a message if a person leaves at night but does not return for an extended period of time. I implemented MNFL as a web browser application, called EasyNotify, then I evaluated the efficacy of lay persons describing assistive monitoring systems with EasyNotify.
We found spatial programming to be more intuitive than temporal, as evidenced by lay-persons requiring under 8 minutes on average to solve a monitoring goal compared to CS10 students requiring weeks to learn temporal programming. The language also did not require users to compile, instead the blocks are always executing, so users got instant feedback.
20Edgcomb, A., and F. Vahid. Feature extractors for integration of cameras and sensors during end-user programming of assistive monitoring systems. In Proceedings of the 2nd Conference on Wireless Health, p. 13. ACM, 2011.
21Edgcomb, A., and F. Vahid. MNFL: the monitoring and notification flow language for assistive monitoring. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, pp ACM, 2012.
Copyright © 2014 Alex Edgcomb, UC Riverside.

28. Estimating daily energy expenditure from video for assistive monitoring22
Fidelity = correlation(video-based Calories, BodyBugg)
Energy expenditure levels on Monday
Actor
Fidelity 1
r = 0.996 2
r = 1.000 3
r = 0.983 4
r = 0.997
Combined
We considered 12 features
Motion in video not correlated with energy expenditure (r = -0.01, p = 0.53)
Horizontal acceleration had highest correlation (r = 0.80,p < 0.01)
Power regression had best fit (R2-value = 0.76) compared to linear, logarithmic, and exponential regressions.
Feature F
We developed a video-based algorithm that estimates a person’s daily energy expenditure. Many negative health conditions are correlated with low energy expenditure, such as increased likelihood of a fall and an earlier onset of dementia. The idea is to produce an easy to read graph for each day, so that a caregiver can glance at a week of graphs to determine whether intervention is warranted.
We used a commonly used body-worn energy estimation device as ground truth.
Our algorithm uses a feature F, standardizes that feature, sums the sequential absolute differences of the standardizations, then regresses the sum into a Calorie estimate.
We considered 11 features of the MBR, as well as motion in the video. Motion in the video was not correlated with energy expenditure due to noise caused by non-person movements, such as shadows. Horizontal acceleration had the highest correlation with energy expenditure. We also considered 4 regression models and found power regression to have the best fit to our training data.
A key expected use of energy estimation is to compare a person’s energy expenditure levels across many days, to detect negative trends and thus introduce interventions.
Fidelity is the correlation between the video-based energy estimation and the BodyBugg’s energy estimate. The fidelity was very close to the ideal.
We define accuracy as 1 minus the approximation error, which is the absolute difference between the expected value and the observed value divided by the expected value. Accuracy of 100% is ideal.
The average accuracy was 90.9%, which is about the same as our body-worn device compared to a respiratory chamber.
(Ideal is 1.0)
Accuracy = 1 - (|expected - observed|/expected)
Average accuracy = 90.9%
(about same as body-worn device)
22Edgcomb, A., and F. Vahid. Estimating Daily Energy Expenditure from Video for Assistive Monitoring, IEEE International Conference on Healthcare Informatics (ICHI), (to appear)
Data set available:
videoBasedEnergyEstimate.html
Copyright © 2014 Alex Edgcomb, UC Riverside.

29. Privacy perception and fall detection accuracy with privacy-enhanced video23
Privacy critical for adoption but makes events harder to detect
Does this style provide sufficient privacy for grandpa? Yes / No.
Did a fall occur? If so, at about what second in the video?
376 participants
Common privacy enhancements not providing sufficient privacy
Privacy is critical for the adoption of video-based monitoring systems, but makes events harder to detect. In this work, we explored the trade offs among participant’s privacy perceptions and the participant’s fall detection accuracy across raw and privacy-enhanced video. The oval has the best trade-off between privacy and accuracy.
(press button to reveal red text and box)
Blur and silhouette are common privacy enhancements but do not provide sufficient privacy. The reason is that blur and silhouette do not obscure the person’s actions, just the person’s identity, whereas oval and box obscure the actions and identity.
23Edgcomb, A., and F. Vahid. Privacy Perception and Fall Detection Accuracy for In-Home Video Assistive Monitoring with Privacy Enhancements, ACM SIGHIT (Special Interest Group on Health Informatics) Record, 2012.
Copyright © 2014 Alex Edgcomb, UC Riverside.

30. Copyright © 2014 Alex Edgcomb, UC Riverside.
Falls have a characteristic shape that is nearly identical for raw and privacy-enhanced video
A fall has a characteristic shape with some MBR features, such as the width of the MBR. That same characteristic shape is nearly identical for raw and privacy-enhanced video.
Copyright © 2014 Alex Edgcomb, UC Riverside.

31. Privacy-enhanced fall detection24 (1 of 2)
Dynamic time warping
Binary tree classification25
Observed shape
Characteristic
fall shape
Similarity
0.84
Non-fall
Fall
Observed shape
0.46
0.88
Script to produce this image provided by Professor Keogh, 2012.
That characteristic fall shape may be exploited by logical-shapelet classification, a technique developed in Professor Keogh’s. lab The technique uses dynamic time warping to compare an observed shape to a characteristic shape, which produces a similarity score. A binary tree classifier determines whether an observed shape receives the label: fall or non-fall. Each node in the tree contains a characteristic shape and a threshold value. The observed shape starts at the root node. If the observed and characteristic shape’s similarity score is above the threshold value, then we traverse right. Otherwise, we traverse left. The observed shape receives the label reached.
DTW established time series technique
24Edgcomb, A. and F. Vahid. Automated Fall Detection on Privacy-Enhanced Video. IEEE Engineering in Medicine and Biology Society, 2012.
25Mueen, A., E. Keogh and N. Young. Logical-shapelets: An Expressive Primitive for Time Series Classification. Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2011.
Copyright © 2014 Alex Edgcomb, UC Riverside.

32. Privacy-enhanced fall detection (2 of 2)
Privacy enhancement
Average sensitivity
Average specificity
Raw
0.91
0.92
Blur
1.00
0.67
Silhouette
0.75
Oval
Box
0.82
Binary tree classifier trained on raw video only
Evaluated using leave-one-out method
23 videos, 1 min each
Each video labeled fall or not-fall
The binary tree classifier was trained on raw video only, and each privacy enhancement was evaluated using the leave-one-out method.
Raw and oval video had the same accuracy. Furthermore, this automated approach was more accurate than human observers for the oval and box privacy enhancements.
A weakness is that this method does not consider the amount of time a person spends on the ground post-fall.
+More accurate fall detection than human observers.
-This method does not consider the time a person spends on the ground post-fall.
Copyright © 2014 Alex Edgcomb, UC Riverside.

33. Automated in-home assistive monitoring with privacy-enhanced video26
Already discussed
Energy trends
Fall detection
In room too long
Enter to left
Exit from left
In region too long
Arisen in morning
Person enters main living area
Abnormally inactive
Person home but inactive for extended period
This work evaluates the efficacy of common in-home monitoring goals with privacy-enhanced video.
A person inside a room, such as a study or laundry room, for an extended period of time may indicate a problem. One solution is to start a timer when a person enters a specified room. If the person does not leave before the timer runs out, then send a notification to the caregiver. From the picture in the middle, the specified room is located off-screen to the left. A person can be detected as entering the specified room by positioning the camera perpendicular to the room's entrance such that a person enters the room by leaving the camera's view to the left. Similarly, the person exits the specified room by entering the camera's view from the left. A similar solution can be used for detecting that a person has left home at night but not returned for an extended period of time.
A basic concern of a caregiver for an elderly person is whether that person arose in the morning. One solution is to send a notification to the caregiver when the first significant motion has been seen outside the bedroom between 6AM and 11AM. A person can be detected as arisen by positioning a camera in the main living area. The person's movement in the main living area will generate a significant amount of motion thus indicating the person has arisen. A similar technique can be used to determine that a person has not arisen in the morning.
A person being in a particular region, such as a hallway, for an extended period of time may indicate a problem. One solution is send a notification to the caregiver if the person is inside a specified region for over 10 minutes.
A person having an abnormally inactive day may indicate a problem. One solution is to notify the caregiver if the person is at home but has not moved for 3 hours. A person can be determined to be at home by positioning a camera perpendicular to the house's entrance/exit such that the person exits the house by leaving the camera's view to the left, and enters the house by entering the camera's view from the left. A person can be determined to not have moved for 3 hours by running a timer that is reset whenever a camera positioned in the main living area detects significant motion.
26Edgcomb, A. and F. Vahid. Automated In-Home Assistive Monitoring with Privacy-Enhanced Video, IEEE International Conference on Healthcare Informatics (ICHI), (to appear)
Copyright © 2014 Alex Edgcomb, UC Riverside.

34. Most goals were achieved equally well even with privacy enhancements
MNFL goals
Energy estimation fid./acc.
Fall detection sens./spec.
In room too long sens./ spec.
Arisen in morning sens. / spec.
In region too long sens. / spec.
Abnormally inactive during day sens./spec.
Raw
0.997 /
90.9%
0.91 /
0.92
1.0 / 1.0
Blur
0.994 /
80.5%
1.00 /
0.67
0.5 / 1.0
Silhouette
0.998 /
85.0%
0.75
Oval
85.6%
Box
1.000 /
84.3%
0.82 /
Trained on raw video only
MNFL goals trained on different person than tested
Most goals were achieved equally well even with privacy enhancements. The bolded values are privacy enhancement results that are lower than the raw video results.
Energy estimation and fall detection yielded the widest variation of results of the monitoring goals. For energy estimation accuracy, all the privacy enhancements performed worse than raw video. For fall detection, bounding-oval and raw video tied for the highest sensitivity and specificity. Of the remaining monitoring goals, the sensitivity and specificity were perfect 1.0, except blur video's sensitivity of 0.5 for in region too long.
Data set available:
Copyright © 2014 Alex Edgcomb, UC Riverside.

35. The background model tended to learn the blue much more than the actor
Current frame
Background model
Foreground and MBR
We expected raw and box video to have identical results, but raw tended to outperform box video. The cause is that the background model tended to learn the blue box much more than the actor. However, foregrounding still yields a good approximation of where the person is.
Copyright © 2014 Alex Edgcomb, UC Riverside.

36. Accurate and efficient algorithms that adapt to privacy-enhanced video27
Fall detection
Specific-color hunter
Energy estimation
Edge-void filler
I developed two techniques to mitigate the loss in goal performance by privacy-enhanced video: specific-color hunter and edge-void filler. Specific-color hunter was adapted if the moving region was mostly one color for a sufficient amount of time. Edge-void filler was adapted if the moving region was void of edges for a sufficient amount of time.
Specific-color hunter tends to improve goal performance for silhouette, oval, and box privacy enhancements, while edge-void filler tends to improve blur and box.
PE-aware chooses the optimal technique for a each privacy enhancement.
Fall detection performance improved from 0.86 sensitivity and 0.79 specificity to 0.92 and Energy estimation accuracy improved from 83.9% to 87.1%.
(Fall detection accuracy is higher because I updated that data set, which is available for use)
27Edgcomb, A. and F. Vahid. Accurate and Efficient Algorithms that Adapt to Privacy-Enhanced Video for Improved Assistive Monitoring, ACM Transactions on Management Information Systems (TMIS): Special Issue on Informatics for Smart Health and Wellbeing, 2013.
Copyright © 2014 Alex Edgcomb, UC Riverside.

37. Copyright © 2014 Alex Edgcomb, UC Riverside.
Contributions
MBR and synchSM-based fall detection is more accurate and efficient than all previous work.
MBR and head tracking are equally accurate, except in very specific cases.
Although monitoring goal accuracy degrades with privacy-enhanced video, adaptive algorithms can compensate without loosing computational efficiency.
The common privacy enhancements of silhouette and blur provide insufficient privacy, whereas a bounding oval or box were sufficient.
MBR and synchSM-based fall detection is more accurate and efficient than all previous work.
MBR and head tracking are equally accurate, except in very specific cases.
Although monitoring goal accuracy degrades with privacy-enhanced video, adaptive algorithms can compensate without loosing computational efficiency.
The common privacy enhancements of silhouette and blur provide insufficient privacy, whereas a bounding oval or box were sufficient.
Copyright © 2014 Alex Edgcomb, UC Riverside.

38. Summary of graduate research
Monitoring and notification flow language for assistive monitoring
Edgcomb, A., and F. Vahid. Feature extractors for integration of cameras and sensors during end-user programming of assistive monitoring systems. In Proceedings of the 2nd Conference on Wireless Health, p. 13. ACM, (2 pages)
Edgcomb, A., and F. Vahid. MNFL: the monitoring and notification flow language for assistive monitoring. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, pp ACM, 2012.
Estimating daily energy expenditure from video for assistive monitoring
Edgcomb, A., and F. Vahid. Estimating Daily Energy Expenditure from Video for Assistive Monitoring, IEEE International Conference on Healthcare Informatics (ICHI), (to appear)
Participant privacy perceptions and fall detection accuracy with privacy enhancements
Edgcomb, A., and F. Vahid. Privacy Perception and Fall Detection Accuracy for In-Home Video Assistive Monitoring with Privacy Enhancements, ACM SIGHIT (Special Interest Group on Health Informatics) Record, 2012.
Automated fall detection on video
Edgcomb, A. and F. Vahid. Automated Fall Detection on Privacy-Enhanced Video. IEEE Engineering in Medicine and Biology Society, (4 pages)
Edgcomb, A. and F. Vahid. Accurate and Efficient Video-based Fall Detection using Moving-Region and State Machines. (To be submitted)
Automated in-home assistive monitoring with privacy-enhanced video
Edgcomb, A. and F. Vahid. Automated In-Home Assistive Monitoring with Privacy-Enhanced Video, IEEE International Conference on Healthcare Informatics (ICHI), (to appear)
Edgcomb, A. and F. Vahid. Accurate and Efficient Algorithms that Adapt to Privacy-Enhanced Video for Improved Assistive Monitoring, ACM Transactions on Management Information Systems (TMIS): Special Issue on Informatics for Smart Health and Wellbeing, 2013.
Efficacy of digitally-enhanced education
Edgcomb, A. and F. Vahid. Interactive Web Activities for Online STEM Learning Materials, American Society for Engineering Education Pacific Southwest Section Conference, 2013.
Edgcomb, A. and F. Vahid. Effectiveness of Online Textbooks vs. Interactive Web-Native Content, Proceedings of ASEE Annual Conference, (to appear)
My graduate research so far has resulted in the following published papers.
Copyright © 2014 Alex Edgcomb, UC Riverside.