Lynne Grewe, Steven Magaña-Zook CSUEB, A cyber-physical system for senior collapse detection

Seniors Falling Over 1/3rd of seniors above 65 fall each year Lead to serious injury and even death Falls account for 25% of all hospital admissions, and 40% of all nursing home admissions 40% of those admitted do not return to independent living; 25% die within a year. Fast medical attention can make a difference Many falls do not result in injuries, yet a large percentage of non-injured fallers (47%) cannot get up without assistance.

Cost of Falling? 2005, CDC study – Cost for Falls leading to fatality

Goal create a “smart home” system to predict and detect the falling of senior/geriatric participants in home environments More seniors living at home autonomously

SCD: Senior Collapse Detection Overview

SCD: uses Kinect Sensor Inexpensive, commercial, well tested, good API support Modalityexample 2D 3D Audio

Feature Extraction Perform Skeleton Tracking Ideal – fall indicators often involve joint locations and range of motion Good Resolution – 21 joints

Skeleton Tracking Has Noise Degrading performance with occlusion General Twitching Also degrades as more occlusion from being on floor << not bad << notice rear leg position problem from self occlusion

Noise Reduction: Physical Therapy Skeleton Model Use Physical Therapy Model data to determine normal range of motion and joint distances. Calculate joint certainty metric = f(joint angles, joint distances, physical therapy skeleton model) = 1 if within limits of model <1 non-linear function of deviation from model Currently use 1 model based on maximum ranges Future = model for different demographics (age, height, weight), or learned from user. Concept = Can use Joint Reliability to determine if a joint should be used in Fall Detection OR can use in determination of confidence of a Fall detected

What is a Fall? How can we detect it? SCD defines fall as “loss of control resulting in downward motion ending with body on floor” Previous work: Wearable devices: Accelerometers, gyroscopes, movement sensors Autonomous: 2D with mixed results 3D beginning work Detection Ideas Quick movement (acceleration) – whole or what part of body? Body Orientation – parallel to floor Location – little but, some looking at general location

SCD Fall Detectors Currently 3 based on all ideas (location, orientation and acceleration). Currently operate independently – any can trigger fall detection event

Location –need Floor Detection Uses 3D floor plane detected by Kinect Sensor One for each skeleton calculated Good News- Gamers want this accurate Ax + By + Cz + D = 0

SCD: Head Movement Detector Falling Detector / Idea: quick movement indicates falling Measure: both head joint velocity and trajectory (downward) and the head ends up near the floor. Buffer 1 second of data (30 frames / second) Trajectory – 2 slopes Empirically chosen Thresholds velocity>1ft/second Last frame of 1 second head position within 1.5 ft of floor Trajectory toward floor

SCD: Head Movement Detector – Reliability and Confidence Reliability: function (number tracked joints, number inferred joints) Confidence: function (velocity)

SCD: Horizontal Ratio Detector Fall Detector / Idea: senior lands on the floor in horizontal- parallel to floor orientation Concept = 3D bounding box 2 Ratios = Width/Height and Depth/Height Empirically chosen Threshold: 1.5 for either Ratio = elongated, parallel to floor Head Height Ratio FALL

SCD: On Floor Detector Fall Detector / Idea: senior lands on the floor Hip near floor Minimum number of joints near floor Empirically Chosen Thresholds Minimum 1 hip joint (out of 3 possible) Minimum 8 joints “near” floor “near” = 1.5 ft Reliability = #tracked / (#tracked + #inferred) = 0.25 threshold

How Many Falls? Some of our detectors are “Fallen” detectors Don’t want too many triggers for same fall Minimum time between fall events is set currently at 15 seconds. No data but, seemed fastest time between different falls Example:

Emergency Response Configure Emergency Contact(s) Phone – sms text

SCD: Speech Processing Use Microsoft SDK Text-To-Speech Use Microsoft SDK Speech Recognition Kinect has microphone array.

Fall Detection Event and Emergency Response System Senior Hears Audio Prompts from System – asking if assistance is needed. If Yes or No Response the predetermined emergency response is triggered Here you see both the Diagnostics GUI and an illustration of the final Audio

Examining Test case Head Motion Detector: FALL Trajectory = slope average was Head Position Last Frame = 1.37ft from floor Velocity = 1.003ft/sec On Floor Detector: FALL 9 joints near floor All 3 hip joints on floor Horizontal Ratio Detector: FALL W/H = 1.7, D/H = 0.89 Head Distance to Floor = 1.37ft from floor

Both Live and Semi-Automated Testing Have ability to cycle through sets of pre-recorded data Output to HTML results

SCD: RESULTS OnFloor Performs best 100% On Floor Detector Horizontal Detector Head Movement Detector True Positive Rate 24 out of 2421 out of 2423 out of 24 False Positive Count 022 True Negative Rate 14 out of 1412 out of 14 False Negative Count 031

Issues, Problems and Future work

Limitations with Kinect Limited depth range(solution: multiple Kinect) Occlusion (solution: multiple Kinect or use tilt feature of Kinect)

Issues Skeleton engine needs some number of frames to recognize when user enters frame. This is unavoidable with current concept of skeleton tracking Processing – on common commercial home use laptops and desktops ($ ) we experience a lag time when all diagnostics are being displayed from 1 to 20 seconds worst case to process frame leading to detection. Typical (little data) around seconds.

Future Work More Testing Combine Decisions? Learn Formulation? Fine Tune/ Learn Thresholds Improve Performance Speeds Other modules Fall prediction = gait tracking Post Fall detection = rolling, vocalizations Learning Individual Physical Model Multi-Kinect System calibration, sensor inference, coordinated communication and decision making Kinect 1 improvement in resolution.