Pose Recognition using Kinect for Home Care System Zihang Huang CS2310 Project presentation

Goal Detect human being in a room using Kinect Segment each parts of the body, such as torso, hands, head Diagnose symptoms based on video stream Falling detection Identify hand gestures to decide data flow

System Overview 3D skeleton Diagnose Symptom frames OpenPose status ……. ……. ……. OpenPose

Component Diagram Falling Detection Index Cell M2 M3 Pose Estimation index cell Status Index Cell M0 M1 M4 Diagnose Symptoms Index Cell M5 Frame message(M0) is sent to pose estimation index cell generating 3D skeleton message(M1). Status index cell sends initial status(M2 and M5) to both falling detection index cell and diagnose symptoms index cell. Falling detection index cell will return detection result(M3) back to status index cell. When diagnose symptoms index cell detects symptom, it will send symptom message(M5) to status index cell updating current status.

Emergency Officer(EO) Diagnose Symptoms T1 T2 Emergency Officer(EO) T3 P0 P5 P1 P2 P3 T4 T1: Headache T2: Heart Attack T3: Stomachache T4: Left Knee Pain T5: Right Knee Pain P0: initial status P1:Symptom diagnosed P2: Show symptom text P3: Send message to EO P4: Confirm gesture P5: message fetching P4 T5

Emergency Officer(EO) Falling Detection T1 Emergency Officer(EO) P0 P5 P1 P2 P3 T2 T1: Movement velocity per 10 frames T2: The distance between the middle point of the body and ground P4 P0: initial status P1: Falling detected P2: Show “falling detected” text P3: Send message to EO P4: Confirm gesture P5: message fetching

Headache demo -headache

Headache Demo -confirm

Headache Demo -cancel

Falling Demo

Falling Demo

Demo https://www.youtube.com/watch?v=xthx0zhqqA0

Conclusion So far, this system can detect five symptoms, headache, heart attack, stomachache, left and right knee pain. And each of components are able to run individually. A valid symptom detection requires a specific hand confirmation gesture which is a designed fist gesture. Walking around or jumping in the room will not affect falling detection.

Future Work Explore multiview application Improve detection accuracy and precision Integrate face emotion detection to diagnose symptoms Record symptom times, duration and frequency and apply deep learning to predict potential risk of diseases

Reference Paper: [1] Cao, Z., Simon, T., Wei, S. E., & Sheikh, Y. (2016). Realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1611.08050. [2] Simon, T., Joo, H., Matthews, I., & Sheikh, Y. (2017). Hand Keypoint Detection in Single Images using Multiview Bootstrapping. arXiv preprint arXiv:1704.07809. GitHub: OpenPose https://github.com/CMU-Perceptual-Computing-Lab/openpose OpenKinect https://github.com/OpenKinect/libfreenect2 OpenCV https://github.com/opencv/opencv My project repository https://github.com/derekwong66/pose-estimation-in-home-care-system