1. Addressing Stress and Addictive Behavior in the Natural Environment Using AutoSense Santosh Kumar Computer Science, University of Memphis

2. Our Team Behavioral Science Engineering 4/30/2015Santosh Kumar, University of Memphis2  Dr. Mustafa al’Absi, UMN  Dr. J Gayle Beck, Memphis  Dr. David Epstein, NIDA, NIH  Dr. Tom Kamarck, Pittsburgh  Dr. Satish Kedia, Memphis  Dr. Kenzie Preston, NIDA, NIH  Dr. Marcia Scott, NIAAA, NIH  Dr. Saul Shiffman, Pittsburgh  Dr. Annie Umbricht, Johns Hopkins  Dr. Kenneth Ward, Memphis  Dr. Larry Wittmers, UMN  Dr. Anind Dey, CMU  Dr. Emre Ertin, Ohio State  Dr. Deepak Ganesan, UMass  Dr. Greg Pottie, UCLA  Dr. Justin Romberg, Georgia Tech  Dr. Dan Siewiorek, CMU  Dr. Asim Smailagic, CMU  Dr. Mani Srivastava, UCLA  Dr. Linda Tempelman, Giner Inc.  Dr. Jun Xu, Georgia Tech

3. Students & Postdocs Memphis CMU, OSU, UCLA, Georgia Tech., UMN 4/30/2015Santosh Kumar, University of Memphis3  Dr. Andrew Raij (now at USF)  Dr. Kurt Plarre  Dr. Karen Hovsepian  Amin Ahsan Ali  Santanu Guha  Monowar Hussain  Somnath Mitra  Mahbub Rahman  Sudip Vhaduri  Dr. Motohiro Nakajima, UMN  Patrick Blitz, CMU  Brian French, CMU  Scott Frisk, CMU  Nan Hua, Georgia Tech  Taewoo Kwon, OSU  Moaj Mustang, UMass  Siddharth Shah, OSU  Nathan Stohs, OSU

4. Paradigm Shift in Disease Prevalence 4/30/2015Santosh Kumar, University of Memphis4  Infectious diseases, and those from poor hygiene & nutrition not as prevalent  They are replaced by diseases of slow accumulation  Heart diseases  Cancer, Ulcer  Depression, Migraine

5. Growing Epidemic – Stress & Addiction 4/30/2015Santosh Kumar, University of Memphis5  Stress & addictive behavior lead to or worsen diseases of slow accumulation  Stress: headaches, fatigue, heart failures, hypertension, depression, addiction, anxiety, rage  Smoking: cancer, lung diseases, heart diseases  Yet, both continue to be widespread  Stress: 43% adults suffer adverse health effects  Smoking: responsible for 20% of deaths in US  An urgency to help individuals reduce stress & abstain from addictive behavior

6. Addressing Stress & Addiction 4/30/2015Santosh Kumar, University of Memphis6  An unobtrusively wearable sensor suite called AutoSense  So, individuals can wear it in natural environment  Robust inference of stress from physiological measures  Automatically measure physiological and psychological stress  Automatic inference of addictive behaviors  Smoking, drinking, drug usage from sensor measurements  Detect addiction urges to provide timely intervention  Craving for smoking and drug usage  Contexts/cues that may lead to craving and eventual relapse  Infer other moderating behavioral & social contexts  Conversation, physical activity, traffic stressors, etc.

7. Outline 4/30/2015Santosh Kumar, University of Memphis7  Hardware and Software Platforms  AutoSense sensor suite  FieldStream mobile phone framework  Inferring Stress  Detecting stress from physiology  Predicting perceived stress  Ongoing User Studies  Detecting smoking, drinking, craving, drug usage, etc.  Roadmap & Long-term Vision

8. AutoSense Wearable Sensor Suite 4/30/2015Santosh Kumar, University of Memphis8 Chestband sensors: ECG, Respiration, GSR, Ambient & Skin Temp., Accelerometer Armband sensors: Alcohol (WrisTAS), GSR, Temp., Accelerometer Android G1 Smart Phone

9. Key Features of AutoSense Hardware 4/30/2015Santosh Kumar, University of Memphis9  Ultra low power  Six sensors (ECG, GSR, Resp., Temp, Accel) consume 1.75 mA  Overall current consumption < 3mA (for 10+ days of lifetime)  Sampling and transmission of 132 samples/sec (i.e., 1.8 kbps)  Reliable radio  ANT with integrated quality of service and duty cycling  Reliable and timely wireless transmissions in crowding scenarios  Antenna impedance is matched for human body  Power loss reduced from 33% (for free space configuration) to 0.1%  Operates at 2480-2524 MHZ band to be immune to Wi-Fi  Average packet loss rate of 0.57% even when Wi-Fi activity is intense

10. FieldStream – Mobile Phone Framework 4/30/2015Santosh Kumar, University of Memphis10  For use in conducting scientific user studies  In both supervised lab settings and in uncontrolled field settings  It collects measurements  Sensor measurements from wearable and phone sensors  Self-reports from subjects  Computes tens of features and various statistics over them (e.g., HR, HRV, RR, Minute Ventilation)  Makes inferences using machine learning algorithms  Stress, posture, activity, conversation, and commuting  Detects sensor detachments and loosening  Is reconfigurable  So, no need for change in source code for use in a new user study

11. 4/30/2015Santosh Kumar, University of Memphis11 Converts stream of sensor measurements into packets & delivers to intended recipient Provides a common interface to all sensors & populates buffers for feature computation Computes base features (e.g., R-R interval) & statistics over them

12. Deployment Experiences and Findings 4/30/2015Santosh Kumar, University of Memphis12  21 subjects in UMN - completed  Lab session on stress; 10-14 hours per day for 2 days in field  36 subjects in Memphis - completed  3 consecutive days in field with daily visits to the lab Some findings on human behaviors in our subject pool  Stress occurrence in daily life ( Plarre et. al., in ACM IPSN’11 )  Subjects were psychologically stressed 26-28% of time  Natural conversations ( Rahman et. al., in ACM Wireless Health’11 )  Frequency of conversations : 3 per hour  Avg. duration of a conversation: 3.82 minutes  Avg. Time between conversations: 13.3 minutes

13. Outline 4/30/2015Santosh Kumar, University of Memphis13  Hardware and Software Platforms  AutoSense sensor suite  FieldStream mobile phone framework  Inferring Stress  Detecting stress from physiology  Predicting perceived stress  Ongoing User Studies  Detecting smoking, drinking, craving, drug usage, etc.  Roadmap & Long-term Vision

14. Measuring Stress in the Field  Self-reports have been used for a long time  Questionnaires or surveys  Measures perceived stress  Strengths and limitations  (+) Captures detailed information  (+) Proximal predictor of mental health  (-) Distal predictor of physical health  (-) Discrete sampling  (-) Burden to participant  Need an automated approach for continuous stress measurement in the field 14Santosh Kumar, University of Memphis4/30/2015

15. Continuous Measure of Stress 4/30/2015Santosh Kumar, University of Memphis15  Can use physiological measurements to assess stress, but  Physiology is affected by several factors, not only stress  How to map physiology to psychology?  Activity, change in posture, speaking, food, caffeine, drink, etc.  How to separate out the changes in physiology due to stress?

16. The Quest for Automated Stress Measure  Predicting psychological state from physiology  William James – pioneering work (1880)  John Cacioppo and others – revitalized interest (1990)  Several studies on emotion and stress prediction  Identified physiological markers of stress and emotion Example: Heart rate, skin conductance response  But, confined to controlled settings  Few studies in uncontrolled environments  M. Myrtek’96, J. Healey’05, J. Healey’10,  Either no validated stressors, no lab session to train models, not able to account for confounders, or tried to match self- reports directly 16Santosh Kumar, University of Memphis4/30/2015

17. In the AutoSense Project  We developed a new wearable sensor suite  Conducted a scientific study with validated stress protocol  21 participants, 2 hour lab study, 2 day field study  Protocol designed by behavioral scientists  Stressors used are validated and known to produce stress  Self-reports designed by expert behavioral scientists  Developed new stress models to measure  Physiological response to stress  To measure adverse physiological effects of stress  Perception of stress in mind  To derive a continuous rating of perceived stress 17Santosh Kumar, University of Memphis4/30/2015

18. Lab Study – Stress Protocol  2 hour lab session  Subjects exposed to three types of stressors  Public speaking – psychosocial stress  Mental arithmetic – mental load  Cold pressor – physical stress  Physiological signals recorded at all times  Using AutoSense  Also, collected self-reported stress rating 14 times 18 Baseline 10 Min10 4444444 4555 Recovery Public Speaking Cold Pressor StartEnd Mental Arithmetic Santosh Kumar, University of Memphis4/30/2015

19. Self-Report Measures of Stress  Self-report questions related to affective state Santosh Kumar, University of Memphis19 QuestionPossible AnswerCode Cheerful?YES yes no NO3 2 1 0 Happy?YES yes no NO3 2 1 0 Frustrated/Angry?YES yes no NO0 1 2 3 Nervous/Stressed?YES yes no NO0 1 2 3 Sad?YES yes no NO0 1 2 3 4/30/2015

20. Santosh Kumar, University of Memphis20

21. Our Aproach 4/30/2015Santosh Kumar, University of Memphis21

22. Identified 22 Features from Respiration 22 Inhalation Duration Exhalation Duration Respiration Duration Stretch Mean Median Quartile Deviation 80 th Percentile Breathing Rate Minute Ventilation Insp./Exp. Ratio Basic FeaturesStatistical Features Santosh Kumar, University of Memphis4/30/2015

23. Computed 13 Features from ECG 23 Mean Median Quartile Deviation 80 th Percentile RR Intervals Power in low/medium/high frequency bands Ratio of low frequency/high power Variance RSA Basic FeaturesStatistical Features Santosh Kumar, University of Memphis4/30/2015

24. Feature and Classifier Selection  Used Weka for Training  Evaluated Decision Tree, DT with Adaboost, and Support Vector Machine  Using 10-fold cross validation, and training/test data  Classification results using 35 features  After feature selection, 13 features  8 Respiration, 5 ECG 24 J48 Decision Tree J48 with Adaboost SVM 87.67%90.17%89.17% Santosh Kumar, University of Memphis4/30/2015

25. Classification Accuracy on Lab Data 25Santosh Kumar, University of Memphis4/30/2015

26. Our Aproach 4/30/2015Santosh Kumar, University of Memphis26

27. Perceived Stress Model  Use a binary Hidden Markov Model  To reduce number of parameters, we approximate by   models the gradual decay of stress with time   models the accumulation of stress in mind due to repeated exposures to stress  Both  and  are person dependent and are learned from self-reported ratings of stress 27Santosh Kumar, University of Memphis4/30/2015

28. Evaluation of the Model (on Lab Data)  Correlation of perceived stress model and self-report rating in the lab session  Over 21 participants  Median correlation  0.72 28Santosh Kumar, University of Memphis4/30/2015

29. Field Study Protocol  Participants wore AutoSense continuously for 2 days  Going about their daily life (home, school, etc.)  Except when sleeping at night  Field self-reports  Participants responded to self-reports 20+ times each day  Same questions about affect state as in the lab  Additional context information  Additional behaviors automatically collected  Speaking, from respiration patterns  Physical activity, from accelerometer 29Santosh Kumar, University of Memphis4/30/2015

30. Realities of Natural Environment 30  Data eliminated  37% affected by activity  30% by bad quality  Less than 4 min consecutive data  4 subjects missing data or self-report Santosh Kumar, University of Memphis4/30/2015

31. Evaluation of the Model (Field) 31  Compared average stress ratings over both days  Accumulation model versus self- report  Linear interpolation Santosh Kumar, University of Memphis4/30/2015

32. Outline 4/30/2015Santosh Kumar, University of Memphis32  Hardware and Software Platforms  AutoSense sensor suite  FieldStream mobile phone framework  Inferring Stress  Detecting stress from physiology  Predicting perceived stress  Ongoing User Studies  Detecting smoking, drinking, craving, drug usage, etc.  Roadmap & Long-term Vision

33. Ongoing User Studies 4/30/2015Santosh Kumar, University of Memphis33  Memphis Study  40 daily smokers and social drinkers  A lab study followed by one week in the field  Stress, drinking, smoking, and craving for cigarettes marked  National Institute on Drug Abuse (NIDA) Study  20 drug addicts undergoing treatment  Two lab sessions and 4 weeks in field  Smoking, craving, and stress marked in lab;  Craving, stress, and drug usage reported in the field  Johns Hopkins Study  10 drug addicts in residential treatment  Drug injection in lab, daily behaviors marked in the field  To develop detectors for smoking, craving, and drug usage

34. Roadmap 4/30/2015Santosh Kumar, University of Memphis34  The near-term goal is to develop personalized stress and addiction assistants on the mobile phone to  Help reduce stress, e.g., least stressful route for driving  Break addiction urges where and when they occur  But, these applications will impact someone’s health  Will it indeed be helpful to each user and not hurt anyone?  Will it help maintain healthy behaviors even after the novelty phase?  How do we generate evidence for its validity, efficacy, safety?  Within reasonable time and effort, unlike multiyear RCTs  How do we design it so it has greater chance of success?  Various theories exist (e.g., stages of change, social cognitive theory)  But, no overall theory for designing adaptive interventions exist today

35. Long-term Vision 4/30/2015Santosh Kumar, University of Memphis35  Use these experiences to discover the scientific principles that can be used broadly in mobile health (mHealth)  To design and develop  New mHealth measures that are robust enough for field usage  New mHealth treatments and interventions that work  To generate evidence of validity, efficacy, and safety of mHealth Contribute to the newly emerging science of mHealth

36. Further Reading 4/30/2015Santosh Kumar, University of Memphis36 1. E. Ertin, N. Stohs, S. Kumar, A. Raij, M. al'Absi, T.Kwon, S. Mitra, Siddharth Shah, and J. W. Jeong, “AutoSense: Unobtrusively Wearable Sensor Suite for Inferencing of Onset, Causality, and Consequences of Stress in the Field,” ACM SenSys, 2011. 2. Md. Mahbubur Rahman, Amin Ahsan Ali, Kurt Plarre, Mustafa al'Absi, Emre Ertin, and Santosh Kumar, “mConverse: Inferring Conversation Episodes from Respiratory Measurements Collected in the Field,” ACM Wireless Health, 2011. 3. Mohamed Mustang, Andrew Raij, Deepak Ganesan, Santosh Kumar and Saul Shiffman, “Exploring Micro-Incentive Strategies for Participant Compensation in High Burden Studies,” to appear in ACM UbiComp, 2011. 4. K. Plarre, A. Raij, M. Hossain, A. Ali, M. Nakajima, M. al'Absi, E. Ertin, T. Kamarck, S. Kumar, M. Scott, D. Siewiorek, A. Smailagic, and L. Wittmers, “Continuous Inference of Psychological Stress from Sensory Measurements Collected in the Natural Environment,” ACM IPSN, 2011. 5. Andrew Raij, Animikh Ghosh, Santosh Kumar and Mani Srivastava, “Privacy Risks Emerging from the Adoption of Inoccuous Wearable Sensors in the Mobile Environment,” In ACM CHI, 2011. Nominated for best paper award

37. Outline 4/30/2015Santosh Kumar, University of Memphis37  Hardware and Software Platforms  AutoSense sensor suite  mStress mobile phone framework  Inferring Stress  Detecting stress from physiology  Predicting perceived stress  Ongoing User Studies  Detecting smoking, drinking, craving, drug usage, etc.  Privacy Issues in mHealth research

38. Behavior Revelation from Sensors 4/30/2015Santosh Kumar, University of Memphis38  Accelerometer & gyroscopes can be used to monitor activity level  Can infer movement pattern and place from these sensors  See SenSys’10 paper on AutoWitness  Could also infer epileptic seizures  Respiration sensor can be used for activity monitoring or estimating the extent of pollution exposure  Can use it to infer conversation, smoking, and stress  Inferring of public speaking episodes could even pinpoint the identity of the subject  Development of other behavioral inferences in progress

39. How Concerned are Study Participants? 4/30/2015Santosh Kumar, University of Memphis39  Conducted a 66 subject (36 in NS) study  Evaluated their concern level as their personal stake in the data is increased  Also, how their concern level changes as modalities are added/removed

40. Awareness & Concern 4/30/2015Santosh Kumar, University of Memphis40  Sharing of stress, commuting, and conversation generate higher concern than the sharing of place

41. Effect of Privacy Transformations 4/30/2015Santosh Kumar, University of Memphis41  Disassociating time is more critical than disassociating place of occurrance  Even reducing timestamp to duration helps