Accelerometer wear and non-wear Classification using an Ensemble of Unsupervised Predictors Madalina Fiterau mfiterau@cs.stanford.edu Computer Science Department, Mobilize Center Jennifer Hicks jenhicks@stanford.edu Scott Delp delp@stanford.edu Bioengineering Department, Mobilize Center Manisha Desai manishad@stanford.edu Thomas Robinson tom.robinson@stanford.edu Jorge Armando Banda jabanda@stanford.edu Farish Haydel kfhaydel@stanford.edu Kristopher I Kapphahn <kikapp@stanford.edu> Manoj Kumar Sharma <manojs@stanford.edu> Hyatt Errol Moore IV <hyatt4@stanford.edu> Stanford University, School of Medicine PHYSICAL ACTIVITY MONITORING ENSEMBLE FOR WEAR DETECTION RESULTS Important for understanding wide array of health problems Obesity, diabetes, heart disease Cancers, osteoporosis, depression Typically performed with wrist and/or hip worn accelerometers Challenges The device is not worn all the time Non-wear looks similar to sleep and sedentary behavior No ground truth on wear time Meaningful analysis depends on unsupervised weartime detection Stanford GOALS dataset[1] 260 subjects, 7-11 year old chidren affected by obesity Triaxial ActiGraph GT3X+ (40Hz), collected 24-hours for 7 days Physical, physiological, behavioral and psychosocial measures States of activity and sleep are not known in this data set Objective: determine the best algorithms to use to discern non-wear/wear particularly for the Stanford GOALS data EXPERT PREDICTIONS ACCURACY ESTIMATION Each algorithm will be used as a base classifier in the ensemble The accuracy of each method will be estimated in an unsupervised way Prediction IS weighted according to experts’ estimated accuracy Different parametrizations of the same algorithm can be included in the ensemble as separate experts Key idea: Estimate agreement over sets of experts, write equations/inequalities linking consistency to accuracy, solve system of equations to obtain accuracy estimates Algorithm Mean (||Estimated - Actual Error||) over subjects Sleep Study Simulated Data Choi 0.0514 0.0107 0.0502 0.0104 NhanesX 0.0094 0.0469 0.0100 NhanesY 0.0527 0.0120 0.0479 0.0125 NahnesZ 0.0494 0.0131 0.0466 NhanesS 0.0498 0.0110 0.0532 0.0136 PadacoX 0.0116 0.0508 0.0126 PadacoY 0.0478 0.0114 0.0492 0.0109 PadacoZ 0.0489 0.0497 0.0128 PVecMag 0.0485 0.0121 0.0545 GMM 0.0512 0.0102 0.0519 METHOD OF MOMENTS FOR ERROR ESTIMATION TOP PERFORMING EXPERTS Agreement between experts i and j E{i} is the error event for expert i A{i,j} is the agreement between i and j a{i,j} = P(E{i}∩E{j}) + P(Ē{i}∩Ē{j}) a{i,j} = 1 – e{i} – e{j} +2e{i,j} Agreement between experts in a set A eA is the probability that all functions in A are wrong aA= P(∩i∈AE{i}) + P(∩i∈AĒ {i}) aA= eA + 1 + ∑I⊆A (-1)|I| eI Objective: minimize dependence between error rates c(e) = ∑A:|A|≥2 (eA - Πi∈Aei)2 subject to eA ≤ mini∈A eA\i Percentage of correctly identified top performing experts Sleep Study Simulated Data Top1 67.74 66.67 Top2 90.32 90 Top3 70.97 70 Top4 100 Top5 WEARTIME ACCURACY Accuracy averaged over subjects Sleep Study Simulated Data Vote 0.3938 0.1548 0.3830 0.1452 Best estimated 0.7464 0.2855 0.7386 0.2871 Best (oracle) WEARTIME DETECTION ALGORITHMS NHANES[2] Based on sequences of 0 counts Applied to each axis separately R ‘accelerometry’ CHOI[3] Uses vector magnitude R ‘Physical Activity’ PADACO[4] Uses smoothing CONCLUSIONS EVALUATION DATA There is no universally best weartime detection algorithm. We can reasonably estimate the accuracy of each individual expert based on agreement using the method of moments. We are able to select the best classifier for each subject. The original data do not have labels Data from a sleep study is used in the evaluation Ground truth is available 31 subjects, 12 hours of monitoring Each algorithm performs better in certain cases We estimate expert accuracy for each subject separately Simulated data based on Stanford GOALS, with added periods of non-wear sampled from known non-wear in Sleep Data References: [1] Robinson et al. Family, Community and Clinic Collaboration to Treat Overweight and Obese Children: Stanford GOALS. Contemporary clinical trials. 2013. [2] http://riskfactor.cancer.gov/tools/nhanes_pam [3] Choi, Leena, et al. "Validation of accelerometer wear and nonwear time classification algorithm." Medicine and science in sports and exercise, 2011 [4] http://web.stanford.edu/~hyatt4/pages/software_padaco