Adaptive Rehabilitation using Mixed-reality at Home: The ARM at Home study RIC Margaret Duff, Meghan Buell, and W. Zev Rymer Emory Steven Wolf and Aimee Reiss ASU Pavan Turaga, Nicole Lehrer, Michael Baran, Vinay Venkataraman, Loren Olson and Todd Ingalls CMU Thanassis Rikakis

Adaptive mixed reality rehabilitation  Computational assessment of movement  Abstract audio and visual feedback, evaluation and adaptation  Tangible sensing objects provide functional goals  Increase engagement and enhance motor learning through self-assessment of movement  Recover pre-morbid movement patterns and reduce compensation while increasing function

Completed at Banner Baywood Medical Center Evaluating outcomes of mixed reality compared to traditional therapy

AMRR improves function and kinematics  Both groups improved in the Wolf Motor Function Test  Every AMRR participant saw at least a 30% improvement in composite kinematic impairment measure (KIM), with a much more consistent distribution of improvement % change * AMRR group Control group

Issues to address  Neither group reported a significant change in impaired arm use / quality in ADLs  Long-term plan to both encourage functional and movement quality improvements  Continue therapy at home and with a greater variety of tasks

Scaling AMRR theories for home therapy

Home AMRR system  An engaging therapy environment at home  Task repetition, variability and intensity  Easy to use and understand in a largely unsupervised environment  Useful information (feedback) about task completion and movement quality

Feedback examples

Pilot study of unsupervised training  Test feasibility and effectiveness  Examine how people with stroke use and accept the system  Determine what further work is needed to accommodate the needs of the greatest percentage of people

Study protocol  1 week (3 sessions) of supervised training  4 weeks (12 sessions) of unsupervised training  Pre, post, and 4 week follow-up evaluations Participant demographics MedianRange Age (years) Months post stroke Fugl-Meyer (/66) N = 6 (6M, 0 F)

Wolf Motor Function Test Both FAS and time improve after therapy and are mostly retained at follow-up

Fugl-Meyer and Motor Activity Log FM scores improve after therapy and are retained at follow-up MAL scores are inconsistent after therapy and at follow-up

Kinematic results of trained task Velocity peak trained to about.6 m/s Inconsistent changes in horizontal trajectory

Participant acceptance of system

Preliminary outcomes  System was stable throughout  Successful unsupervised training  FM and WMFT improved after training  Kinematics and MAL were inconsistent

Current and future work  Improved hand function sensing  Track ADLs objectively and transfer therapy gains to everyday  Increased adaptability of therapy protocols  Better classification of movement impairments

Hand function sensing

More adaptive therapy protocols Current Protocol  Two set therapy tracks Future Considerations  Progression based on ability  Objects that vary more in complexity and weight  Variability within a set of reaches  Dissociate objects from table

Assessing the classifiers Problem - building high level metrics of efficiency for complex movements with reduced sensing Assessing new metrics for classifying movement  Correlation to kinematic assessment of simple tasks  Therapist ratings of simple to complex tasks, each rated in terms of overall performance and component performance  Components that are being trained on do not have one- to-one mapping with therapist ratings, which implies no supervised training data to build classifiers  But there is weak supervision !

Kinematic classifiers that drive feedback Curvedness – Measure of spatial error Too Fast / Too Slow – Measure of deviation in velocity profile Smoothness – Measure of variability in velocity profile

Cone grasp (simple, trained) Elevated touch (simple, semi-trained) Transport cylinder (complex, trained) Therapist rated tasks Video recordings of 3 tasks (5 trials each), performed once a week independent of therapy Treating therapist rated each reach, presented randomly

Rating system was developed to assign a score for each of the following: 1.Initial impression of overall trial (Modified FAS) 2.Trajectory 3.Compensation 4.Hand manipulation (grasp, touch) 5.Transport phase (if transport task) 6.Release phase (if transport task) 7.Final impression of overall trial (Modified FAS) If needed, explanation recorded if final impression is different than initial impression Therapist rated tasks

How does therapist rating help in tuning these classifiers? Assume a linear model of kinematic classifiers W1W1 W2W2 W3W3 W4W4 Cumulative Classifier Score

W 1 F 1 (T 1 ) + W 2 F 2 (T 2 ) + W 3 F 3 (T 3 ) + W 4 F 4 (T 4 ) + noise = Cost function = Use Nelder-Mead’s Simplex (fminsearch algorithm in MATLAB) to perform optimization Movement quality assessment Therapist Rating (R) {Initial impression of overall score}

Initial Results –3 participants (mild impairment) recorded at Emory –4 video recorded sessions each –Total of 55 reaches to grasp the cone

Initial Results Before optimization: Observe the overlap in score distributions, which implies classifiers are not tuned properly 45 Means are close

Initial Results Optimizing only the combination weights: The score distribution overlap does not get affected, suggesting that the problem really lies with the classifiers 45

Initial Results 45 Reduced overlap After optimization of weights and thresholds: score distribution overlap reduces

Conclusions  Changes in therapy protocols and tasks needed to benefit a larger subset of the population  Movement classifiers need to be generalized and improved, while staying accurate  Monitor and encourage transfer of therapy strategies to everyday life

Thank You!!!