Making Touchscreen Keyboards Adaptive to Keys, Hand Postures, and Individuals – A Hierarchical Spatial Backoff Model Approach Ying Yin 1,2, Tom Ouyang 1, Kurt Partridge 1, and Shumin Zhai 1 1 Google Logo here 2 MIT Logo here

Foundations to current methods Language modeling – vocabulary – 1-gram, 2-gram … N gram frequencies Spatial models – converting input touch points into probabilities of letters Edit distance correction – assigning cost to insertion, deletion, and other spelling errors User and posture independent

Research questions One promising area for improvement is by making them adapt to the user – What types of adaption are possible? – How do they affect performance?

Contributions A novel hierarchical adaptive model Show benefits of posture and user adaptation Online posture classification method 13.2% reduction in character error rate – compared to base model – without language model

Types of adaptation Individual differences (cf. Findlater & Wobbrock, 2012) Furthermore, people use different hand postures to type (cf. Azenkot and Zhai, 2012)

Different typing postures: two thumbs, one finger, or one thumb

Types of adaptation Different postures  different touch patterns Touch patterns also depend on letter keys (Azenkot & Zhai, 2012) Need adaptation

Challenges of adaptation Complexity – three adaptive factors: key, posture, individual – large number of submodels – need sufficient data to build each submodel Model selection – wrong selection may hurt keyboard quality – uncertainty in posture classification

Hierarchical spatial backoff model (SBM)

Combinatorial and fine grained adaptation Conservative Does not require an extra training phase Updates the model continuously online

Research method “Pepper” dataset (Azenkot & Zhai, 2012) – 30 right-handed participants – given random phrases to type – between-subject: each person uses one posture – 84,292 touch points in total 10-fold cross validation

Comparison of spatial models

Two-thumb One-finger Effective key areas

Posture classification SVM-based classifier Based on correlation between time and distance between consecutive touch points – no additional sensors required – speed independent 86.4% accuracy Real-time

Posture adaptation

Individual adaptation

Prototype implementation of SBM 13.2% reduction in character error rate – compared to base model – without language model Integrated with real keyboard – combined with language model – runs on Android phone in real-time

Future work Weighted average of submodels instead of making binary decisions More data: real-use logging and game playing User studies – validate the accuracy and speed improvement – how users adapt their behavior to SBM Combine spatial and language models

Contributions A novel hierarchical adaptive model Show benefits of posture and user adaptation Online posture classification method Opens up many more interesting HCI questions

Q & A

Prototype implementation of SBM Posture & key adaptation models – supervised and batch learning Individual adaptation models – unsupervised and online learning Backs-off to more basic models when – posture estimation is uncertain (conf. < 0.94) – there is insufficient user data (< 50 data points)