Magic Pen: automatic pen-mode switching for document annotation

Abstract

Traditional digital pen interfaces use menu buttons to change pen modes, resulting in time and cognitive load spent on round-trips and potential errors from tapping small mode selection buttons. This thesis presents Magic Pen, a technique that automatically switches between digital pen modes. The Magic Pen system is driven by a Long Short-Term Memory (LSTM) model trained on pen data collected from nine participants and uses Transfer Learning (TL) to tune itself towards a user’s specific annotations iteratively. If Magic Pen chooses the incorrect mode, mitigation techniques incorporate flick gestures and screen taps to correct or remove a stroke. An annotation environment was also developed to rapidly prototype annotation systems with user-supplied documents. Magic Pen was originally evaluated with 18 participants and further evaluated with 8 participants. Magic Pen was preferred over a more conventional menu approach, and using TL allowed for greater model predictability and stability.