Text Entry by Inference: Eye Typing, Stenography, and Understanding Context of Use

In this project we have investigated how to use machine learning and other AI approaches to improve text entry and control of computer systems. We have investigated dwell-free eye-typing and found that is has the potential to be twice as fast as regular dwell-based eye-typing. (Proc. ETRA 2012; Best Paper Honourable Mention). We have also investigated how to create error correction interfaces for speech recognition.

We have also devised a new way to perform voice-only correction we call ‘one-step’ correction. Using our method one can correct speech recognition errors by merely speak the correction interface and there is no need for the user to first ‘select’ erroneous text. The system automatically infers the location of incorrect text and replaces it with a spoken correction (Proc. SLT 2010). We have used the same algorithm to create a system that fuses gesture and speech interaction. It enables users to enter text by speaking, gesturing, or a combination of both. Our system automatically fuses both input modalities and generates the most likely result (Proc. Interspeech 2011).

Another issue in intelligent text entry is the underlying language model. A language model assigns probabilities to word sequences. For a language model to be useful it needs to be trained on appropriate text data. A predictive text entry system is limited by the predictive power of the underlying language model. This problem of creating appropriate language models is particularly acute for predictive Augmentative and Alternative Communication (AAC) devices that predict text that motor-disabled non-speaking individuals want to communicate. This lack of efficient language models due to lack of representative data has been a long-standing problem in the AAC field for over 25 years. We invented a new method to create efficient language models for AAC using a combination of crowdsourcing and intelligent mining of social media (Twitter and blog data). Using our new method we could create efficient language models for AAC that outperform existing models (Proc. EMNLP 2011; article in New Scientist in February 2012). We later used our language model to enable an illiterate AAC user to communicate on her own for the first time (Proc. SLPAT 2012; Proc. ASSETS 2012; ACM SIGACCESS Best Student Paper Award). Our new method and models are now actively used in the AAC field.

The language models were also used (in combination with an error correction model) in a paper that investigated how to support efficient text entry on touchscreen tablets (Proc. CHI 2013).

Two surveys on text entry and intelligent interaction have also been produced (one ‘Research Highlight’ in Communications of the ACM and one survey article in Foundations and Trends in Human-Computer Interaction.

Understanding gesture interaction is often crucial for efficient intelligent interaction. We have investigated the memorability of gestures and found that in a series of three experiments self-defined gestures were significantly easier to remember than pre-designed gestures, even when one takes controls for training time (Proc. CHI 2013).

Finally, we have explored how to leverage proxemics to design new intelligent interactive systems (Proc. IUI 2013; Pervasive and Mobile Computing 2013; Ext. Abstracts CHI 2013).

To help the text entry field progress we have also acted as the lead organiser for the text entry workshops at CHI 2012 and CHI 2013.