Perceptual and context aware interfaces on mobile devices

With an estimated 4.6 billion units in use, mobile phones have already become the most popular computing device in human history. Their portability and communication capabilities may revolutionize how people do their daily work and interact with other people in ways PCs have done during the past 30 years. Despite decades of experiences in creating modern WIMP (windows, icons, mouse, pointer) interfaces, our knowledge in building effective mobile interfaces is still limited, especially for emerging interaction modalities that are only available on mobile devices. This dissertation explores how emerging sensors on a mobile phone, such as the built-in camera, the microphone, the touch sensor and the GPS module can be leveraged to make everyday interactions easier and more efficient. We present studies and models to quantify the capabilities of these sensing channels, and show how effective interfaces in text entry, gaming, and CSCW can be built on mobile phones. The first such technology is TinyMotion. TinyMotion detects the movements of a mobile phone in real time by analyzing image sequences captured by its built-in camera, providing a usable analog pointing channel to existing mobile phone users. We quantified TinyMotion's human performance as a basic input control sensor. We found target acquisition tasks via TinyMotion follow Fitts' law, and Fitts' law parameters can be used for TinyMotion-based pointing performance measurements. We show that using camera phone as a handwriting capture device and performing large vocabulary, multilingual real time handwriting recognition on the mobile phone are feasible. Based on experiences and lessons learned from TinyMotion, this dissertation also introduces SHRIMP (Small Handheld Rapid Input with Motion and Prediction), a predictive mobile text input method runs on camera phones equipped with a standard 12-key keypad. SHRIMP maintains the speed advantage of Dictionary-Based Disambiguation (DBD) driven predictive text input while enabling the user to overcome collision and OOV problems seamlessly without explicit mode switching. Then, FingerSense is presented as another example of perceptual interface to enhance the expressiveness of physical buttons on space-constrained mobile devices. This dissertation also introduces a context-aware system named GLAZE (Generalized Location Aware ModelZ for End-users). GLAZE allows average user without any programming experiences, to create everyday location-aware applications directly on their mobile phones. Last, this thesis describes the design, implementation and evaluation of Event Maps, a web-based calendaring system targeted at improving the experience of attending and organizing large, multi-track conferences on both desktop computers and mobile devices. Event Maps has been successfully deployed in multiple large, real world conferences.