Featured News

Latest Research

  • Lip-Interact: Improving Mobile Device Interaction with Silent Speech Commands

    (UIST 2018) Ke Sun,Chun Yu,Weinan Shi,Lan Liu,Yuanchun Shi
    We present Lip-Interact, an interaction technique that allows users to issue commands on their smartphone through silent speech. Lip-Interact repurposes the front camera to capture the user’s mouth movements and recognize the issued commands with an end-to-end deep learning model. Our system supports 44 commands for accessing both system-level functionalities (launching apps, changing system settings, and handling pop-up windows) and application-level functionalities (integrated operations for two apps). We verify the feasibility of Lip-Interact with three user experiments: evaluating the recognition accuracy, comparing with touch on input efficiency, and comparing with voiced commands with regards to personal privacy and social norms. We demonstrate that Lip-Interact can help users access functionality efficiently in one step, enable one-handed input when the other hand is occupied, and assist touch to make interactions more fluent.
    [Video] [Paper]

  • TOAST: Ten-Finger Eyes-Free Typing on Touchable Surfaces

    (Ubicomp 2018) Weinan Shi, Chun Yu,Xin Yi, Zhen Li, Yuanchun Shi
    Touch typing on flat surfaces (e.g. interactive tabletop) is challenging due to lack of tactile feedback and hand drifting. In this paper, we present TOAST, an eyes-free keyboard technique for enabling efficient touch typing on touch-sensitive surfaces.We first formalized the problem of keyboard parameter (e.g. location and size) estimation based on users’ typing data.
    [Video] [Paper]