论文成果 / Publications

具身感（sense of embodiment）指人认为自己拥有、控制和处于一个身体的感觉，这个身体可能是自己，也可以是另外一个表示自己的虚拟对象。具身感起源于心理学中关于人的意识和身体关系的问题，随着虚拟现实的普及和元宇宙的兴起，使之成为一个需要研究的内容。 一个经典实验称之为“橡胶手错觉实验”，1998年发表在Nature的一篇论文中。实验很简单，一块挡板的两边有两只手，一只是被试者的真手，另外一只是橡胶手，实验员同时用毛刷刷两只手，当刷橡胶手，不刷真手时，被试者也会有被刷的感觉：甚至还做了比如拿东西去扎假橡胶手的实验时，真手也会有被扎的感觉，马上躲开。 这个心理学实验的一个有趣的结论是，虽然橡胶手是物理的假手不能动，也不逼真，但是用户对这只假橡胶手产生与他自身相关的感觉，现在称之为具身感的感觉。相比橡胶手，今天虚拟现实的交互中，提供用户的是一个可以同步其姿态、动作，甚至是表情等信息丰富的虚拟化身，因而自然而然产生一个问题，虚拟化身对用户具身感会产生什么影响？

We present DRG-Keyboard, a gesture keyboard enabled by dualIMU rings,allowing the user to swipe the thumb on the index fingertip to perform word gesture typing as if typing on a miniature OWERTY keyboard. With dual IMUs attached to the user's thumb and index finger, DRG-Keyboard can 1)measure the relative attitude while mapping it to the 2D fingertip coordinates and 2) detect the thumb's touch-down and touch-up events combining the relative attitude data and the synchronous frequency domain data,based on which a fingertip gesture keyboard canbe implemented. To understand users typing behavior on the index fingertip with DRG-Keyboard,we collected and analyzed user data in two typing manners.Based on the statistics of the gesture data,we enhanced the elastic matching algorithm withrigid pruning and distance measurement transform.The user study showed DRG-Keyboard achieved an input speed of 12.9WPM(68.3%of their gesture typing speed on the smartphone for all participants. The appending study also demonstrated the superiority of DRG-Keyboard for better form factors and wider usage scenarios. To sum up, DRG-Keyboard not only achieves good text entry speed merely on a tiny fingertip input surface, but is also well accepted by the participants for the input subtleness, accuracy, good haptic feedback, and availability.

Flipping is a potential interaction method for dual-display phones with front and rear screens. However, little is known about users’ phone flipping behaviors. To investigate it, we iteratively conduct three user studies in this research. We first elicit 36 flipping gestures from 22 users and present a design space according to the results. We then collect users’ flipping data and subjective evaluation of all user-defined gestures through the second user study. We design a flipping detection algorithm based on the data collected and deploy it on an off-the-shelf dual-display hone. Another evaluation study shows that it can detect users’ flipping efficiently with an average accuracy of 97.78%. Moreover, users prefer many flip-based applications on dual-display phones to existing non-flipping pproaches on regular single-screen phones. In conclusion, our work provides empirical support that flipping is an intuitive and promising input modality for dual-display phones and sheds light on its design implications.

An avatar mirroring the user’s movement is commonly adopted in Virtual Reality(VR). Maintaining the user-avatar movement consistency provides the user a sense of body ownership and thus an immersive experience. However, breaking this consistency can enable new interaction functionalities, such as pseudo haptic feedback or input augmentation, at the expense of immersion. We propose to quantify the probability of users noticing the movement inconsistency while the inconsistency amplitude is being enlarged, which aims to guide the intervention of the users’ sense of body ownership in VR. We applied angular offsets to the avatar’s shoulder and elbow joints and recorded whether the user identified the inconsistency through a series of three user studies and built a statistical model based on the results. Results show that the noticeability of movement inconsistency increases roughly quadratically with the enlargement of offsets and the offsets at two joints negatively affect the probability distributions of each other. Leveraging the model, we implemented a technique that amplifies the user’s arm movements with unnoticeable offsets and then evaluated implementations with different parameters(offset strength, offset distribution). Results show that the technique with medium-level and balanced-distributed offsets achieves the best overall performance. Finally, we demonstrated our model’s extendability in interventions in the sense of body ownership with three VR applications including stroke rehabilitation, action game and widget arrangement.

Gaze-based target sufers from low input precision and target occlusion. In this paper, we explored to leverage the continuous eyelid movement to support high-efcient and occlusion-robust dwellbased gaze pointing in virtual reality. We frst conducted two user studies to examine the users’ eyelid movement pattern both in unintentional and intentional conditions. The results proved the feasibility of leveraging intentional eyelid movement that was distinguishable with natural movements for input. We also tested the participants’ dwelling pattern for targets with diferent sizes and locations. Based on these results, we propose DEEP, a novel technique that enables the users to see through occlusions by controlling the aperture angle of their eyelids and dwell to select the targets with the help of a probabilistic input prediction model. Evaluation results showed that DEEP with dynamic depth and location selection incorporation signifcantly outperformed its static variants, as well as a naive dwelling baseline technique. Even for 100% occluded targets, it could achieve an average selection speed of 2.5s with an error rate of 2.3%.

We propose to explore teeth-clenching-based target selectionin Augmented Reality (AR), as the subtlety in the interaction can be beneficial to applications occupying the user's hand orthat are sensitive to social norms. To support the investigation, we implemented an EMG-based teeth-clenching detectionsystem (ClenchClick), where we adopted customized thresholds for different users. We first explored and compared the potential interaction design leveraging head movements and teeth clenching in combination. We finalized the interaction to take the form of a Point-and-Click manner with clenches as the confirmation mechanism. We evaluated the taskload and performance of ClenchClick by comparing it with two baseline methods in target selection tasks.Results showed that ClenchClick outperformed hand gestures in workload, physical load, accuracy and speed, and outperformed dwell in work load andl temporal load. Lastly, through user studies, we demonstrated the advantage of ClenchClick in real-world tasks, including efficient and accurate hands-free target selection, natural and unobtrusive interaction in public, and robust head gesture input.

Revisitation, the process of non-linearly returning to previously visited regions, is an important task in academic reading. However, listening to content on mobile phones via a screen reeader fails to support eyes-free revisiting due to its linear audio stream, ineffective text organization, and inaccessible interaction. To enhance the efficiency and experience of eyes-free revisiting, we identified visually impaired people's behaviorsand difficulties during text revisiting through a survey (N=37) and an observation study (N=12). We proposed a series of desiign guidelines targeting high precision, high flexibility, and low workload in interaction, and iteratively designed and deyveloped a reading prototype application. Our implementation supports dynamic text structure and is supplemented by bothlinear and non-linear layered text navigation. The evaluation results(N=8)show that compared to existing methods, our prototype improves the clarity of text understanding and fluency of revisiting with reduced workload.

Standard sleep quality assessment methods require custom hardware and professional observation, limiting the diagnosis of sleep disorders to specialized sleep clinics. In this work, we leverage the internal and external microphones present in active noise-cancelling earbuds to distinguish sounds associated with poor or disordered sleep, thereby enabling at-home continuous sleep sound monitoring. The sleep sounds our system is able to recognize include, but are not limited to, snoring. teeth grinding, and restless movement. We analyze the resulting dual-channel audio using a lightweight deep learning model built around a variation of the temporal shift module that has been optimized for audio. The model was designed to have a low memory and computational footprint, making it suitable to be run on a smartphone or the earbuds themselves. We evaluate our approach on a dataset of 8 sound categories generated from 20 participants. We achieve a classification accuracy of 91.0% and an F1-score of 0.845.