OmniSense: Exploring Novel Input Sensing and Interaction Techniques on Mobile Device with an Omni-Directional Camera

At the near device level - the space proximate to the device contains detailed information about the user’s gripping hand and finger. For example, OmniSense can detect the grip handedness (which hand is holding the device or both hands) (Figure 1.1 & 1.4) [16] or the grip type (firm or loose). OmniSense can track the thumb hovering above the screen (Z-height, speed of tap) (Figure 1.7) [24] or the fingers on the back-of-device (back-tapping, swiping) (Figure 1.2) [37, 58]. OmniSense can also track the 3D location of the finger (Figure 1.5) [74] of the other hand, including the hovering state, pre- and post-touch gesture [14, 24], and speed of tap. OmniSense can also recognize which part of the finger (index, middle, knuckle) or whether a stylus (Figure 4 d) [21, 73] is being used to touch the screen.

At this level — the space around the device expands the interaction area beyond the screen’s boundaries which contains information about the user’s body. OmniSense can detect full body pose (Figure 1 f & 1 g) [9], which includes the body posture (standing or sitting) and state (walking or running), leg gesture (Figure 1.3) [3], along with head and gaze direction and facial features (expression, mood) (Figure 5 e). Similar to Surround-See [73], OmniSense can also track the non-gripping hand’s spatial movement around the device (gesture, hand pose, pointing direction) (Figure 1.8) [22, 79] or touching the face (Figure 1.9) [45]. Finally, the camera can measure the device’s spatial relationship to the body (proximity, orientation) (Figure 1.6) [13, 18, 73].

At this level — the extended, large area surrounding the device, including the environment, context and the surface where the device is resting can be sensed, which were explored in Surround-See [73]. For example, when held by a user who is on the move, OmniSense can detect various objects in front of the users, such as approaching cars, humans and obstacles (Figure 1 a). OmniSense can sense the context, such as which room (bedroom or lecture hall) [5, 73] or which vehicle (car or train) the user is currently residing in (Figure 1 b). Owing to the omni-directional field of view, it can also recognize landmark buildings (e.g., AR StreetView) or localize indoor positions even when the device is lowered. This does not require the users to raise the device for point-and-shoot, hence lower cognitive load while on the move. When the device is resting on a flat surface (e.g., a desk), or being mounted (e.g., phone charger), it can detect various objects (e.g., food, tableware, stationery, finger) (Figure 1 d & 1 h) or human actions (activity monitoring, in-car gesture) (Figure 1 c, 1 e & 1 i) [73] surrounding the device.

i) Adaptive UI — Knowing which hand is holding the phone allows for the placement of UI elements intelligently [16]. For example, a menu or virtual keyboard that resizes and adapts to handedness and grip automatically, as shown in Figure 4 (a).

ii) Ad-lib Interface — Tracking the thumb’s hover state and Z-height allow for an ad-lib interface [24]. For example, Figure 4 (b) demonstrates that the menu interface fades in when the thumb approaches and fades out when the thumb moves away, so as to not clutter the video content in a full-screen video player.

iii) Back-of-Device Interaction — Tracking the index finger on the back-of-device [6] and its 2D location allow for triggering smart actions. For example, tapping on the back of the device acts as shortcut to launch applications, sliding vertically pulls down the notification panel, while sliding horizontally on the back adjusts the volume continuously, as shown in Figure 4 (c).

iv) Seamless Mode Switching — Recognizing which finger or object is touching the screen (index, middle, knuckle or stylus) [21, 78] allows for seamless mode switching. For example, in a paint application, touching the screen with the index finger switches to a brush whereas the middle finger switches to an eraser. A touch by the knuckle calls out the color palette, while a stylus changes to a thin brush. All of these work seamlessly without requiring the user to open the menu and choose option, as shown in Figure 4 (d).

v) 3D Finger Interaction — Tracking the 3D location of the finger above the screen allows for various Air+Touch [14] and PreTouch [24] interaction. For example, users can preview a message by hovering their finger on top of it, as shown in Figure 4 (e). A slow tap selects an item normally while a fast tap deletes it. By manipulating the Z-height of the hovering finger, the user can control the scrolling speed of a list.

i) VR Video Avatar Controller — Tracking the full body pose and facial features allow for immersive interaction and experience (e.g., virtual avatar, virtual YouTuber). For example, in a video call or live streaming, users can control a 3D virtual avatar using full body movement and facial expression, as shown in Figure 5 (a).

ii) Face Touch Detection — Knowing the hand and body location allows for on-body interaction. For example, the hand touching the body parts or face can be detected [45] to enable mnemonic body shortcut [19] or discreet interaction [38]. It can also give a warning when users unconsciously touch their faces during a pandemic, as shown in Figure 5 (b).

iii) Around Body Interaction — Knowing the device’s spatial relationship to the user allows for around-body interaction [13, 73]. For example, users can vary the distance of the device to the body for zooming, or alter the position and orientation of the device relative to the body for panning a map, as shown in Figure 5 (c).

iv) Leg Gesture Interaction — When both hands are busy or dirty, the legs are free to perform various inputs. Tracking the user’s leg allows for leg gesture interaction [3, 46, 49]. A pie menu of multiple selections can be performed by stepping one leg forward, as shown in Figure 5 (d). Precise step counting is also possible.

v) Stay Awake and Shoulder Surfing Prevention — Tracking the user’s eye gaze allows for intelligently adjusting the screen content. For example, when the user is not looking at the screen, the screen can be dimmed automatically to conserve battery, as shown in Figure 5 (e). If the system detects any stranger other than the owner who is peeking at the screen (e.g., shoulder surfing), the system can automatically hide sensitive information on the screen and warn the user.

vi) Spatial Gestures — Tracking the other hand interacting around the device allows for spatial manipulation [22, 79] or pointing [73]. For example, in a game (e.g., Pokemon GO, basketball), a user can perform real hand-throwing gestures rather than swiping on the touchscreen, as shown in Figure 5 (f). Various shortcut commands [47], such as changing the hand sign (C to copy, V to paste, OK to enter) can also be enabled.

The user’s surroundings and context contain much information for realizing ubiquitous computing. Indeed, prior work such as Surround-See [73] have explored capabilities such as recognizing the device’s peripheral environment, objects and user activities in vicinity to the device. OmniSense covers a full 360° FoV that enables further unexplored use cases, such as:

i) Danger Detection — Understanding the surrounding allows for danger detection and prevention [23, 64]. For example, it is dangerous to keep looking at the phone screen and not paying attention to the road while on the move, commonly known as smartphone zombies. Since the camera has a full view of the environment, such that it can provide a warning to the user if it detects obstacles (an approaching car, pole, or human), as shown in Figure 6 (a). Especially during the pandemic era, it supports practicing social distancing where people are supposed to stay at least two meters away.

ii) Tabletop, Tablet, and Stylus — Detecting the objects surrounding the device when it is resting on a surface allows for various tabletop applications. For example, it can enable various board games or tangible interaction using real-world objects [52] for input. In a kitchen scenario, a cookpad application recognizes types of food in a bowl and estimates the calorie amount, while also suggesting recipes. Moving a mug into position sets a timer, which the duration is based on its location, as shown in Figure 6 (b). In addition, it can track the user’s finger and allow multi-touch interaction on a large surface [8, 36]. The color of a stylus can be automatically detected for changing the virtual brush’s color, while hand pose can be recognized to draw different shapes (fist for circle, palm for square), as shown in Figure 6 (c) [47].

iii) Context Awareness — Understanding the surrounding context also allows for context-aware sensing and interaction. For example, the system can recognize which room (office, cafeteria, bedroom) or which transport vehicle (car, train, flight) the user is currently residing in, and then automatically switches into different modes, as shown in Figure 6 (d). For example, in an office, the phone will switch to silence mode whereas, in a bedroom, it will set an alarm for the next day. In a personal car, it will launch a navigation application whereas in a flight it will switch to flight mode.

iv) In-Car Scenario — Tracking various events happening inside a car is a very compelling use case. This includes detecting the number of passengers and seat occupancy, monitoring driver attention and drowsiness, reminding about unfastened seat belt, warning if hands not handling the steering wheel, detecting hand gestures for controlling infotainment system, detecting child presence, monitoring various vital signs (respiration, heart rate), and finally, capturing fun in-car moments automatically, as shown in Figure 6 (e).

v) Smart Home — When the phone rests on a desk or sofa, it can still track the user’s hand and the pointing gesture from a distance. For example, in a smart home scenario, the user can control various appliances just by pointing at them and perform simple gestures [73], while sitting comfortably on the sofa, as shown in Figure 6 (f). It also supports monitoring various human activities, such as if someone entered or leave the room, the user’s sitting posture, counting the reps of an exercise, or monitoring how long the user has been using the PC without resting.

vi) Conference Call — In a conference call, it provides a panoramic overview of the entire meeting site [55], recognize all the attendees by name, and detect the active speaker based on detected mouth movement or hand gesture, as shown in Figure 6 (g).

On the Protruly smartphone, the captured image or video is saved locally in a 360-degree equirectangular format. We convert the equirectangular image to multiple perspective images, each focusing on a different region of interest (ROI) for a different purpose, as shown in Figure 3 and 7. Then, for each ROI, we apply a different detection module. First, for the handedness and finger ROI, we apply four custom detection modules (described in the next Subsection 4.3) which detect handedness, active finger, 2D finger and 3D finger position. For the upper body and full body ROI, we apply the body pose estimation module to extract body joints information, which is further used for inferring body states (e.g., standing vs. sitting, touching the face, eye blinked), as shown in Figure 7 (middle). For the environment ROI, we apply an object detection module to detect approaching obstacles (e.g., car or human for danger detection) or nearby objects (e.g., mug or food for tangible interaction), as shown in Figure 7 (right). While this approach allows tracking bodies and objects in multiple regions simultaneously, it does not allow for a live demonstration, because the captured image is saved in the phone for later access, and there is no API available to access the built-in 360° camera in real-time.

We circumvent the limitation of lack of API access with a novel workaround, which is by using a real-time screen capture method with Android debug bridge (ADB). By launching the preloaded stock camera app, the screen renders the camera viewfinder preview in spherical format. Our software then captures the screen content (Figure 9) in real-time and forwards it to a remote PC for image processing and machine learning, using either a USB cable or WiFi.

Field of View — Specifically, the main hardware we used (Protruly V10S smartphone) has a screen resolution of 1920 x 1088 where the spherical preview area is 1088 x 1088 in pixels, with approximately 220+ degree FoV that can be adjusted (drag to rotate viewpoint, pinch to zoom). One downside of this screen capture method is that it is limited to the fisheye view on the screen, rather than the equirectangular format as described previously. Therefore, our processing and recognition pipeline is redesigned to deal with such limitations (e.g., barrel distortion) and built around it.

Opaque Overlay — To demonstrate a prototype system that works in real-time, we need to capture the camera preview on the screen while also rendering results to the user at the same time. For this reason, we created an opaque overlay using Android service to render results but reserved a transparent region at the center of the spherical area to allow the camera view to be screen captured (Figure 9). No rendering is allowed over this reserved region. We then developed various applications and UI using the overlay. And since the gripping hand and finger are always visible at the center of the image, the transparent region is set to the center 600 x 600 pixels, which only occupied 17% of the total screen’s real estate, where the remaining area can be used for rendering, as shown in Figure 9. Furthermore, this cropped region is fed as input to the neural network, which simplifies the machine learning task.

Distortion — Although this fisheye image has barrel distortion, the user’s body is close to the center of the image where the distortion is minimal, standard body pose tracking methods will work without modification. In particular, we tried the OpenPose [9] library which worked very well for tracking the body, hand and face, as shown in Figure 10 (a) fisheye.

Un-distortion — Nonetheless, to obtain optimal body pose results, we first unwrap the fisheye image and convert it to different formats (perspective, equirectangular & cubemap), before applying body pose tracking. To calibrate, we took multiple chessboard images from different angles, covering as wide the FoV as possible. Then, we apply calibration using OpenCV and OCamCalib [56] procedure, to extract parameters of the fisheye camera. From the parameters, we generated a pixel-to-pixel conversion map for real-time undistortion, which is applied for every new frame. Examples of the body tracking results for different formats can be seen in Figure 10. Fisheye has the most information, but the body size is unbalanced, where the shoulder appears larger and the legs appear shorter. Cubemap appears to strike a good balance, with minimal information loss (leg slightly cropped when near to edge). In the Perspective image, because the conversion causes high distortion near the edge, we had to limit the FoV (approximately 140°). This causes the leg to be cropped when it is extended. Equirectangular looks good for the middle part of the image, but is highly distorted at the top and bottom part of the image. Further, detailed comparisons are presented in Appendix A.

The recognition of handedness and active finger can be posed as a classification task with a few classes. As the hand gripping the phone and the finger touching the screen are always visible in the center of the image, we crop only the center ROI (Figure 9) and fed this as input to a neural network. We adopted the EfficientNet [60] network architecture, as it achieves higher accuracy and efficiency over existing architectures. Specifically, we chose EfficientNet-B4 [60] as it strikes a good balance between accuracy and speed. We replace the last layer with another fully connected layer, using a softmax activation function. Figure 11 shows example dataset.

There are existing methods that regress human 3D keypoints from a single image [50, 69, 75], of which the most common ones use heatmap-based regression. Yet, these approaches largely target a single undistorted image. Different from the previous work on body pose estimation, a finger captured by the 360� camera is distorted and occasionally cropped due to its close distance to the camera, thus a conventional method does not work well. Hence, in this paper, we trained a neural network using the dataset we collected, to achieve faster inference time and obtain higher precision. Our network architecture is shown in Figure 12.

For 2D fingertip tracking, we use a PoseResNet-like [69] encoder-decoder-based convolutional network, which employs ResNet-50 as the backbone to regress 2D heatmaps of different fingertips of both hands. This is similar to the approach used by Xu et al.� [71]. Since the 2D regression here is a relatively simple task where the fingertip is clearly shown in the camera view without any occlusion, we suggest a simple yet effective network such as the PoseResNet [69] is sufficient for a baseline implementation, with the advantage of reducing both training and inference time.

For 3D fingertip tracking, estimating the depth of a fingertip from a 2D image is a challenging task, especially for a fisheye lens. Xu et al. [71] succeeded in estimating the distance of body joints from a head-mounted fisheye camera using another stream to estimate the depth map, which is adapted to our case. To achieve better accuracy, we performed a 2-level stacked training to estimate the depth both from the 1st-level output and 2nd-level features, as shown in Figure 12. The network will output a 2D key point heatmap as well as a 3D depth map of the corresponding fingertip which we can extract the 3D camera coordinates from.

Hereby, in the following experiment, we only collected data and trained the network for estimating the three representative joints that are mostly used in mobile interaction: the left thumb, left index, and right index fingertip. Nonetheless, it is possible to expand this tracking to other fingertips or even estimating a full-hand pose.

We collected data from five participants from a local campus (mean age: 28.6, one female), for two sessions separated by a day. Considering that different users may grip the phone differently, and even a single user may grip the phone differently each time. With this in mind, we designed our data collection procedure to elicit such behaviors. For handedness recognition, participants were asked to hold the phone with their natural and comfortable grip, as if they are using the phone normally. They were asked to vary their handedness and their gripping behavior slightly while walking around the campus (both indoor and outdoor), as shown in Figure 11. For active finger recognition, participants were asked to hold the phone with their left hand, while performing random finger movements on the screen using the right hand’s index finger, middle finger, knuckle or a stylus. 1500 images are collected for each case. In total, there are 45000 images for handedness and 60000 images for active finger recognition. We train the EfficientNet-B4 [60] network (with pre-trained weights) with this dataset, for 30 epochs, using a learning rate of 1e-5 and the Adam [34] optimizer.

We conducted both user-dependent and user-independent evaluations. For the user-dependent test, we take a single day’s data for training, and the other day’s for testing, then repeat it for 2 folds. For the user-independent test, we take data from both days but leave-1-user-out, train a model for each fold and test on the remaining user data, for 5 folds.

For the handedness task, the accuracy is 99.55% (user-dependent) and 98.98% (user-independent). For the active finger task, the accuracy is 97.13% (user-dependent) and 96.12% (user-independent). Overall, the classification results of both the handedness task and active finger task are very accurate and robust, even across users (over 96%). Both of these are rather simple tasks, with well-defined crop area and small input size, owing to the fact that hand and fingers are always visible at the center of image when user is holding and interacting with the touchscreen.

We collected data from five participants from a local campus (mean age: 28.0, two females). For the left index finger and thumb, we asked the participants to hold the phone with their comfortable grip. They were asked to perform random finger movements while walking around a lab. In total, approximately 49,000 images were collected. 2D (X, Y) location of the left thumb and left index fingertip are annotated manually. For the right index finger, an Intel RealSense depth camera is mounted (Figure 12) on the phone only for collecting ground truth depth data, which is used for the purpose of training the deep neural network. The depth camera is not used during real-time inference and demonstration. Since the depth camera is tethered to a PC, participants remain seated in front of a green backdrop. In total, approximately 30,000 RGB and 30,000 depth images were collected. 3D (X, Y, Z) location of the right index fingertip is annotated by the authors. The 2D heatmap is generated from the XY coordinates using the Gaussian kernel, while the depth map is generated based on the Z-value from a depth camera. All training for the fingertip tracking uses a PyTorch framework, with an initial learning rate of 1e-4, batch size of 64, and Adam [34] optimizer with a multi-step learning rate.

In both 2D and 3D evaluations, we evaluated a user-dependent general accuracy and a cross-user validation (leave-1-user-out). In the general evaluation, we trained a general model using data from all participants while keeping 20% data randomly from each participant for testing. On the other hand, in the leave-1-user-out evaluation, we trained the network using 4 users’ data and evaluated the remaining user, for 5 folds. This aims to study the cross-user generalizability of the model. The results are shown in Table 2.

We use a mean absolute error (MAE) as the evaluation metric for the 2D real-world coordinates, of which the unit is in millimeters (mm), similar to related work [51, 67]. To measure the difference in heatmap result, we calculate the root mean square error (RMSE) in pixel-scale to show the similarity of the predicted heatmap and ground truth. We also include a Percentage of Correct Keypoint (PCK) metric with a threshold of 15 mm to represent a more intuitive accuracy. On the other hand, for the 3D fingertip results, as shown in Table 2 (bottom), there are two mean absolute errors (MAE) for either the 3D coordinates or the Z-value only, where the units are all in millimeters (mm). Additionally, the PCK becomes 3D-PCK which is based on the average 3D Euclidean distance between the estimated 3D fingertip position and the ground truth value.

For the fingertip tracking, the 2D results have an acceptable accuracy (MAE: Left Index 2.04 mm, Left Thumb 2.31 mm) in the general models and a slightly larger error in cross-user models (MAE: Left Index 2.78 mm, Left Thumb 2.72 mm), where we believe an average error under 3 mm is sufficient for mobile interaction. For comparison, TouchPose [2] used a capacitive touchscreen to reconstruct full hand poses and it showed around 10 mm of error when all fingers are touching the screen (the evaluation setup is different from ours, hence not a direct comparison). Our system tracks the 2D position of only the thumb and index but achieved error of less than 3 mm accordingly. Importantly, the small difference between general and cross-user models indicates the generalizability to new users.

By contrast, the mean errors of coordinates in 3D tracking increase noticeably (MAE: General 4.11 mm, Leave-1-out 5.46 mm). This result is comparable with related work although the settings are not identical. For instance, DeepFishEye [51] tracks five fingertips and yielded errors of ∼ 20 mm for both index and thumb on the smartphone-sized screen. The depth (Z) value has a higher error (MAE of Z-only: General 8.25 mm, Leave-1-out 10.80 mm) when compared to the 2D results. This is because the depth (the distance from the camera) is much more difficult to estimate (with an error of 8 mm ∼ 10 mm), especially from a distorted, low-resolution fisheye image. In fact, estimating precise depth information from only a 2D image in real-time remains a challenging task in the CV literature. In addition, the human labeling process may have contributed to the increase in the error rate, especially when the finger is very near to the camera, where a small imprecision in the human annotation may lead to a large error offset. As we used a cropped ROI, when the finger is extended to the extreme, it goes out of view and causes a failure prediction (as shown in Figure 13). In 2D tracking, the prediction jumps to other fingers while in 3D tracking it results in wrong depth estimation. This can be fixed by using a larger ROI, or via filtering. Nevertheless, the overall 3D-PCK is around 90% which means most of the estimated points are close to the ground truth and we suggest that 5 mm mean 3D position error and 10 mm mean depth error is acceptable for many use cases that do not require high precision, such as i) grouping the hover areas based on zones and ii) coarse finger gestures or iii) PreTouch [24] and Air+Touch [14] style of interaction based on hover detection.