Enhanced In-air Signature Verification via Hand Skeleton Tracking to Defeat Robot-level Replays

Behavioral biometrics has emerged as an important security factor for user authentication. Compared to static biometrics (e.g., faces, irises, and fingerprints), using human motion behaviors for authentication causes lower concern about privacy abuse, and behavior biometrics are shown hard to be replicated by humans. In-air 3D signature is one representative of behavioral biometrics. Specifically, a user’s hand movements can be tracked by visual or wireless sensors for contact-free signature authentication, where both the fingertip trajectory and the dynamic motion features are verified to provide enhanced security. However, with the advancement of 3D printing and robot technology, we find that 1) existing hand-tracking interfaces (e.g., Leap Motion and Google MediaPipe) are easily tricked by a fake hand, and 2) a robotic arm can reproduce a user’s in-air 3D signature with high similarity regarding both trajectory and motion behaviors. Thus, this work investigates the security of in-air signatures under robot-level replays and proposes to extend the signature verification from a single-point fingertip to multiple hand joints for enhanced security. We develop the hand skeleton-based 3D signature verification system, which can be deployed on any single camera devices (2D or 3D). The key insight is that current robots could hardly replicate the minute and unique inter-joint motions of a user. In particular, we track the hand skeleton using a single camera and reconstruct/draw the trajectories of its joints in a virtual 3D space, using the color gradients to represent time-lapse and using varying line widths to describe joint significance. Based on that, we extract the three-view skeleton signatures and inter-joint motion features and develop a convolutional neural network for verification. Extensive experiments show that our system not only achieves high authentication performance but also effectively mitigates robot-level replay attacks.