An accessible, voice-controlled digital hand has been developed by researchers at the University of Michigan, designed to improve the convenience and accessibility of VR systems.
The software, called HandProxy, allows VR and AR users to interact with digital spaces by commanding a disembodied hand.
This hand can grab and move virtual objects, drag and resize windows, and perform gestures such as a thumbs up.
The system has been designed to tackle complex problems in VR, with tasks such as “clear the table” completed without
precise instructions for each object on the virtual table by using GPT-4o, the AI model behind ChatGPT.
The researchers claim
that the hand’s ability to independently complete tasks on the fly which the researchers say makes it more flexible than
current VR voice command features.
Anhong Guo, Morris Wellman faculty development assistant professor of computer science and engineering, University of
Michigan, commented: “Mobile devices have supported assistive technologies that enable alternative input modes and automated user-interface control, including AI-powered task
assistants like Siri. But such capabilities are largely absent in VR and AR hand interactions.
“HandProxy is our attempt to
enable users to fluidly transition between multiple modes of interaction in virtual and augmented reality, including controllers, hand gestures, and speech,”