The VirtualSign project intends to reduce the linguistic barriers between the deaf community and those not suffering from hearing disabilities by offering them a bidirectional translator of Portuguese Sign Language. More precisely, it translates the performed Sign Language gestures into written text and translates written text into the respective gestures in Sign Language.
This application was developed by a team of University teachers as a response to the challenges and difficulties faced while trying to communicate with hearing impaired students. Funded by the Portuguese government through the FCT – Fundação para a Ciência e Tecnologia, the project includes researchers from ISEP - School of Engineering of the Polytechnic of Porto, the Faculty of Engineering of the University of Porto and the Portuguese Open University and an expert in Portuguese Sign Language. Praxis is cooperating with VirtualSign and will use it to have a translation of the Praxis portal to international sign language.
As a holistic approach has the best chances to successfully address the communication and learning problems, the Virtual Sign team designed a translation system that involves both hardware and software together. The translator from Sign Language to text resorts to two devices, 5DT Sensor Gloves and the Microsoft Kinect, in order to gather data about the shape of the hands and the position of the body. Subsequently, the text goes through As for the translation of written Portuguese into Portuguese Sign Language, the process is supported by a 3D avatar which interprets the entered text and performs the corresponding animations.
As a part of the project, it was also developed a serious game directed to assist in the difficult task of learning the Portuguese Sign Language. The game has three main scenes. In the first, the user learns the alphabet. The second scene presents several of the most used words. Lastly, the third scene contains frequently used sentences. The learning process is based on the observation of the avatar and on the performance of the player using the Kinect. There is an inventory system which stores all the gestures acquired by the player, which can be accessed at any time and performed by the avatar. In this way, the player is able to learn and correctly make the gestures. All these signs are scattered around the scenes and some must be acquired by challenges, sometimes requiring to perform a sign in order to obtain another, or beating a fixed score on a minigame. The motivation that drives users to play the game is thus exploited in order to make them learn while still enjoying the gameplay. Moreover, this methodology smoothens the learning process and allows the user to practice PSL while getting rewarded for it.
The VirtualSign team hopes that the implementation of this project helps those who have limitations in communicating, providing them with more equitable access to education and opportunities.
Find more information about VirtualSign project at their website and Facebook profile.