SignLab: Gesture Recognition System
Our internal research project in sign language recognition. Not a ready-made product, but a technical foundation we use to build custom solutions for specific business needs.
Client
SignLab is an AI-powered sign language translation system developed as an internal project. Within our Data Science team, we’ve built a habit of taking on complex and unconventional ideas to push our capabilities further. In 2022, we decided to build SignLab to test whether the concept works in practice.
We had solid experience in AI at that point, but no prior background in sign language.
Challenges
Sign language is a full linguistic system with its own structure and logic. However, it works differently from spoken and written languages. Meaning is conveyed not only through hand shapes and movement, but also through word order, spatial references, facial expressions, and other non-manual signals.
Many of the grammatical cues familiar from spoken languages do not map directly onto sign languages. Articles and punctuation are not expressed in the same way, and relationships that are typically conveyed through function words may instead be expressed through space, movement, and context. This makes the problem more complex. It is not enough for a model to recognize individual gestures. It needs to interpret the utterance as a whole.
For example, in many sign languages, time can be established early in the utterance, and the rest of the message is interpreted within that context. Sentence structure may differ significantly from spoken languages, with meaning shaped by word order, spatial organization, and non-manual markers rather than verb forms.
Another challenge comes from the three-dimensional nature of signing. The same sign may vary depending on hand position, movement, trajectory, and the use of signing space.
1. Training the Model
The first challenge was data. Where do you find a dataset suitable for training a sign language recognition model?
Public datasets were not a good fit for our task. Most available video datasets are designed for object detection or classification, not for modeling continuous gestures over time.
Adapting them would require significant effort in cleaning and restructuring, with no guarantee of achieving the required quality. From the start, it was clear that we would need to collect and label our own dataset.
2. Linguistic Nuances
Sign languages vary significantly across countries. They are not a single unified system, and users of different sign languages may not understand each other.
There are also individual differences in signing style. Some users shorten gestures, skip parts of a sign, or produce them with different timing and rhythm.
Non-manual signals play an important role. Facial expressions, hand rhythm, speed, and articulation all contribute to meaning and need to be taken into account.
Another layer of complexity comes from fingerspelling, where words are spelled out letter by letter. This is often used for abbreviations, names, or domain-specific terms. How should a model handle all of this?
Development process
From Recognition to Understanding
To translate sign language correctly, a model needs to go beyond recognizing individual gestures and learn to interpret meaning. How do you capture spatial information about body and hand positions? How do you interpret that information in context? How do you detect where one sign ends and another begins? How do you handle abstract concepts that have no direct visual representation?
Recognition Models
The system was built in Python, using open-source tools as part of the overall architecture.
MediaPipe
We used MediaPipe by Google to extract hand and body landmarks.
Training the Model
We recorded and annotated our own video data and built a gesture state vocabulary from scratch.
Sliding Window
We developed a sliding window algorithm that analyzes gesture sequences over time and uses this context to interpret each gesture.
How Gesture Recognition Works in SignLab
Step 1
MediaPipe Holistic processes the input image, detects the person in the frame, and estimates a 3D skeleton using body and hand keypoints.
Step 2
The model transforms the extracted keypoints into a vector representation that encodes the essential features of movement and spatial position.
Step 3
Each vector represents the body and hand configuration in a single frame. By processing sequences of these vectors over time, the model is able to recognize individual signs.
Step 4
The recognized signs are converted into text. Text-to-speech technologies are then used to produce natural text and voice output, adding grammatical structure and punctuation where needed.
Result
We built a working prototype that recognizes gestures in video streams and converts them into context-aware text. SignLab became a hands-on project where we worked at the intersection of computer vision and sequence modeling, in a setting where traditional language processing approaches do not apply directly.
The project covered the full pipeline end to end, from data collection and annotation to model training and evaluation, and to develop approaches for interpreting complex visual patterns in motion.
Reach out to us at ask@zuzex.com. We can adapt this solution to fit your business requirements.
SignLab is an internal project. It is not offered as a standalone product, but serves as a foundation for building similar systems for specific business needs, from gesture recognition to broader scenarios involving motion, behavior, and visual context analysis.
What happens next:
Having received and processed your request, we will reach you shortly to detail your project needs.
After examining requirements, our analysts and developers devise a project proposal with the scope of works, team size, time and cost estimates.
We arrange a meeting with you to discuss the offer and come to an agreement.
We sign a contract and start working on your project as quickly as possible.