Hello, my name is Sarah Alyami
I will present our paper titled
CLIP-SLA: Parameter Efficient CLIP Adaptation for Continuous Sign Language Recognition
This work is in collaboration with
Dr. Hamza Luqman at King Fahd University of Petroleum and Minerals
Starting with the motivation behind this
work while Continuous Sign Language Recognition has seen significant progress
current frameworks mostly rely on visual backbones pretrained on visual data.
However, recently vision-language models like CLIP
have demonstrated impressive generalization across a wide range of tasks
by learning strong semantic representations from large-scale image–text data
But despite their success, these models
haven't been explored in the context of Continuous Sign Language Recognition
In this work, we aimed to
investigate how can we adapt the
powerful representations of CLIP to benefit Continuous Sign Language Recognition
However, using CLIP directly in our
task is challenging because first the
CLIP model is designed for image understanding
and lacks temporal modeling, which is
necessary for Continuous Sign Language Recognition
Also, now naive full fine-tuning risks overwriting its pretrained representations
To overcome these challenges, we proposed
frameworks that integrate temporal modules within CLIP
Also, instead of fully fine-tuning CLIP,
we proposed using parameter-efficient fine-tuning techniques
As a result, we proposed
CLIP-SLA or CLIP Sign Language Adaptation,
a parameter-efficient framework that adapts the CLIP visual encoder for
Continuous Sign Language Recognition without fully fine-tuning it
The adapted CLIP visual encoder is
integrated with the standard sequence Module in Continuous Sign Language Recognition
We keep the CLIP weights frozen
using the Vision Transformer backbone and introduce two variants
SLA-Adapter and SLA-LoRA to inject task-specific understanding
I'll explain each of these two variants
briefly starting with SLA Lora
SLA Lora uses low rank rank adaptation to modify the weights
within the attention and feed-forward blocks of the CLIP backbone
We also incorporate Temporal Shift Module,
which shifts a few channels across
time steps, enabling cross-frame temporal understanding
Moving on to the second variant SLA-Adapter,
In SLA Adaptor, we experimented with
a different method of adaptation by inserting lightweight trainable adapter blocks
The adapters include depth-wise 3D convolutions
to efficiently learn spatial temporal futures
while keeping the CLIP weights frozen
Both variants SLA-Lora and SLA-Adapter
avoid the cost of full fine-tuning and enable temporal reasoning
We evaluated our frameworks on four
datasets, including Phoenix2014 and
also on a new dataset Isharah-500
which is a diverse dataset
in Saudi sign language recorded using smartphone cameras
Here are the results reported in
Word Error Rate, where lower Word Error Rate indicates better performance
Our frameworks achieved competitive performance across
all three datasets, ranking
second after the SlowFastSign framework
We conducted an efficiency analysis to
understand the benefits our proposed adaptation
strategies, including the number of tunable
and training time
Here we show the number of tunable parameters versus the Word Error Rate
Our frameworks achieved strong balance between accuracy and efficiency.
For example, SLA-Adaptor has 41%
fewer tunable parameters than the best
models SlowFastSign, while achieving
comparable performance with only 0.5 Word Error Rate difference
As for the results on the
Isharah-500 dataset, our frameworks
outperformed previous baselines with a significant margin
These results validate the robustness of
our frameworks in handling realistic and
challenging scenarios, such as poor lighting
and clutter backgrounds
We ran extensive ablations such as
component analysis that verified the effectiveness
of the proposed modules, also conducted
ablations to find the best settings
for the LoRA and adapter blocks.
We also visualized the class activation
maps, which showed that our models
can effectively focus on the informative
regions, as you can see here,
with focus on the hands and mouth regions
In conclusion, the CLIP-SLA frameworks
effectively adapt CLIP for
continuous Sign Language Recognition using parameter efficient fine-tuning
The findings of this paper highlight
the potential of pre-trained vision language
models for Continuous Sign Language Recognition,
and future work can explore other
adaptation techniques such as visual prompt tuning
Thank you everyone