Hello, today we are introducing BUTID,
which is the first large-scale dataset
for Turkish Sign Language translation.
Our data set consists of 529
hours of sign language footage coming
from 27 unique signers. Our data
consists of 256,000
automatically aligned translations in
daily language use. Let me
compare our data with other
large-scale data sets in the literature.
We can see that it is the sixth largest
dataset introduced in overall sign
language translation, while it is the
fourth largest one in single language
use, while it is the largest one for
Turkish sign
language. It provides two subsets
for our dataset,
differing in the context length. When we
look at the subsets, single caption
dataset consists of isolated captions.
While the multi-caption subset
consists of merged captions with
connected speech, meaning there is almost
no interference in between those
captions. For alignment,
we first select sentences or captions
with conversational gaps.
After extracting these frames
videos with an end offsets,
we apply pose estimation to find
the active frames for filtering,
meaning the active signing
occurs.
After this aligned data,
we apply the methodology
introduced in the YouTube ASL paper,
which consists of mapping the
pose estimation input with a projection
layer into the token space of the
language model for learning translation.
To do that, we. Use mT5 small
model with two layer MLP
projection with GeLU activation
function. To further
improve our methods, we apply
augmentation and paraphrasing.
For augmentation we use
frame dropping as well as
cue masking. For paraphrasing
we use paraphrasing both a method of
augmentation, population as well as the
evaluation method since
Turkish is an agglutinative language and
it has flexible word order,
so the sentence level alignment is harder
than when we compare it with English.
So to provide paraphrases
we use fine-tuned TURNA
which is a Turkish based LLM
trained on OpenSubtitles data set.
Using these methods we
extract our baseline scores. So when we
look at these scores we see that a
lower BLEU-4 score because
of the limited context length as well as
we see augmentation adds
noise since the context window is small
while in Multi caption, with
longer sequences, we see higher
BLEU-4 and we also
see augmentation has improvements
over the results.
For evaluating the effect of
paraphrasing and augmentation, we see
paraphrasing has significant
improvements in the single caption data,
while it is not applicable due
to the large context length of the text.
So when we look at the qualitative part
of our models, these in these
examples we see high
quality translations while seeing a
very low BLEU-4 score. So
this is due to the
BLEU-4 is almost zero in the single
context data because the average token
length is so small.
As well as the word order differs in
the language. So we see lower BLEU-3
and BLEU-4 matches. So we can say that
blue four metric is not a good
metric to evaluate Turkish based models.
We can also see that in these examples
while the root word base
word is the same in ground truth and
predictions. uh Due to the Turkish
morphology, we can even see lower
scores with the BLEU-1
metric. Thank you for listening.