Automatic video dubbing and translation from lb to other languages

Published on June 9, 2023

See the reuse

Hackathon {lang:lb} laachen team

This org. is the "Laachen" team from the 2023 {lang:lb} hackathon

1 reuses

audio
dubbing
letzebuergesch
luxembourgeois
luxembourgish
sous-titrage
traduction
translation
video

Informations

Type: Application
Topic: Science and Technology
ID: 64831d2e199953278c2d450d

Publication

Creation date: June 9, 2023

Participate

Add a reuse Contact the reuser

Embed

<div data-udata-reuse="64831d2e199953278c2d450d"></div><script  src="https://data.public.lu/static/oembed.js" async defer></script>

Permalink

Description

This project was made in 2 days during the '{lang: lb}' hackathon in 2023.

The goal of our proof of concept is to be able to do automated machine dubbing from video in Luxembourgish. The goal is to fasten the work of translators by providing a working base they can edit, but also fill the gap of lack of Luxembourgish support in automated video dubbing system.

This python programs let you:

Either download a video and extract audio track or use an already existing audio track
Convert the track to mono 16khz mp3 track
Run this track through schreifmaschinn's API (using Meta XLSR model) to get text extracted from speech
Group all words per ±3s (± 11 words considering human average speech speed is 220 wpm) in a CSV format transcript.csv
Send this, alongside a custom prompt, to chatGPT API in order to get it
1. Translated in another language
2. Rewrite to get a decent quality
3. Split the translation back to the original timecode
4. Change the timecode into dubbing timecode format
5. Generate text following SRT template
Save the result, per language, in several transcript-XX.srt

You'll find a diagram of how it works, and you'll see in blue the already identified possible optimisation:

You can also find a reference video at https://www.youtube.com/watch?v=4dEOXDq5lgU and see also the result of our automated translation embeded as Youtube subtitles.