Automatic video dubbing and translation from lb to other languages

Published on June 9, 2023

See the reuse

Hackathon {lang:lb} laachen team

This org. is the "Laachen" team from the 2023 {lang:lb} hackathon

1 reuses

  • audio
  • dubbing
  • letzebuergesch
  • luxembourgeois
  • luxembourgish
  • sous-titrage
  • traduction
  • translation
  • video


Science and Technology





This project was made in 2 days during the '{lang: lb}' hackathon in 2023.

The goal of our proof of concept is to be able to do automated machine dubbing from video in Luxembourgish. The goal is to fasten the work of translators by providing a working base they can edit, but also fill the gap of lack of Luxembourgish support in automated video dubbing system.

This python programs let you:

  1. Either download a video and extract audio track or use an already existing audio track
  2. Convert the track to mono 16khz mp3 track
  3. Run this track through schreifmaschinn's API (using Meta XLSR model) to get text extracted from speech
  4. Group all words per ±3s (± 11 words considering human average speech speed is 220 wpm) in a CSV format transcript.csv
  5. Send this, alongside a custom prompt, to chatGPT API in order to get it
    1. Translated in another language
    2. Rewrite to get a decent quality
    3. Split the translation back to the original timecode
    4. Change the timecode into dubbing timecode format
    5. Generate text following SRT template
  6. Save the result, per language, in several

You'll find a diagram of how it works, and you'll see in blue the already identified possible optimisation:

You can also find a reference video at and see also the result of our automated translation embeded as Youtube subtitles.

Used datasets 1

Discussion between the organization and the community about this reuse.

More reuses

Discover more reuses.