See the reuse

Automatic video dubbing and translation from lb to other languages

See the reuse

Description

This project was made in 2 days during the '{lang: lb}' hackathon in 2023.

The goal of our proof of concept is to be able to do automated machine dubbing from video in Luxembourgish. The goal is to fasten the work of translators by providing a working base they can edit, but also fill the gap of lack of Luxembourgish support in automated video dubbing system.

This python programs let you:

  1. Either download a video and extract audio track or use an already existing audio track
  2. Convert the track to mono 16khz mp3 track
  3. Run this track through schreifmaschinn's API (using Meta XLSR model) to get text extracted from speech
  4. Group all words per ±3s (± 11 words considering human average speech speed is 220 wpm) in a CSV format transcript.csv
  5. Send this, alongside a custom prompt, to chatGPT API in order to get it
    1. Translated in another language
    2. Rewrite to get a decent quality
    3. Split the translation back to the original timecode
    4. Change the timecode into dubbing timecode format
    5. Generate text following SRT template
  6. Save the result, per language, in several transcript-XX.srt

You'll find a diagram of how it works, and you'll see in blue the already identified possible optimisation:

You can also find a reference video at https://www.youtube.com/watch?v=4dEOXDq5lgU and see also the result of our automated translation embeded as Youtube subtitles.

Hackathon {lang:lb} laachen team

Topic

Science and Technology

Type

Application

Tags

  • audio

  • dubbing

  • letzebuergesch

  • luxembourgeois

  • luxembourgish

  • sous-titrage

  • traduction

  • translation

  • video

Creation date

June 9, 2023

Last update

July 21, 2025

1 used dataset

Discussions

There are no discussions for this reuse yet.

Reuses from the same creator

There are no other reuses from this creator.