Voir la réutilisation

Automatic video dubbing and translation from lb to other languages

Voir la réutilisation

Description

This project was made in 2 days during the '{lang: lb}' hackathon in 2023.

The goal of our proof of concept is to be able to do automated machine dubbing from video in Luxembourgish. The goal is to fasten the work of translators by providing a working base they can edit, but also fill the gap of lack of Luxembourgish support in automated video dubbing system.

This python programs let you:

  1. Either download a video and extract audio track or use an already existing audio track
  2. Convert the track to mono 16khz mp3 track
  3. Run this track through schreifmaschinn's API (using Meta XLSR model) to get text extracted from speech
  4. Group all words per ±3s (± 11 words considering human average speech speed is 220 wpm) in a CSV format transcript.csv
  5. Send this, alongside a custom prompt, to chatGPT API in order to get it
    1. Translated in another language
    2. Rewrite to get a decent quality
    3. Split the translation back to the original timecode
    4. Change the timecode into dubbing timecode format
    5. Generate text following SRT template
  6. Save the result, per language, in several transcript-XX.srt

You'll find a diagram of how it works, and you'll see in blue the already identified possible optimisation:

You can also find a reference video at https://www.youtube.com/watch?v=4dEOXDq5lgU and see also the result of our automated translation embeded as Youtube subtitles.

Hackathon {lang:lb} laachen team

Thématique

Science et technologie

Type

Application

Tags

  • audio

  • dubbing

  • letzebuergesch

  • luxembourgeois

  • luxembourgish

  • sous-titrage

  • traduction

  • translation

  • video

Date de création

9 juin 2023

Dernière mise à jour

21 juillet 2025

1 jeu de données

Discussions

Il n'y a pas encore de discussion pour cette réutilisation.

Réutilisations du même créateur

Il n'y a pas d'autres réutilisations de ce créateur.