Automatic video dubbing and translation from lb to other languages
Publié le 9 juin 2023
Hackathon {lang:lb} laachen team
This org. is the "Laachen" team from the 2023 {lang:lb} hackathon
1 réutilisations
Informations
- Type
- Application
- Thématique
- Science et technologie
- ID
- 64831d2e199953278c2d450d
Publication
Intégrer sur votre site
URL stable
Description
This project was made in 2 days during the '{lang: lb}' hackathon in 2023.
The goal of our proof of concept is to be able to do automated machine dubbing from video in Luxembourgish. The goal is to fasten the work of translators by providing a working base they can edit, but also fill the gap of lack of Luxembourgish support in automated video dubbing system.
This python programs let you:
- Either download a video and extract audio track or use an already existing audio track
- Convert the track to mono 16khz mp3 track
- Run this track through schreifmaschinn's API (using Meta XLSR model) to get text extracted from speech
- Group all words per ±3s (± 11 words considering human average speech speed is 220 wpm) in a CSV format
transcript.csv - Send this, alongside a custom prompt, to chatGPT API in order to get it
- Translated in another language
- Rewrite to get a decent quality
- Split the translation back to the original timecode
- Change the timecode into dubbing timecode format
- Generate text following SRT template
- Save the result, per language, in several
transcript-XX.srt
You'll find a diagram of how it works, and you'll see in blue the already identified possible optimisation:
You can also find a reference video at https://www.youtube.com/watch?v=4dEOXDq5lgU and see also the result of our automated translation embeded as Youtube subtitles.
Jeux de données utilisés 1
Discussion entre l'organisation et la communauté à propos de cette réutilisation.
Plus de réutilisations
Découvrez davantage de réutilisations.
