The core principle of video translation software is: recognizing text from the speech sounds in the video, then translating that text into the target language, dubbing the translated text, and finally embedding the dubbing and text into the video.

As you can see, the first step is to recognize text from the speech sounds in the video. The recognition accuracy directly affects subsequent translation and dubbing.

Faster Local Mode

Recommended for use. This is a model converted from OpenAI's open-source Whisper. As the name implies, it offers faster recognition speed without compromising accuracy.

After selecting the Faster Mode, you can choose the model to use on the right side.

tiny --> base --> small --> medium --> large-v3-turbo --> large-v1 --> large-v2 --> large-v3 The model size increases, and similarly, the recognition accuracy also improves.

For Chinese videos, it is recommended to select a model at least equal to or larger than medium.

Models with the .en suffix and models starting with distil can only be used for English videos.

To speed up tasks, on Windows and Linux, if you have an NVIDIA graphics card, you can configure and install the CUDA and cuDNN environments, then enable CUDA Acceleration. This will significantly improve execution speed.

View CUDA and cuDNN Installation Tutorial

Model Download

By default, models are automatically downloaded from https://huggingface.co. This address is blocked in China and cannot be accessed without scientific internet access. Domestically, the mirror site https://hf-mirror.com will be automatically used as a replacement, but it is not very stable and downloads may fail. If it fails, please download manually using the following methods.

• Download the tiny model:

  1. Create a folder: Create a models--Systran--faster-whisper-tiny folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-tiny/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the tiny.en model:

  1. Create a folder: Create a models--Systran--faster-whisper-tiny.en folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-tiny.en/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the base model:

  1. Create a folder: Create a models--Systran--faster-whisper-base folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-base/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the base.en model:

  1. Create a folder: Create a models--Systran--faster-whisper-base.en folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-base.en/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the small model:

  1. Create a folder: Create a models--Systran--faster-whisper-small folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-small/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the small.en model:

  1. Create a folder: Create a models--Systran--faster-whisper-small.en folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-small.en/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the medium model:

  1. Create a folder: Create a models--Systran--faster-whisper-medium folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-medium/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the medium.en model:

  1. Create a folder: Create a models--Systran--faster-whisper-medium.en folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-medium.en/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the large-v3-turbo model:

  1. Create a folder: Create a models--mobiuslabsgmbh--faster-whisper-large-v3-turbo folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/mobiuslabsgmbh/faster-whisper-large-v3-turbo/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the large-v1 model:

  1. Create a folder: Create a models--Systran--faster-whisper-large-v1 folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-large-v1/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the large-v2 model:

  1. Create a folder: Create a models--Systran--faster-whisper-large-v2 folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-large-v2/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the large-v3 model:

  1. Create a folder: Create a models--Systran--faster-whisper-large-v3 folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-whisper-large-v3/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the distil-small.en model:

  1. Create a folder: Create a models--Systran--faster-distil-whisper-small.en folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-distil-whisper-small.en/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the distil-medium.en model:

  1. Create a folder: Create a models--Systran--faster-distil-whisper-medium.en folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-distil-whisper-medium.en/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the distil-large-v2 model:

  1. Create a folder: Create a models--Systran--faster-distil-whisper-large-v2 folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-distil-whisper-large-v2/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

• Download the distil-large-v3 model:

  1. Create a folder: Create a models--Systran--faster-distil-whisper-large-v3 folder inside the models folder located in the same directory as sp.exe(sp.py).
  2. Open the model download address: https://huggingface.co/Systran/faster-distil-whisper-large-v3/tree/main
  3. Download all the .json/.bin/.txt files from that page and copy them into the folder created above.

Automatic Language Detection

After version v2.59, the "Auto Detect" option has been added to the original language dropdown. When you don't know the language or the language is not among the 24 supported languages, you can select the "Auto Detect" option. The program will attempt to automatically recognize the spoken language.

Of course, if possible, try to avoid using this option, especially when there is no clear speech in the first 30 seconds of the video. This is because the principle of automatic detection is to use the first 30-second audio clip to determine the language, which is then set for the entire video. Another point to note: Some languages with similar pronunciation but different writing systems cannot be accurately identified and may be recognized as any one of them. For example, a Chinese video might be randomly recognized as Simplified or Traditional.