gpocali/ffmpeg-post-generator repository overview

⁠ffmpeg-post-generator

ffmpeg-post-generator is a powerful and flexible video generation tool that automates the creation of videos from text. It combines text overlays, images, background videos, and audio to produce engaging video content. With its template-driven approach, you can easily define the structure and style of your videos. The integrated Text-to-Speech (TTS) engine brings your text to life with high-quality narration.

⁠Features

• Template-Driven: Define video structure and styling using a flexible JSON template.
• Text-to-Speech (TTS): Automatically generate high-quality narration from your text using Coqui TTS.
• Dynamic Timeline: The "slide mode" automatically adjusts the video timeline based on the narration length.
• Rich Media Support: Combine video backgrounds, audio tracks, images, and text overlays.
• Customizable Text Styles: Control text color, alignment, and position.
• Audio Processing: Automatic audio normalization and mixing of background music and narration.
• Batch Processing: Generate multiple videos from a single input file.
• Dockerized: Easy to set up and run in a containerized environment.

⁠How it Works

The video generation process is orchestrated by the script.php file and can be broken down into the following steps:

1. Initialization: The script creates temporary and output directories.
2. Input Processing: It reads the main input text from input/input.txt and shuffles the lines to create a varied order for video generation. It also reads taglines from input/tagline.txt.
3. Template Loading: The template.json file is loaded to get the video settings and layer definitions.
4. Narration Generation:
   • For each text layer with narration enabled, the script collects the text.
   • It calls generate_and_split_narration.py to generate a single audio file containing the narration for all text layers.
   • This script uses whisper to transcribe the generated audio and find the precise start and end times for each text segment.
5. Timeline Adjustment (Slide Mode): If slide_mode is enabled, the script adjusts the timeline of the layers to accommodate the duration of the narration.
6. Asset Selection: Background video and audio files are selected from the background and audio directories.
7. Image Generation: For text layers, imagemagick is used to create PNG images with the specified text, color, and alignment.
8. FFmpeg Command Construction: A complex ffmpeg command is built based on the template, including:
   • Inputs for background video, audio, images, and narration.
   • Video filters for scaling, cropping, padding, and fading.
   • Audio filters for mixing, volume adjustments, and normalization.
9. Video Encoding: The ffmpeg command is executed to generate the final MP4 video.
10. File Cleanup: Temporary files are deleted.

⁠Requirements

This project is designed to run in a Docker container, which encapsulates all the necessary dependencies. To run the generator, you will need:

• Docker: To build and run the container.
• An NVIDIA GPU (Recommended): For significantly faster TTS processing. The Docker container is configured to use CUDA if a GPU is available.

The Dockerfile handles the installation of all other dependencies, including:

• PHP
• ffmpeg
• ImageMagick
• Python 3
• PyTorch with CUDA support
• Coqui TTS
• Whisper
• thefuzz

⁠Directory Structure

The project uses a specific directory structure to organize input files, assets, and output videos.

• input/: This directory contains the text files that serve as the primary input for the video generation.
• input/taglines/: This directory is used to store pre-generated audio files for taglines. The audio files should be in .wav format and named according to the sanitized tagline text.
• background/: Place your background video files in this directory. The script will randomly select a video from this folder for each generated video.
• audio/: This directory should contain your background audio tracks. The script will choose a random audio file for each video.
• logo/: Store your logo images and other static image assets in this directory.
• output/: The final generated videos will be saved in this directory.

⁠Input Files

⁠input/input.txt

This is the main input file for the video generator. Each line in this file will be used to generate a separate video. The script will process the lines in a random order.

Example:

This is the first video.
This is the second video.
This is the third video.

⁠input/tagline.txt

This file contains a list of taglines that can be used in the videos. Each line represents a single tagline. The script will cycle through the taglines for each generated video.

Example:

My Awesome Channel
Subscribe for more!

You can also pre-generate audio for your taglines to save time during the video creation process. See the generate_taglines.py script for more details.

⁠The template.json File

The template.json file is the heart of the video generator, defining the structure, styling, and behavior of the output videos.

⁠Global Settings

⁠Layers

The layers array defines the visual elements of the video. Each layer is an object with the following properties:

⁠Running the Generator

The project is designed to be run as a Docker container.

⁠1. Build the Docker Image

First, build the Docker image using the provided Dockerfile:

docker build -t ffmpeg-post-generator .

⁠2. Run the Container

Once the image is built, you can run the generator using the following command. This command mounts the project directories into the container, allowing the script to access your input files and save the output to your local machine.

docker run --rm --gpus all \
    -v $(pwd)/input:/app/input \
    -v $(pwd)/background:/app/background \
    -v $(pwd)/audio:/app/audio \
    -v $(pwd)/logo:/app/logo \
    -v $(pwd)/output:/app/output \
    ffmpeg-post-generator

Note on the --gpus all flag: This flag enables the container to use all available NVIDIA GPUs. If you do not have an NVIDIA GPU, you can remove this flag. However, TTS generation will be significantly slower.

⁠Python Scripts

The project uses several Python scripts to handle specific tasks.

• generate_narration.py: A script to generate a single audio file from a given text using Coqui TTS.
• generate_and_split_narration.py: The main script for handling narration. It takes a JSON object of narration "jobs", generates a single combined audio file, and then uses whisper to analyze the audio and determine the start and end times of each job. This is crucial for synchronizing the narration with the video layers.
• generate_taglines.py: A utility script to pre-generate audio files for all taglines listed in input/tagline.txt. This can save time during the main video generation process.
• check_speech.py: A helper script that uses whisper to detect if an audio file contains speech. This is used to ensure that background audio tracks without narration do not contain speech.
• test_tts.py: A simple script for testing the TTS setup and generating a sample audio file.

⁠Advanced Usage

⁠Custom TTS Voices

You can customize the voice used for narration by changing the narration_voice, narration_language, and narration_speaker properties in the template.json file.

• narration_voice: This should be the name of a Coqui TTS model. You can find a list of available models on the Coqui TTS website⁠.

• narration_language: This is required for some models, especially multilingual ones like xtts_v2.

• narration_speaker: For multi-speaker models, you can specify the speaker ID here. To get a list of available speakers for a model, you can run the following command inside the Docker container:

  tts --model_name "tts_models/en/vctk/vits" --list_speaker_idxs
  

  Copy

⁠Slide Mode

The slide_mode is a powerful feature that automatically adjusts the video timeline to fit the generated narration. When slide_mode is set to true in template.json:

1. The script generates the narration and calculates its duration.
2. It compares the narration duration to the available time for the layer in the timeline.
3. If the narration is longer than the available time, the script "stretches" the timeline by extending the outTime of the current layer and shifting the inTime and outTime of all subsequent layers.
4. This ensures that the narration is not cut off and that all layers are displayed for their intended duration relative to the narration.

This is particularly useful for creating presentation-style videos where the timing of the visuals should follow the narration.

⁠License

This project is licensed under the GNU General Public License v3.0. See the LICENSE⁠ file for details.