miratmu/ffmpeg-tensorflow repository overview

⁠FFMpeg with Libtensorflow

Since Google Summer of Code 2018⁠, FFMpeg supports the sr filter⁠ for applying super-resolution methods based on convolutional neural networks. However, compiling FFMpeg with proper libraries and preparing models for super-resolution requires expert knowledge. This repository provides a Dockerfile that makes super-resolution in FFMpeg a breeze!

Below, we show how you can apply super-resolution to your video in no time. For more information, see our GitHub repository⁠.

⁠Upscale a video using super-resolution

Download an example video⁠ and use the ffmpeg-tensorflow docker image to upscale it using one of the super-resolution models (here ESPCN):

wget https://media.xiph.org/video/derf/y4m/flower_cif.y4m
alias ffmpeg-tensorflow='docker run --rm --gpus all -u $(id -u):$(id -g) -v "$PWD":/data -w /data -it ffmpeg-tensorflow'
ffmpeg-tensorflow -i flower_cif.y4m -filter_complex '[0:v] format=pix_fmts=yuv420p, extractplanes=y+u+v [y][u][v]; [y] sr=dnn_backend=tensorflow:scale_factor=2:model=/models/espcn.pb [y_scaled]; [u] scale=iw*2:ih*2 [u_scaled]; [v] scale=iw*2:ih*2 [v_scaled]; [y_scaled][u_scaled][v_scaled] mergeplanes=0x001020:yuv420p [merged]' -map [merged] -sws_flags lanczos -c:v libx264 -crf 17 -c:a copy -y flower_cif_2x.mp4

The flower_cif_2x.mp4 file with the upscaled example video should be produced. Compare upscaling using Lanczos filtering (left) with upscaling using the ESPCN super-resolution model (right):

Besides ESPCN, the docker image includes pre-trained SRCNN, VESPCN, and VSRNET models in the /models directory. The architectures⁠ and experimental results⁠ for the super-resolution results are described in the HighVoltageRocknRoll/sr GitHub repository.