Using image super resolution models with vapoursynth and speeding them up with TensorRT

Last update: Aug 23, 2022

Overview

vs-RealEsrganAnime-tensorrt-docker

Using image super resolution models with vapoursynth and speeding them up with TensorRT. Also a docker image since TensorRT is hard to install. Testing showed ~70% more speed on my 1070ti compared to normal PyTorch in 480p. Using the 2x model with TensorRT and 848x480 input was 0.517x realtime speed for 24fps video.

I was forced to use onnx/onnx-tensorrt instead of NVIDIA/Torch-TensorRT because of convertion errors with PyTorch, but the only disadvantage should be that a new onnx model needs to be created for a different input resolution, which takes a bit time.

This repo uses a lot of code from HolyWu/vs-realesrgan and xinntao/Real-ESRGAN. The models are from here.

Usage:

# install docker, command for arch
yay -S docker nvidia-docker nvidia-container-toolkit
# Put the dockerfile in a directory and run that inside that directory
docker build -t realsr_tensorrt:latest .
# run with a mounted folder
docker run --privileged --gpus all -it --rm -v /home/Desktop/tensorrt:/workspace/tensorrt realsr_tensorrt:latest
# you can use it in various ways, ffmpeg example
vspipe --y4m inference.py - | ffmpeg -i pipe: example.mkv

If docker does not want to start, try this before you use docker:

# fixing docker errors
systemctl start docker
sudo chmod 666 /var/run/docker.sock

If you don't want to use docker, vapoursynth install commands are here and a TensorRT example is here.

Set the input video path in inference.py and access videos with the mounted folder. You can also choose between the 4x and 2x model.

It is also possible to directly pipe the video into mpv. Change the mounted folder path to your own videofolder and use the mpv dockerfile instead. Only tested in Manjaro.

yay -S pulseaudio

# i am not sure if it is needed, but go into pulseaudio settings and check "make pulseaudio network audio devices discoverable in the local network" and reboot

# start docker
docker run --rm -i -t \
    --network host \
    -e DISPLAY \
    -v /home/Schreibtisch/test/:/home/mpv/media \
    --ipc=host \
    --privileged \
    --gpus all \
    -e PULSE_COOKIE=/run/pulse/cookie \
    -v ~/.config/pulse/cookie:/run/pulse/cookie \
    -e PULSE_SERVER=unix:${XDG_RUNTIME_DIR}/pulse/native \
    -v ${XDG_RUNTIME_DIR}/pulse/native:${XDG_RUNTIME_DIR}/pulse/native \
    realsr_tensorrt:latest
    
# run mpv
vspipe --y4m inference.py - | mpv -

Using image super resolution models with vapoursynth and speeding them up with TensorRT

Related tags

Overview

vs-RealEsrganAnime-tensorrt-docker

Owner

A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) based on Deep Filtering.

ComPhy: Compositional Physical Reasoning ofObjects and Events from Videos

This repository allows you to anonymize sensitive information in images/videos. The solution is fully compatible with the DL-based training/inference solutions that we already published/will publish for Object Detection and Semantic Segmentation.

HiFi-GAN: High Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks

Exploring Classification Equilibrium in Long-Tailed Object Detection, ICCV2021

The Deep Learning with Julia book, using Flux.jl.

Balancing Principle for Unsupervised Domain Adaptation

Convnext-tf - Unofficial tensorflow keras implementation of ConvNeXt

Implementation of Google Brain's WaveGrad high-fidelity vocoder

Official Implementation for the "An Empirical Investigation of 3D Anomaly Detection and Segmentation" paper.

This repo holds codes of the ICCV21 paper: Visual Alignment Constraint for Continuous Sign Language Recognition.

This repository contains code for the paper "Disentangling Label Distribution for Long-tailed Visual Recognition", published at CVPR' 2021

Implement the Pareto Optimizer and pcgrad to make a self-adaptive loss for multi-task

Transformer model implemented with Pytorch

(CVPR2021) Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation (CVPR'21)

Analysing poker data from home games with friends

[CVPR 2022 Oral] TubeDETR: Spatio-Temporal Video Grounding with Transformers

A Loss Function for Generative Neural Networks Based on Watson’s Perceptual Model

Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework