Using VapourSynth with super resolution models and speeding them up with TensorRT.

Overview

VSGAN-tensorrt-docker

Using image super resolution models with vapoursynth and speeding them up with TensorRT. Using NVIDIA/Torch-TensorRT combined with rlaphoenix/VSGAN. This repo makes the usage of tiling and ESRGAN models very easy. Models can be found on the wiki page. Further model architectures are planned to be added later on.

Currently working:

  • ESRGAN
  • RealESRGAN (adjust model load manually in inference.py, settings wont be adjusted automatically currently)

Usage:

# install docker, command for arch
yay -S docker nvidia-docker nvidia-container-toolkit
# Put the dockerfile in a directory and run that inside that directory
docker build -t vsgan_tensorrt:latest .
# run with a mounted folder
docker run --privileged --gpus all -it --rm -v /home/Desktop/tensorrt:/workspace/tensorrt vsgan_tensorrt:latest
# you can use it in various ways, ffmpeg example
vspipe --y4m inference.py - | ffmpeg -i pipe: example.mkv

If docker does not want to start, try this before you use docker:

# fixing docker errors
systemctl start docker
sudo chmod 666 /var/run/docker.sock

Windows is mostly similar, but the path needs to be changed slightly:

Example for C://path
docker run --privileged --gpus all -it --rm -v //c/path:/workspace/tensorrt vsgan_tensorrt:latest

If you don't want to use docker, vapoursynth install commands are here and a TensorRT example is here.

Set the input video path in inference.py and access videos with the mounted folder.

It is also possible to directly pipe the video into mpv, but you most likely wont be able to archive realtime speed. Change the mounted folder path to your own videofolder and use the mpv dockerfile instead. If you use a very efficient model, it may be possible on a very good GPU. Only tested in Manjaro.

yay -S pulseaudio

# i am not sure if it is needed, but go into pulseaudio settings and check "make pulseaudio network audio devices discoverable in the local network" and reboot

# start docker
docker run --rm -i -t \
    --network host \
    -e DISPLAY \
    -v /home/Schreibtisch/test/:/home/mpv/media \
    --ipc=host \
    --privileged \
    --gpus all \
    -e PULSE_COOKIE=/run/pulse/cookie \
    -v ~/.config/pulse/cookie:/run/pulse/cookie \
    -e PULSE_SERVER=unix:${XDG_RUNTIME_DIR}/pulse/native \
    -v ${XDG_RUNTIME_DIR}/pulse/native:${XDG_RUNTIME_DIR}/pulse/native \
    vsgan_tensorrt:latest
    
# run mpv
vspipe --y4m inference.py - | mpv -
Comments
  • Invalid data found when processing input

    Invalid data found when processing input

    Hey when i start the inference.py script this happen :

    someone can help me ?

    
    > ffmpeg version N-62110-g4d45f5acbd-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2022 the FFmpeg developers
    >   built with gcc 8 (Debian 8.3.0-6)
    >   configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
    >   libavutil      57. 26.100 / 57. 26.100
    >   libavcodec     59. 33.100 / 59. 33.100
    >   libavformat    59. 24.100 / 59. 24.100
    >   libavdevice    59.  6.100 / 59.  6.100
    >   libavfilter     8. 40.100 /  8. 40.100
    >   libswscale      6.  6.100 /  6.  6.100
    >   libswresample   4.  6.100 /  4.  6.100
    >   libpostproc    56.  5.100 / 56.  5.100
    > Information: Generating grammar tables from /usr/lib/python3.8/lib2to3/Grammar.txt
    > Information: Generating grammar tables from /usr/lib/python3.8/lib2to3/PatternGrammar.txt
    > Script evaluation failed:
    > Python exception: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
    > 
    > Traceback (most recent call last):
    >   File "src\cython\vapoursynth.pyx", line 2890, in vapoursynth._vpy_evaluate
    >   File "src\cython\vapoursynth.pyx", line 2891, in vapoursynth._vpy_evaluate
    >   File "inference.py", line 85, in <module>
    >     clip = ESRGAN_inference(clip=clip, model_path="/workspace/RealESRGAN_x4plus_anime_6B.pth", tile_x=480, tile_y=480, tile_pad=16, fp16=False, tta=False, tta_mode=1)
    >   File "/workspace/tensorrt/src/esrgan.py", line 680, in ESRGAN_inference
    >     import torch_tensorrt
    >   File "/usr/local/lib/python3.8/dist-packages/torch_tensorrt/__init__.py", line 11, in <module>
    >     from torch_tensorrt._compile import *
    >   File "/usr/local/lib/python3.8/dist-packages/torch_tensorrt/_compile.py", line 2, in <module>
    >     from torch_tensorrt import _enums
    >   File "/usr/local/lib/python3.8/dist-packages/torch_tensorrt/_enums.py", line 1, in <module>
    >     from torch_tensorrt._C import dtype, DeviceType, EngineCapability, TensorFormat
    > ImportError: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
    > 
    > pipe:: Invalid data found when processing input
    
    
    opened by NeoBurgerYT 10
  • Module not found 'scipy'

    Module not found 'scipy'

    I can't run my inference.py without getting this error message. Can someone direct me to where I can get the repo?

    File "/usr/local/lib/python3.8/dist-packages/mmedit/core/evaluation/metrics.py", line 7, in from scipy.ndimage import convolve ModuleNotFoundError: No module named 'scipy'

    pipe:: Invalid data found when processing input

    opened by terminatedkhla 8
  • Tutorial?

    Tutorial?

    Hi! This is amazing technology! I’m blown away. I’d love to contact you directly on how to use it in colab, I’m quite confused with the process. I’ve tried running it but not sure I’m running it correctly. Thanks in advance!

    opened by AIManifest 6
  • Trying On A M1 Mac

    Trying On A M1 Mac

    So I followed this tutorial https://www.youtube.com/watch?v=B134jvhO8yk&t=0s But when docker run --privileged --gpus all -it --rm -v /home/vsgan_path/:/workspace/tensorrt styler00dollar/vsgan_tensorrt:latest it just gives me an error that it doesn't find the right amd64 or somthing and I rage quit deleted it without seeing the full error. PLS HELP ME :(

    opened by Ghostkwebb 6
  • Crash when using RIFE ensemble models in vsmlrt

    Crash when using RIFE ensemble models in vsmlrt

    I get this error

    vapoursynth.Error: operator (): expects 8 input planes
    

    from this

    import vapoursynth as vs
    from vapoursynth import core
    core = vs.core
    import vsmlrt
    
    clip = core.lsmas.LWLibavSource(source=r"R:\output.mkv",cache=1, prefer_hw=1)
    clip = core.resize.Bicubic(clip, matrix_in_s="709", transfer_in_s='709', format=vs.RGBS)
    clip = vsmlrt.RIFE(clip, multi=4, model=46, backend=vsmlrt.Backend.TRT(fp16=True), tilesize=[1920,1088])
    clip = core.std.AssumeFPS(clip=clip, fpsnum=60, fpsden=1)
    clip = core.resize.Bicubic(clip, format=vs.RGB24, matrix_in_s="709")
    clip.set_output()
    
    opened by banjaminicc 4
  • Support for AITemplate?

    Support for AITemplate?

    There is something that came out recently and it's look promising in terms of performance/speed. Would it be possible to implement it for ESERGAN mode? https://github.com/facebookincubator/AITemplate

    opened by kodxana 4
  • CUDA out of Memory

    CUDA out of Memory

    System Specs: Ryzen 9 5900HX, NVidia 3070 Mobile, Arch Linux (EndeavorOS) on Kernel 5.17.2

    Whenever I try to run a model that is relying on CUDA, for example cugan, the program exits with

    Error: Failed to retrieve frame 0 with error: CUDA out of memory. Tried to allocate 148.00 MiB (GPU 0; 7.80 GiB total capacity; 5.53 GiB already allocated; 68.56 MiB free; 5.69 GiB reserved in total by PyTorch)

    and stops after having output 4 frames.

    However, TensorRT works fine for models that support it (like RealESRGAN for example).

    Edit: Running nvidia-smi while the command is executed reveals that vspipe is allocating GPU Memory, but <2 GiB of VRAM, far from the 8GiB my model has.

    opened by mmkzer0 4
  • No module named 'vsbasicvsrpp'

    No module named 'vsbasicvsrpp'

    Traceback (most recent call last): File "src\cython\vapoursynth.pyx", line 2832, in vapoursynth._vpy_evaluate File "src\cython\vapoursynth.pyx", line 2833, in vapoursynth._vpy_evaluate File "inference.py", line 12, in from vsbasicvsrpp import BasicVSRPP ModuleNotFoundError: No module named 'vsbasicvsrpp'

    opened by xt851231 4
  • Google colab request?

    Google colab request?

    I recently stumbled upon this VSGAN-tensorrt-docker and found it so incredible! Could anyone make a google colab notebook that features everything from this VSGAN-tensorrt-docker, so that we could experience the speed of TensorRT! Thanks in advance!

    opened by mikebilly 3
  • model conversion from onnx to trt

    model conversion from onnx to trt

    @styler00dollar this is not issue but a question, I read the scripts in inference.py and found real-esrgan 2x is loaded from trt engine file, since real-2x uses dynamic shapes as input, could you share any ideas how to convert this model to trt, thanks!

    opened by deism 3
  • ESRGAN with full episode

    ESRGAN with full episode

    Hello,

    I'm trying to upscale MKV files of full episodes with ESRGAN. I tried using vspipe -c y4m inference.py - | ffmpeg -i pipe: example.mkv, and it seems to run up to the point where it starts to give an ETA. Once there the time doesn't move and eventually, it says it was killed.

    Can you give me some tips on how to make this work better? I'm not familiar with most of the tools I've been given.

    opened by Ultramonte 2
  • [SUGGESTION] per-scene processing

    [SUGGESTION] per-scene processing

    Hi there, this project is awesome so thanks for your - voluntary - work !

    Since GANs-based processing is quite heavy computing task, it could be very useful to split it into multiple "segments" to allow parallel/scalable/collaborative/resumable instances.

    We suggest you to check @master-of-zen's Av1an framework, wich implements it.

    Hope that inspires.

    opened by forart 1
Releases(models)
Owner
I like Google Colab and Python.
Official Implementation of 'UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers' ICLR 2021(spotlight)

UPDeT Official Implementation of UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers (ICLR 2021 spotlight) The

hhhusiyi 96 Dec 22, 2022
An Inverse Kinematics library aiming performance and modularity

IKPy Demo Live demos of what IKPy can do (click on the image below to see the video): Also, a presentation of IKPy: Presentation. Features With IKPy,

Pierre Manceron 481 Jan 02, 2023
Multi-task Multi-agent Soft Actor Critic for SMAC

Multi-task Multi-agent Soft Actor Critic for SMAC Overview The CARE formulti-task: Multi-Task Reinforcement Learning with Context-based Representation

RuanJingqing 8 Sep 30, 2022
Implementation of "Learning to Match Features with Seeded Graph Matching Network" ICCV2021

SGMNet Implementation PyTorch implementation of SGMNet for ICCV'21 paper "Learning to Match Features with Seeded Graph Matching Network", by Hongkai C

87 Dec 11, 2022
A graph-to-sequence model for one-step retrosynthesis and reaction outcome prediction.

Graph2SMILES A graph-to-sequence model for one-step retrosynthesis and reaction outcome prediction. 1. Environmental setup System requirements Ubuntu:

29 Nov 18, 2022
根据midi文件演奏“风物之诗琴”的脚本 "Windsong Lyre" auto play

Genshin-lyre-auto-play 简体中文 | English 简介 根据midi文件演奏“风物之诗琴”的脚本。由Python驱动,在此承诺, ⚠️ 项目内绝不含任何能够引起安全问题的代码。 前排提示:所有键盘在动但是原神没反应的都是因为没有管理员权限,双击run.bat或者以管理员模式

御坂17032号 386 Jan 01, 2023
STEAL - Learning Semantic Boundaries from Noisy Annotations (CVPR 2019)

STEAL This is the official inference code for: Devil Is in the Edges: Learning Semantic Boundaries from Noisy Annotations David Acuna, Amlan Kar, Sanj

469 Dec 26, 2022
AVD Quickstart Containerlab

AVD Quickstart Containerlab WARNING This repository is still under construction. It's fully functional, but has number of limitations. For example: RE

Carl Buchmann 3 Apr 10, 2022
Research using Cirq!

ReCirq Research using Cirq! This project contains modules for running quantum computing applications and experiments through Cirq and Quantum Engine.

quantumlib 230 Dec 29, 2022
[AAAI 2021] MVFNet: Multi-View Fusion Network for Efficient Video Recognition

MVFNet: Multi-View Fusion Network for Efficient Video Recognition (AAAI 2021) Overview We release the code of the MVFNet (Multi-View Fusion Network).

Wenhao Wu 114 Nov 27, 2022
Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

Jiachen Xu 5 Jul 14, 2022
A python library for face detection and features extraction based on mediapipe library

FaceAnalyzer A python library for face detection and features extraction based on mediapipe library Introduction FaceAnalyzer is a library based on me

Saifeddine ALOUI 14 Dec 30, 2022
Personal thermal comfort models using digital twins: Preference prediction with BIM-extracted spatial-temporal proximity data from Build2Vec

Personal thermal comfort models using digital twins: Preference prediction with BIM-extracted spatial-temporal proximity data from Build2Vec This repo

Building and Urban Data Science (BUDS) Group 5 Dec 02, 2022
Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation

DistMIS Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation. DistriMIS Distributing Deep Learning Hyperparameter Tuning

HiEST 2 Sep 09, 2022
PyGCL: Graph Contrastive Learning Library for PyTorch

PyGCL: Graph Contrastive Learning for PyTorch PyGCL is an open-source library for graph contrastive learning (GCL), which features modularized GCL com

GCL: Graph Contrastive Learning Library for PyTorch 594 Jan 08, 2023
Simulating an AI playing 2048 using the Expectimax algorithm

2048-expectimax Simulating an AI playing 2048 using the Expectimax algorithm The base game engine uses code from here. The AI player is modeled as a m

Subha Ramesh 2 Jan 31, 2022
Pynomial - a lightweight python library for implementing the many confidence intervals for the risk parameter of a binomial model

Pynomial - a lightweight python library for implementing the many confidence intervals for the risk parameter of a binomial model

Demetri Pananos 9 Oct 04, 2022
Fast mesh denoising with data driven normal filtering using deep variational autoencoders

Fast mesh denoising with data driven normal filtering using deep variational autoencoders This is an implementation for the paper entitled "Fast mesh

9 Dec 02, 2022
🗺 General purpose U-Network implemented in Keras for image segmentation

TF-Unet General purpose U-Network implemented in Keras for image segmentation Getting started • Training • Evaluation Getting started Looking for Jupy

Or Fleisher 2 Aug 31, 2022
IOT: Instance-wise Layer Reordering for Transformer Structures

Introduction This repository contains the code for Instance-wise Ordered Transformer (IOT), which is introduced in the ICLR2021 paper IOT: Instance-wi

IOT 19 Nov 15, 2022