Deep Exemplar-based Video Colorization (Pytorch Implementation)
Paper | Pretrained Model | Youtube video
🔥
| Colab demo
Deep Exemplar-based Video Colorization, CVPR2019
Bo Zhang1,3, Mingming He1,5, Jing Liao2, Pedro V. Sander1, Lu Yuan4, Amine Bermak1, Dong Chen3
1Hong Kong University of Science and Technology,2City University of Hong Kong, 3Microsoft Research Asia, 4Microsoft Cloud&AI, 5USC Institute for Creative Technologies
Prerequisites
- Python 3.6+
- Nvidia GPU + CUDA, CuDNN
Installation
First use the following commands to prepare the environment:
conda create -n ColorVid python=3.6
source activate ColorVid
pip install -r requirements.txt
Then, download the pretrained models from this link, unzip the file and place the files into the corresponding folders:
video_moredata_l1under thecheckpointsfoldervgg19_conv.pthandvgg19_gray.pthunder thedatafolder
Data Preparation
In order to colorize your own video, it requires to extract the video frames, and provide a reference image as an example.
- Place your video frames into one folder, e.g.,
./sample_videos/v32_180 - Place your reference images into another folder, e.g.,
./sample_videos/v32
If you want to automatically retrieve color images, you can try the retrieval algorithm from this link which will retrieve similar images from the ImageNet dataset. Or you can try this link on your own image database.
Test
python test.py --image-size [image-size] \
--clip_path [path-to-video-frames] \
--ref_path [path-to-reference] \
--output_path [path-to-output]
We provide several sample video clips with corresponding references. For example, one can colorize one sample legacy video using:
python test.py --clip_path ./sample_videos/clips/v32 \
--ref_path ./sample_videos/ref/v32 \
--output_path ./sample_videos/output
Note that we use 216*384 images for training, which has aspect ratio of 1:2. During inference, we scale the input to this size and then rescale the output back to the original size.
Train
We also provide training code for reference. The training can be started by running:
python --data_root [root of video samples] \
--data_root_imagenet [root of image samples] \
--gpu_ids [gpu ids] \
We do not provide the full video dataset due to the copyright issue. For image samples, we retrieve semantically similar images from ImageNet using this repository. Still, one can refer to our code to understand the detailed procedure of augmenting the image dataset to mimic the video frames.
Comparison with State-of-the-Arts
More results
Citation
If you use this code for your research, please cite our paper.
@inproceedings{zhang2019deep,
title={Deep exemplar-based video colorization},
author={Zhang, Bo and He, Mingming and Liao, Jing and Sander, Pedro V and Yuan, Lu and Bermak, Amine and Chen, Dong},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={8052--8061},
year={2019}
}
Old Photo Restoration
🔥
If you are also interested in restoring the artifacts in the legacy photo, please check our recent work, bringing old photo back to life.
@inproceedings{wan2020bringing,
title={Bringing Old Photos Back to Life},
author={Wan, Ziyu and Zhang, Bo and Chen, Dongdong and Zhang, Pan and Chen, Dong and Liao, Jing and Wen, Fang},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={2747--2757},
year={2020}
}
License
This project is licensed under the MIT license.
6 Aug 18, 2022
58 Dec 28, 2022
45 Dec 25, 2022
36 Dec 05, 2022
44 Dec 03, 2022
407 Dec 15, 2022
21 Dec 22, 2022
50 Jan 04, 2023
1 Jan 22, 2022
10 Dec 26, 2022
9 May 30, 2022
16 Dec 15, 2022
61 Oct 17, 2022
1 Jun 30, 2022
130 Dec 25, 2022
1k Jan 09, 2023
35 Nov 10, 2022
2.3k Jan 02, 2023
121 Dec 24, 2022
6 Dec 04, 2022