Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Related tags

Deep Learningsuper-resolutioniccv2021image-rescalingnormalization-flowface-image-super-resolution
Overview

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

This repository is the official PyTorch implementation of Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (arxiv, supp).

🚀 🚀 🚀 News:

Normalizing flows have recently demonstrated promising results for low-level vision tasks. For image super-resolution (SR), it learns to predict diverse photo-realistic high-resolution (HR) images from the low-resolution (LR) image rather than learning a deterministic mapping. For image rescaling, it achieves high accuracy by jointly modelling the downscaling and upscaling processes. While existing approaches employ specialized techniques for these two tasks, we set out to unify them in a single formulation. In this paper, we propose the hierarchical conditional flow (HCFlow) as a unified framework for image SR and image rescaling. More specifically, HCFlow learns a bijective mapping between HR and LR image pairs by modelling the distribution of the LR image and the rest high-frequency component simultaneously. In particular, the high-frequency component is conditional on the LR image in a hierarchical manner. To further enhance the performance, other losses such as perceptual loss and GAN loss are combined with the commonly used negative log-likelihood loss in training. Extensive experiments on general image SR, face image SR and image rescaling have demonstrated that the proposed HCFlow achieves state-of-the-art performance in terms of both quantitative metrics and visual quality.

         

Requirements

  • Python 3.7, PyTorch == 1.7.1
  • Requirements: opencv-python, lpips, natsort, etc.
  • Platforms: Ubuntu 16.04, cuda-11.0
cd HCFlow-master
pip install -r requirements.txt

Quick Run (takes 1 Minute)

To run the code with one command (without preparing data), run this command:

cd codes
# face image SR
python test_HCFLow.py --opt options/test/test_SR_CelebA_8X_HCFlow.yml

# general image SR
python test_HCFLow.py --opt options/test/test_SR_DF2K_4X_HCFlow.yml

# image rescaling
python test_HCFLow.py --opt options/test/test_Rescaling_DF2K_4X_HCFlow.yml

Data Preparation

The framework of this project is based on MMSR and SRFlow. To prepare data, put training and testing sets in ./datasets as ./datasets/DIV2K/HR/0801.png. Commonly used SR datasets can be downloaded here. There are two ways for accerleration in data loading: First, one can use ./scripts/png2npy.py to generate .npy files and use data/GTLQnpy_dataset.py. Second, one can use .pklv4 dataset (recommended) and use data/LRHR_PKL_dataset.py. Please refer to SRFlow for more details. Prepared datasets can be downloaded here.

Training

To train HCFlow for general image SR/ face image SR/ image rescaling, run this command:

cd codes

# face image SR
python train_HCFLow.py --opt options/train/train_SR_CelebA_8X_HCFlow.yml

# general image SR
python train_HCFLow.py --opt options/train/train_SR_DF2K_4X_HCFlow.yml

# image rescaling
python train_HCFLow.py --opt options/train/train_Rescaling_DF2K_4X_HCFlow.yml

All trained models can be downloaded from here.

Testing

Please follow the Quick Run section. Just modify the dataset path in test_HCFlow_*.yml.

Results

We achieved state-of-the-art performance on general image SR, face image SR and image rescaling.

For more results, please refer to the paper and supp for details.

Citation

@inproceedings{liang21hcflow,
  title={Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling},
  author={Liang, Jingyun and Lugmayr, Andreas and Zhang, Kai and Danelljan, Martin and Van Gool, Luc and Timofte, Radu},
  booktitle={IEEE Conference on International Conference on Computer Vision},
  year={2021}
}

License & Acknowledgement

This project is released under the Apache 2.0 license. The codes are based on MMSR, SRFlow, IRN and Glow-pytorch. Please also follow their licenses. Thanks for their great works.

Comments
  • Testing without GT

    Testing without GT

    Is there a way to run the test without GT? I just want to infer the model. I found a mode called LQ which -I think- should only load the images in LR directory. But this mode gives me the error: assert real_crop * self.opt['scale'] * 2 > self.opt['kernel_size'] TypeError: '>' not supported between instances of 'int' and 'NoneType'

    in LQ_dataset.py", line 88

    solved ✅ 
    opened by AhmedHashish123 4
  • Add Docker environment & web demo

    Add Docker environment & web demo

    Hey @JingyunLiang !👋

    This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

    This also means we can make a web page where other people can try out your model! View it here: https://replicate.ai/jingyunliang/hcflow-sr, which currently supports Image Super-Resolution.

    Claim your page here so you can edit it, and we'll feature it on our website and tweet about it too.

    In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

    opened by chenxwh 2
  • The code implementation and the paper description seem different

    The code implementation and the paper description seem different

    Hi, your work is excellent, but there is one thing I don't understand.

    What is written in the paper is:

    "A diagonal covariance matrix with all diagonal elements close to zero"

    But the code implementation in HCFlowNet_SR_arch.py line 64 is: basic. Gaussian diag.logp (LR, - torch. Ones_ like(lr)*6, fake_ lr_ from_ hr)

    why use - torch. Ones_ like(lr)*6 as covariance matrix? This seems to be inconsistent with the description in the paper

    opened by xmyhhh 2
  • environment

    environment

    ImportError: /home/hbw/gcc-build-5.4.0/lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /home/hbw/anaconda3/lib/python3.8/site-packages/scipy/fft/_pocketfft/pypocketfft.cpython-38-x86_64-linux-gnu.so)

    Is this error due to my GCC version being too low, and your version is? looking forward to your reply!

    opened by hbw945 2
  • Code versions of BRISQUE and NIQE used in paper

    Code versions of BRISQUE and NIQE used in paper

    Hi, I have run performance tests with the Matlab versions of the NIQE and BRISQUE codes and found deviations from the values reported in the paper. Could you please provide a link to the code you used? thanks a lot~

    solved ✅ 
    opened by xmyhhh 1
  • Update on Replicate demo

    Update on Replicate demo

    Hello again @JingyunLiang :),

    This pull request does a few little things:

    • Updated the demo link with an icon in README as you suggested
    • A bugfix for cleaning temporary directory on cog

    We have added more functionality to the Example page of your model, now you can add and delete to customise the example gallery as you like (as the owner of the page)

    Also, you could run cog push if you like to update the model of any other models on replicate in the future 😄

    opened by chenxwh 1
  • About training and inference time?

    About training and inference time?

    Thanks for your nice work!

    I want to know how much time do you need to train and inference with your models.

    Furthermore, will information about params / FLOPs be reported?

    Thanks.

    solved ✅ 
    opened by TiankaiHang 1
  • RuntimeError: The size of tensor a (20) must match the size of tensor b (40) at non-singleton dimension 3

    RuntimeError: The size of tensor a (20) must match the size of tensor b (40) at non-singleton dimension 3

    Hi, I've encountered the error when I trained the HCFlowNet. I changed my ".png" dataset to ".pklv4" dataset. I was trained on the platform of windows 10 with 1 single GPU. Could you please help me find the error? Thanks a lot.

    opened by William9Baker 0
  • How to build an invertible mapping between two variables whose dimensions are different ?

    How to build an invertible mapping between two variables whose dimensions are different ?

    Maybe this is a stupid question, but I have been puzzled for quite a long time. In the image super-resolution task, the input and output have different dimensions. How to build an invertible mapping between them? I notice that you calculate the determinant of the Jacobian, so I thought the mapping here is strictly invertible?

    opened by Wangbk-dl 0
  • How to make an invertible mapping between two variables whose dimensions are different ?

    How to make an invertible mapping between two variables whose dimensions are different ?

    Maybe this is a stupid question, but I have been puzzled for quite a long time. In the image super-resolution task, the input and output have different dimensions. How to build such an invertible mapping between them ? Take an example: If I have a low-resolution(LR) image x, and I have had an invertible function G. I can feed LR image x into G, and generate an HR image y. But can you ensure that we could obtain an output the same as x when we feed y into G_inverse?

    y = G(x) x' = G_inverse(y) =? x

    I would appreciate it if you could offer some help.

    opened by Wangbk-dl 0
  • New Super-Resolution Benchmarks

    New Super-Resolution Benchmarks

    Hello,

    MSU Graphics & Media Lab Video Group has recently launched two new Super-Resolution Benchmarks.

    If you are interested in participating, you can add your algorithm following the submission steps:

    We would be grateful for your feedback on our work!

    opened by EvgeneyBogatyrev 0
  • Why NLL is negative during the training？

    Why NLL is negative during the training？

    Great work! During the training process, we found that the output NLL is negative. But theoretically, NLL should be positive. Is there any explanation for this?

    opened by IMSEMZPZ 0
Owner
Jingyun Liang
PhD Student at Computer Vision Lab, ETH Zurich
Jingyun Liang
GitHub Repository https://arxiv.org/abs/2108.05301
Dyalog-apl-docset - Dyalog APL Dash Docset Generator

Dyalog APL Dash Docset Generator o alasa e kili sona kepeken tenpo lili a A Dash

Maciej Goszczycki 1 Jan 10, 2022
Curved Projection Reformation

Description Assuming that we already know the image of the centerline, we want the lumen to be displayed on a plane, which requires curved projection

夜听残荷 5 Sep 11, 2022
Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.

Machine Learning From Scratch About Python implementations of some of the fundamental Machine Learning models and algorithms from scratch. The purpose

Erik Linder-Norén 21.8k Jan 09, 2023
MVP Benchmark for Multi-View Partial Point Cloud Completion and Registration

MVP Benchmark: Multi-View Partial Point Clouds for Completion and Registration [NEWS] 2021-07-12 [NEW 🎉 ] The submission on Codalab starts! 2021-07-1

PL 93 Dec 21, 2022
InDuDoNet+: A Model-Driven Interpretable Dual Domain Network for Metal Artifact Reduction in CT Images

InDuDoNet+: A Model-Driven Interpretable Dual Domain Network for Metal Artifact Reduction in CT Images Hong Wang, Yuexiang Li, Haimiao Zhang, Deyu Men

Hong Wang 4 Dec 27, 2022
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

The Apache Software Foundation 20.2k Jan 08, 2023
PyGCL: Graph Contrastive Learning Library for PyTorch

PyGCL: Graph Contrastive Learning for PyTorch PyGCL is an open-source library for graph contrastive learning (GCL), which features modularized GCL com

GCL: Graph Contrastive Learning Library for PyTorch 594 Jan 08, 2023
Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift (ICCV 2021)

Π-NAS This repository provides the evaluation code of our submitted paper: Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training

Jiqi Zhang 18 Aug 18, 2022
Voice Conversion Using Speech-to-Speech Neuro-Style Transfer

This repo contains the official implementation of the VAE-GAN from the INTERSPEECH 2020 paper Voice Conversion Using Speech-to-Speech Neuro-Style Transfer.

Ehab AlBadawy 93 Jan 05, 2023
Official implementation for "Low-light Image Enhancement via Breaking Down the Darkness"

Low-light Image Enhancement via Breaking Down the Darkness by Qiming Hu, Xiaojie Guo. 1. Dependencies Python3 PyTorch=1.0 OpenCV-Python, TensorboardX

Qiming Hu 30 Jan 01, 2023
Self-Supervised Methods for Noise-Removal

SSMNR | Self-Supervised Methods for Noise Removal Image denoising is the task of removing noise from an image, which can be formulated as the task of

1 Jan 16, 2022
Implementation of PyTorch-based multi-task pre-trained models

mtdp Library containing implementation related to the research paper "Multi-task pre-training of deep neural networks for digital pathology" (Mormont

Romain Mormont 27 Oct 14, 2022
[ACM MM 2019 Oral] Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation

Contents Cycle-In-Cycle GANs Installation Dataset Preparation Generating Images Using Pretrained Model Train and Test New Models Acknowledgments Relat

Hao Tang 67 Dec 14, 2022
Code for "Modeling Indirect Illumination for Inverse Rendering", CVPR 2022

Modeling Indirect Illumination for Inverse Rendering Project Page | Paper | Data Preparation Set up the python environment conda create -n invrender p

ZJU3DV 116 Jan 03, 2023
Pytorch implementation for "Density-aware Chamfer Distance as a Comprehensive Metric for Point Cloud Completion" (NeurIPS 2021)

Density-aware Chamfer Distance This repository contains the official PyTorch implementation of our paper: Density-aware Chamfer Distance as a Comprehe

Tong WU 93 Dec 15, 2022
Bunch of different tools which helps visualizing and annotating images for semantic/instance segmentation tasks

Data Framework for Semantic/Instance Segmentation Bunch of different tools which helps visualizing, transforming and annotating images for semantic/in

Bruno Fernandes Carvalho 5 Dec 21, 2022
Robbing the FED: Directly Obtaining Private Data in Federated Learning with Modified Models

Robbing the FED: Directly Obtaining Private Data in Federated Learning with Modified Models This repo contains a barebones implementation for the atta

16 Dec 04, 2022
Official implementation for Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videos

Multi-modal Interaction Graph Convolutioal Network for Temporal Language Localization in Videos Official implementation for Multi-Modal Interaction Gr

Zongmeng Zhang 15 Oct 18, 2022
Tensorflow implementation of Semi-supervised Sequence Learning (https://arxiv.org/abs/1511.01432)

Transfer Learning for Text Classification with Tensorflow Tensorflow implementation of Semi-supervised Sequence Learning(https://arxiv.org/abs/1511.01

DONGJUN LEE 82 Oct 22, 2022
Predictive AI layer for existing databases.

MindsDB is an open-source AI layer for existing databases that allows you to effortlessly develop, train and deploy state-of-the-art machine learning

MindsDB Inc 12.2k Jan 03, 2023