HINet: Half Instance Normalization Network for Image Restoration

Last update: Dec 31, 2022

Related tags

Deep Learning HINet

Overview

HINet: Half Instance Normalization Network for Image Restoration

Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, Chengpeng Chen

Paper: https://arxiv.org/abs/2105.06086

In this paper, we explore the role of Instance Normalization in low-level vision tasks. Specifically, we present a novel block: Half Instance Normalization Block (HIN Block), to boost the performance of image restoration networks. Based on HIN Block, we design a simple and powerful multi-stage network named HINet, which consists of two subnetworks. With the help of HIN Block, HINet surpasses the state-of-the-art (SOTA) on various image restoration tasks. For image denoising, we exceed it 0.11dB and 0.28 dB in PSNR on SIDD dataset, with only 7.5% and 30% of its multiplier-accumulator operations (MACs), 6.8 times and 2.9 times speedup respectively. For image deblurring, we get comparable performance with 22.5% of its MACs and 3.3 times speedup on REDS and GoPro datasets. For image deraining, we exceed it by 0.3 dB in PSNR on the average result of multiple datasets with 1.4 times speedup. With HINet, we won 1st place on the NTIRE 2021 Image Deblurring Challenge - Track2. JPEG Artifacts, with a PSNR of 29.70.

Network Architecture

Installation

This implementation based on BasicSR which is a open source toolbox for image/video restoration tasks.

python 3.6.9
pytorch 1.5.1
cuda 10.1

git clone https://github.com/megvii-model/HINet
cd HINet
pip install -r requirements.txt
python setup.py develop --no_cuda_ext

Image Restoration Tasks

Image denoise, deblur, derain.

Image Denoise - SIDD dataset (Coming soon)

Image Deblur - GoPro dataset (Click to expand)

prepare data
- mkdir ./datasets/GoPro
- download the train set in ./datasets/GoPro/train and test set in ./datasets/GoPro/test (refer to MPRNet)
- it should be like:
```
./datasets/
./datasets/GoPro/
./datasets/GoPro/train/
./datasets/GoPro/train/input/
./datasets/GoPro/train/target/
./datasets/GoPro/test/
./datasets/GoPro/test/input/
./datasets/GoPro/test/target/
```
- python scripts/data_preparation/gopro.py
  - crop the train image pairs to 512x512 patches.
eval
- download pretrained model to ./experiments/pretrained_models/HINet-GoPro.pth
- python basicsr/test.py -opt options/test/REDS/HINet-GoPro.yml
train
- python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train.py -opt options/train/GoPro/HINet.yml --launcher pytorch

Image Deblur - REDS dataset (Click to expand)

prepare data
- mkdir ./datasets/REDS
- download the train / val set from train_blur, train_sharp, val_blur, val_sharp to ./datasets/REDS/ and unzip them.
- it should be like
```
./datasets/
./datasets/REDS/
./datasets/REDS/val/
./datasets/REDS/val/val_blur_jpeg/
./datasets/REDS/val/val_sharp/
./datasets/REDS/train/
./datasets/REDS/train/train_blur_jpeg/
./datasets/REDS/train/train_sharp/
```
- python scripts/data_preparation/reds.py
  - flatten the folders and extract 300 validation images.
eval
- download pretrained model to ./experiments/pretrained_models/HINet-REDS.pth
- python basicsr/test.py -opt options/test/REDS/HINet-REDS.yml
train
- python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train.py -opt options/train/REDS/HINet.yml --launcher pytorch

Image Derain - Rain13k dataset (Click to expand)

prepare data

mkdir ./datasets/Rain13k
download the train set and test set (refer to MPRNet)

it should be like

./datasets/
./datasets/Rain13k/
./datasets/Rain13k/train/
./datasets/Rain13k/train/input/
./datasets/Rain13k/train/target/
./datasets/Rain13k/test/
./datasets/Rain13k/test/Test100/
./datasets/Rain13k/test/Rain100H/
./datasets/Rain13k/test/Rain100L/
./datasets/Rain13k/test/Test2800/
./datasets/Rain13k/test/Test1200/

eval
- download pretrained model to ./experiments/pretrained_models/HINet-Rain13k.pth
- For Test100:
  - python basicsr/test.py -opt options/test/Rain13k/HINet-Test100.yml
- For Rain100H
  - python basicsr/test.py -opt options/test/Rain13k/HINet-Rain100H.yml
- For Rain100L
  - python basicsr/test.py -opt options/test/Rain13k/HINet-Rain100L.yml
- For Test2800
  - python basicsr/test.py -opt options/test/Rain13k/HINet-Test2800.yml
- For Test1200
  - python basicsr/test.py -opt options/test/Rain13k/HINet-Test1200.yml
train
- python -m torch.distributed.launch --nproc_per_node=8 --master_port=4321 basicsr/train_rain.py -opt options/train/Rain13k/HINet.yml --launcher pytorch

Results

Some of the following results are higher than the original paper as we optimized some hyper-parameters.

Citations

If HINet helps your research or work, please consider citing HINet.

@inproceedings{chen2021hinet,
  title={HINet: Half Instance Normalization Network for Image Restoration},
  author={Liangyu Chen and Xin Lu and Jie Zhang and Xiaojie Chu and Chengpeng Chen},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  year={2021}
}

Contact

If you have any questions, please contact [email protected] or [email protected] .

HINet: Half Instance Normalization Network for Image Restoration

Related tags

Overview

HINet: Half Instance Normalization Network for Image Restoration

Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, Chengpeng Chen

Paper: https://arxiv.org/abs/2105.06086

Network Architecture

Installation

Image Restoration Tasks

Results

Citations

Contact

Owner

TGS Salt Identification Challenge

[CVPR 2022] Official PyTorch Implementation for "Reference-based Video Super-Resolution Using Multi-Camera Video Triplets"

The repo for the paper "I3CL: Intra- and Inter-Instance Collaborative Learning for Arbitrary-shaped Scene Text Detection".

The official repository for our paper "The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers". We significantly improve the systematic generalization of transformer models on a variety of datasets using simple tricks and careful considerations.

Focal Loss for Dense Rotation Object Detection

Code for "Primitive Representation Learning for Scene Text Recognition" (CVPR 2021)

Federated learning on graph, especially on graph neural networks (GNNs), knowledge graph, and private GNN.

PSTR: End-to-End One-Step Person Search With Transformers (CVPR2022)

A smaller subset of 10 easily classified classes from Imagenet, and a little more French

Code for "Learning the Best Pooling Strategy for Visual Semantic Embedding", CVPR 2021

[CoRL 2021] A robotics benchmark for cross-embodiment imitation.

Specification language for generating Generalized Linear Models (with or without mixed effects) from conceptual models

Riemann Noise Injection With PyTorch

Interpretable and Generalizable Person Re-Identification with Query-Adaptive Convolution and Temporal Lifting

Official PyTorch implementation of "BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation" (NeurIPS 2021)

Towards Understanding Quality Challenges of the Federated Learning: A First Look from the Lens of Robustness

Session-aware Item-combination Recommendation with Transformer Network

A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling

PyTorch implementation of Advantage Actor Critic (A2C), Proximal Policy Optimization (PPO), Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation (ACKTR) and Generative Adversarial Imitation Learning (GAIL).

Video Representation Learning by Recognizing Temporal Transformations. In ECCV, 2020.