Facial Image Inpainting with Semantic Control

Overview

Facial Image Inpainting with Semantic Control

In this repo, we provide a model for the controllable facial image inpainting task. This model enables users to intuitively edit their images by using parametric 3D faces.

The technology report is comming soon.

  • Image Inpainting results

  • Fine-grained Control

Quick Start

Installation

  • Clone the repository and set up a conda environment with all dependencies as follows
git clone https://github.com/RenYurui/Controllable-Face-Inpainting.git --recursive
cd Controllable-Face-Inpainting

# 1. Create a conda virtual environment.
conda create -n cfi python=3.6
source activate cfi
conda install -c pytorch pytorch=1.7.1 torchvision cudatoolkit=10.2

# 2. install pytorch3d
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install -c bottler nvidiacub
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d && pip install -e .

# 3. Install other dependencies
pip install -r requirements.txt

Download Prerequisite Models

  • Follow Deep3DFaceRecon to prepare ./BFM folder. Download 01_MorphableModel.mat and Expression Basis Exp_Pca.bin. Put the obtained files into the ./Deep3DFaceRecon_pytorch/BFM floder. Then link the folder to the root path.
ln -s /PATH_TO_REPO_ROOT/Deep3DFaceRecon_pytorch/BFM /PATH_TO_REPO_ROOT
  • Clone the Arcface repo
cd third_part
git clone https://github.com/deepinsight/insightface.git
cp -r ./insightface/recognition/arcface_torch/ ./

The Arcface is used to extract identity features for loss computation. Download the pre-trained model from Arcface using this link. By default, the resnet50 backbone (ms1mv3_arcface_r50_fp16) is used. Put the obtained weights into ./third_part/arcface_torch/ms1mv3_arcface_r50_fp16/backbone.pth

  • Download the pretrained weights of our model from Google Driven. Save the obtained files into folder ./result.

Inference

We provide some example images. Please run the following code for inference

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --nproc_per_node=1 --master_port 1234 demo.py \
--config ./config/facial_image_renderer_ffhq.yaml \
--name facial_image_renderer_ffhq \
--output_dir ./visi_result \
--input_dir ./examples/inputs \
--mask_dir ./examples/masks

Train the model from scratch

Dataset Preparation

  • Download dataset. We use Celeba-HQ and FFHQ for training and inference. Please download the datasets (image format) and put them under ./dataset folder.
  • Obtain 3D faces by using Deep3DFaceRecon. Follow the Deep3DFaceRecon repo to download the trained weights. And save it as: ./Deep3DFaceRecon_pytorch/checkpoints/face_recon/epoch_20.pth
# 1. Extract keypoints from the face images for cropping.
cd scripts
# extracted keypoints from celeba
python extract_kp.py \
--data_root PATH_TO_CELEBA_ROOT \
--output_dir PATH_TO_KEYPOINTS \
--dataset celeba \
--device_ids 0,1 \
--workers 6

# 2. Extract 3DMM coefficients from the face images.
cd .. #repo root
# we provide some scripts for easy of use. However, one can use the original repo to extract the coefficients.
cp scripts/inference_options.py ./Deep3DFaceRecon_pytorch/options
cp scripts/face_recon.py ./Deep3DFaceRecon_pytorch
cp scripts/facerecon_inference_model.py ./Deep3DFaceRecon_pytorch/models
cp scripts/pytorch_3d.py ./Deep3DFaceRecon_pytorch/util
ln -s /PATH_TO_REPO_ROOT/third_part/arcface_torch /PATH_TO_REPO_ROOT/Deep3DFaceRecon_pytorch/models

cd Deep3DFaceRecon_pytorch

python face_recon.py \
--input_dir PATH_TO_CELEBA_ROOT \
--keypoint_dir PATH_TO_KEYPOINTS \
--output_dir PATH_TO_3DMM_COEFFICIENT \
--inference_batch_size 100 \
--name=face_recon \
--dataset_name celeba \
--epoch=20 \
--model facerecon_inference

# 3. Save images and the coefficients into a lmdb file.
cd .. #repo root
python prepare_data.py \
--root PATH_TO_CELEBA_ROOT \
--coeff_file PATH_TO_3DMM_COEFFICIENT \
--dataset celeba \
--out PATH_TO_CELEBA_LMDB_ROOT

Train The Model

# we first train the semantic_descriptor_recommender
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 --master_port 1234 train.py \
--config ./config/semantic_descriptor_recommender_celeba.yaml \
--name semantic_descriptor_recommender_celeba

# Then, we trian the facial_image_renderer for image inpainting
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 --master_port 1234 train.py \
--config ./config/facial_image_renderer_celeba.yaml \
--name facial_image_renderer_celeba
Owner
Ren Yurui
Ren Yurui
[ICML 2021] "Graph Contrastive Learning Automated" by Yuning You, Tianlong Chen, Yang Shen, Zhangyang Wang

Graph Contrastive Learning Automated PyTorch implementation for Graph Contrastive Learning Automated [talk] [poster] [appendix] Yuning You, Tianlong C

Shen Lab at Texas A&M University 80 Nov 23, 2022
DeconvNet : Learning Deconvolution Network for Semantic Segmentation

DeconvNet: Learning Deconvolution Network for Semantic Segmentation Created by Hyeonwoo Noh, Seunghoon Hong and Bohyung Han at POSTECH Acknowledgement

Hyeonwoo Noh 325 Oct 20, 2022
A lightweight library to compare different PyTorch implementations of the same network architecture.

TorchBug is a lightweight library designed to compare two PyTorch implementations of the same network architecture. It allows you to count, and compar

Arjun Krishnakumar 5 Jan 02, 2023
CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification (ICCV2021)

CM-NAS Official Pytorch code of paper CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification in ICCV2021. Vis

JDAI-CV 40 Nov 25, 2022
This is an (re-)implementation of DeepLab-ResNet in TensorFlow for semantic image segmentation on the PASCAL VOC dataset.

DeepLab-ResNet-TensorFlow This is an (re-)implementation of DeepLab-ResNet in TensorFlow for semantic image segmentation on the PASCAL VOC dataset. Up

19 Jan 16, 2022
The official homepage of the (outdated) COCO-Stuff 10K dataset.

COCO-Stuff 10K dataset v1.1 (outdated) Holger Caesar, Jasper Uijlings, Vittorio Ferrari Overview Welcome to official homepage of the COCO-Stuff [1] da

Holger Caesar 263 Dec 11, 2022
End-to-end beat and downbeat tracking in the time domain.

WaveBeat End-to-end beat and downbeat tracking in the time domain. | Paper | Code | Video | Slides | Setup First clone the repo. git clone https://git

Christian J. Steinmetz 60 Dec 24, 2022
Metadata-Extractor - Metadata Extractor Script can be used to read in exif metadata

Metadata Extractor The exifextract script can be used to read in exif metadata f

1 Feb 16, 2022
yolov5目标检测模型的知识蒸馏(基于响应的蒸馏)

代码地址: https://github.com/Sharpiless/yolov5-knowledge-distillation 教师模型: python train.py --weights weights/yolov5m.pt \ --cfg models/yolov5m.ya

52 Dec 04, 2022
This is an official pytorch implementation of Fast Fourier Convolution.

Fast Fourier Convolution (FFC) for Image Classification This is the official code of Fast Fourier Convolution for image classification on ImageNet. Ma

pkumi 199 Jan 03, 2023
Implementation of Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

advantage-weighted-regression Implementation of Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning, by Peng et al. (

Omar D. Domingues 1 Dec 02, 2021
Manifold-Mixup implementation for fastai V2

Manifold Mixup Unofficial implementation of ManifoldMixup (Proceedings of ICML 19) for fast.ai (V2) based on Shivam Saboo's pytorch implementation of

Nestor Demeure 16 Jul 25, 2022
This is the official repository for our paper: ''Pruning Self-attentions into Convolutional Layers in Single Path''.

Pruning Self-attentions into Convolutional Layers in Single Path This is the official repository for our paper: Pruning Self-attentions into Convoluti

Zhuang AI Group 77 Dec 26, 2022
A complete speech segmentation system using Kaldi and x-vectors for voice activity detection (VAD) and speaker diarisation.

bbc-speech-segmenter: Voice Activity Detection & Speaker Diarization A complete speech segmentation system using Kaldi and x-vectors for voice activit

BBC 16 Oct 27, 2022
Deeprl - Standard DQN and dueling network for simple games

DeepRL This code implements the standard deep Q-learning and dueling network with experience replay (memory buffer) for playing simple games. DQN algo

Yao Zhou 6 Apr 12, 2020
Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation

Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation

Bae, Gwangbin 95 Jan 04, 2023
Visyerres sgdf woob - Modules Woob pour l'intranet et autres sites Scouts et Guides de France

Vis'Yerres SGDF - Modules Woob Vous avez le sentiment que l'intranet des Scouts

Thomas Touhey (pas un pseudonyme) 3 Dec 24, 2022
ECCV2020 paper: Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards. Code and Data.

This repo contains some of the codes for the following paper Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards. Code

Xuewen Yang 56 Dec 08, 2022
A really easy-to-use and powerful sudoku solver.

SodukuSolver This is a really useful sudoku solver with a Qt gui. USAGE Enter the numbers in and click "RUN"! If you don't want to wait, simply press

Ujhhgtg Teams 11 Jun 02, 2022