Manifold Alignment for Semantically Aligned Style Transfer

Related tags

Deep LearningMAST
Overview

Manifold Alignment for Semantically Aligned Style Transfer

[Paper]

res1 GUI Demo

Getting Started

MAST has been tested on CentOS 7.6 with python >= 3.6. It supports both GPU and CPU inference. If you don't have a suitable device, try running our Colab demo.

Clone the repo:

git clone https://github.com/NJUHuoJing/MAST.git

prepare the checkpoints:

cd MAST
chmod 777 scripts/prepare_data.sh
scripts/prepare_data.sh

Install the requirements:

conda create -n mast-env python=3.6
conda activate mast-env
pip install -r requirements.txt

# If you want to use post smoothing as the same as PhotoWCT, then install the requirements below;
# You can also just skip it to use fast post smoothing, remember to change cfg.TEST.PHOTOREALISTIC.FAST_SMOOTHING=true
pip install -U setuptools
pip install cupy
pip install pynvrtc

Running the Demo

Artistic style transfer

First set MAST_CORE.ORTHOGONAL_CONSTRAINT=false in configs/config.yaml. Then use the script test_artistic.py to generate the artistic stylized image by following the command below:

# not use seg
python test_artistic.py --cfg_path configs/config.yaml --content_path data/default/content/4.png --style_path data/default/style/4.png --output_dir results/test/default

# use --content_seg_path and --style_seg_path to user edited style transfer
python test_artistic.py --cfg_path configs/config.yaml --content_path data/default/content/4.png --style_path data/default/style/4.png --output_dir results/test/default --content_seg_path data/default/content_segmentation/4.png --style_seg_path data/default/style_segmentation/4.png --seg_type labelme --resize 512

Photo-realistic style transfer

First set MAST_CORE.ORTHOGONAL_CONSTRAINT=true in configs/config.yaml. Then use the script test_photorealistic.py to generate the photo-realistic stylized image by following the command below:

# not use seg
python test_photorealistic.py --cfg_path configs/config.yaml --content_path data/photo_data/content/in1.png --style_path data/photo_data/style/tar1.png --output_dir results/test/photo --resize 512

# or use --content_seg_path and --style_seg_path to user edited style transfer
python test_photorealistic.py --cfg_path configs/config.yaml --content_path data/photo_data/content/in1.png --style_path data/photo_data/style/tar1.png --output_dir results/test/photo --content_seg_path data/photo_data/content_segmentation/in1.png --style_seg_path data/photo_data/style_segmentation/tar1.png --seg_type dpst --resize 512

GUI For Artistic style transfer and User Editing

We provide a gui for user-controllable artistic image stylization. Just use the command below to run test_gui.py

python test_gui.py --cfg_path configs/config.yaml

Features

  1. You can use different colors to control the style transfer in different semantic areas.
  2. The button Expand and Expand num respectively control whether to expand the selected semantic area and the degree of expansion.

See the gif demo for more details.

Google Colab

If you do not have a suitable environment to run this project then you could give Google Colab a try. It allows you to run the project in the cloud, free of charge. You may try our Colab demo using the notebook we have prepared: Colab Demo

Citation

@inproceedings{huo2021manifold,
    author = {Jing Huo and Shiyin Jin and Wenbin Li and Jing Wu and Yu-Kun Lai and Yinghuan Shi and Yang Gao},
    title = {Manifold Alignment for Semantically Aligned Style Transfer},
    booktitle = {IEEE International Conference on Computer Vision},
    pages     = {14861-14869},
    year = {2021}
}

References

  • The post smoothing module is borrowed from PhotoWCT
Autoregressive Predictive Coding: An unsupervised autoregressive model for speech representation learning

Autoregressive Predictive Coding This repository contains the official implementation (in PyTorch) of Autoregressive Predictive Coding (APC) proposed

iamyuanchung 173 Dec 18, 2022
Framework for Spectral Clustering on the Sparse Coefficients of Learned Dictionaries

Dictionary Learning for Clustering on Hyperspectral Images Overview Framework for Spectral Clustering on the Sparse Coefficients of Learned Dictionari

Joshua Bruton 6 Oct 25, 2022
A toolset for creating Qualtrics-based IAT experiments

Qualtrics IAT Tool A web app for generating the Implicit Association Test (IAT) running on Qualtrics Online Web App The app is hosted by Streamlit, a

0 Feb 12, 2022
City-Scale Multi-Camera Vehicle Tracking Guided by Crossroad Zones Code

City-Scale Multi-Camera Vehicle Tracking Guided by Crossroad Zones Requirements Python 3.8 or later with all requirements.txt dependencies installed,

88 Dec 12, 2022
An official implementation of the Anchor DETR.

Anchor DETR: Query Design for Transformer-Based Detector Introduction This repository is an official implementation of the Anchor DETR. We encode the

MEGVII Research 276 Dec 28, 2022
Real-time Neural Representation Fusion for Robust Volumetric Mapping

NeuralBlox: Real-Time Neural Representation Fusion for Robust Volumetric Mapping Paper | Supplementary This repository contains the implementation of

ETHZ ASL 106 Dec 24, 2022
TensorFlow implementation of Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction)

Barlow-Twins-TF This repository implements Barlow Twins (Barlow Twins: Self-Supervised Learning via Redundancy Reduction) in TensorFlow and demonstrat

Sayak Paul 36 Sep 14, 2022
Pytorch implementation of our paper under review -- 1xN Pattern for Pruning Convolutional Neural Networks

1xN Pattern for Pruning Convolutional Neural Networks (paper) . This is Pytorch re-implementation of "1xN Pattern for Pruning Convolutional Neural Net

Mingbao Lin (林明宝) 29 Nov 29, 2022
abess: Fast Best-Subset Selection in Python and R

abess: Fast Best-Subset Selection in Python and R Overview abess (Adaptive BEst Subset Selection) library aims to solve general best subset selection,

297 Dec 21, 2022
Simulation-based performance analysis of server-less Blockchain-enabled Federated Learning

Blockchain-enabled Server-less Federated Learning Repository containing the files used to reproduce the results of the publication "Blockchain-enabled

Francesc Wilhelmi 9 Sep 27, 2022
Official implementation of Deep Burst Super-Resolution

Deep-Burst-SR Official implementation of Deep Burst Super-Resolution Publication: Deep Burst Super-Resolution. Goutam Bhat, Martin Danelljan, Luc Van

Goutam Bhat 113 Dec 19, 2022
Space Invaders For Python

Space-Invaders Just download or clone the git repository. To run the Space Invader game you need to have pyhton installed in you system. If you dont h

Fei 5 Jul 27, 2022
Code to reproduce the results in "Visually Grounded Reasoning across Languages and Cultures", EMNLP 2021.

marvl-code [WIP] This is the implementation of the approaches described in the paper: Fangyu Liu*, Emanuele Bugliarello*, Edoardo M. Ponti, Siva Reddy

25 Nov 15, 2022
Implementation of ReSeg using PyTorch

Implementation of ReSeg using PyTorch ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation Pascal-Part Annotations Pascal VOC 2010

Onur Kaplan 46 Nov 23, 2022
Plugin for Gaffer providing direct acess to asset from PolyHaven.com. Only HDRIs at the moment, Cycles and Arnold supported

GafferHaven Plugin for Gaffer providing direct acess to asset from PolyHaven.com. Only HDRIs are supported at the moment, in Cycles and Arnold lights.

Jakub Vondra 6 Jan 26, 2022
Arbitrary Distribution Modeling with Censorship in Real Time 59 2 60 3 Bidding Advertising for KDD'21

Arbitrary_Distribution_Modeling This repo implements the Neighborhood Likelihood Loss (NLL) and Arbitrary Distribution Modeling (ADM, with Interacting

7 Jan 03, 2023
You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks.

AllSet This is the repo for our paper: You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks. We prepared all codes and a subse

Jianhao 51 Dec 24, 2022
[CVPR2021 Oral] UP-DETR: Unsupervised Pre-training for Object Detection with Transformers

UP-DETR: Unsupervised Pre-training for Object Detection with Transformers This is the official PyTorch implementation and models for UP-DETR paper: @a

dddzg 430 Dec 23, 2022
Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts

t5-japanese Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts. The following is a list of models that

Kimio Kuramitsu 1 Dec 13, 2021
Generative Art Using Neural Visual Grammars and Dual Encoders

Generative Art Using Neural Visual Grammars and Dual Encoders Arnheim 1 The original algorithm from the paper Generative Art Using Neural Visual Gramm

DeepMind 231 Jan 05, 2023