Adversarial Framework for (non-) Parametric Image Stylisation Mosaics

Overview

Fully Adversarial Mosaics (FAMOS)

Pytorch implementation of the paper "Copy the Old or Paint Anew? An Adversarial Framework for (non-) Parametric Image Stylization" available at http://arxiv.org/abs/1811.09236.

This code allows to generate image stylisation using an adversarial approach combining parametric and non-parametric elements. Tested to work on Ubuntu 16.04, Pytorch 0.4, Python 3.6. Nvidia GPU p100. It is recommended to have a GPU with 12, 16GB, or more of VRAM.

Parameters

Our method has many possible settings. You can specify them with command-line parameters. The options parser that defines these parameters is in the config.py file and the options are parsed there. You are free to explore them and discover the functionality of FAMOS, which can cover a very broad range of image stylization settings.

There are 5 groups of parameter types:

  • data path and loading parameters
  • neural network parameters
  • regularization and loss criteria weighting parameters
  • optimization parameters
  • parameters of the stochastic noise -- see PSGAN

Update Febr. 2019: video frame-by-frame rendering supported

mosaicGAN.py can now render a whole folder of test images with the trained model. Example videos: lion video with Münich and Berlin

Just specify

python mosaicGAN.py --texturePath=samples/milano/ --contentPath=myFolder/ --testImage=myFolder/ 

with your myFolder and all images from that folder will be rendered by the generator of the GAN. Best to use the same test folder as content folder for training. To use in a video editing pipeline, save all video frames as images with a tool like AVIDEMUX, train FAMOS and save rendered frames, assemble again as video. Note: this my take some time to render thousands of images, you can edit in the code VIDEO_SAVE_FREQ to render the test image folder less frequently.

Update Jan. 2019: new functionality for texture synthesis

Due to interest in a new Pytorch implementation of our last paper "Texture Synthesis with Spatial Generative Adversarial Networks" (PSGAN) we added a script reimplementing it in the current repository. It shares many components with the texture mosaic stylization approach. A difference: PSGAN has no content image and loss, the generator is conditioned only on noise. Example call for texture synthesis:

python PSGAN.py --texturePath=samples/milano/ --ngf=120 --zLoc=50 --ndf=120 --nDep=5 --nDepD=5 --batchSize=16

In general, texture synthesis is much faster than the other methods in this repository, so feel free to add more channels and increase th batchsize. For more details and inspiration how to play with texture synthesis see our old repository with Lasagne code for PSGAN.

Usage: parametric convolutional adversarial mosaic

We provide scripts that have a main loop in which we (i) train an adversarial stylization model and (ii) save images (inference mode). If you need it, you can easily modify the code to save a trained model and load it later to do inference on many other images, see comments at the end of mosaicGAN.py.

In the simplest case, let us start an adversarial mosaic using convolutional networks. All you need is to specify the texture and content folders:

python mosaicGAN.py --texturePath=samples/milano/ --contentPath=samples/archimboldo/

This repository includes sample style files (4 satellite views of Milano, from Google Maps) and a portrait of Archimboldo (from the Google Art Project). Our GAN method will start running and training, occasionally saving results in "results/milano/archimboldo/" and printing the loss values to the terminal. Note that we use the first image found in contentPath as the default full-size output image stylization from FAMOS. You can also specify another image file name testImage to do out-of-sample stylization (inference).

This version uses DCGAN by default, which works nicely for the convolutional GAN we have here. Add the parameter LS for a least squares loss, which also works nicely. Interestingly, WGAN-GP is poorer for our model, which we did not investigate in detail.

If you want to tune the optimisation and model, you can adjust the layers and channels of the Generator and Discriminator, and also choose imageSize and batchSize. All this will effect the speed and performance of the model. You can also tweak the correspondance map cLoss and the content loss weighting fContent

python mosaicGAN.py --texturePath=samples/milano/ --contentPath=samples/archimboldo/ --imageSize=192 --batchSize=8 --ngf=80 --ndf=80  --nDepD=5  --nDep=4 --cLoss=101 --fContent=.6

Other interesting options are skipConnections and Ubottleneck. By disabling the skip connections of the Unet and defining a smaller bottleneck we can reduce the effect of the content image and emphasize more the texture style of the output.

Usage: the full FAMOS approach with parametric and non-parametric aspects

Our method has the property of being able to copy pixels from template images together with the convolutional generation of the previous section.

python mosaicFAMOS.py  --texturePath=samples/milano/ --contentPath=samples/archimboldo/ --N=80 --mirror=True --dIter=2 --WGAN=True

Here we specify N=80 memory templates to copy from. In addition, we use mirror augmentation to get nice kaleidoscope-like effects in the template (and texture distribution). We use the WGAN GAN criterium, which works better for the combined parametric/non-parametric case (experimenting with the usage of DCGAN and WGAN depending on the architecture is advised). We set to use dIter=2 D steps for each G step.

The code also supports a slightly more complicated implementation than the one described in the paper. By setting multiScale=True a mixed template of images I_M on multiple levels of the Unet is used. In addition, by setting nBlocks=2 we will add residual layers to the decoder of the Unet, for a model with even higher capacity. Finally, you can also set refine=True and add a second Unet to refine the results of the first one. Of course, all these additional layers come at a computational cost -- selecting the layer depth, channel width, and the use of all these additional modules is a matter of trade-off and experimenting.

python mosaicFAMOS.py  --texturePath=samples/milano/ --contentPath=samples/archimboldo/ --N=80 --mirror=True --multiScale=True --nBlocks=1 --dIter=2 --WGAN=True

The method will save mosaics occasionally, and optionally you can specify a testImage (size smaller than the initial content image) to check out-of-sample performance. You can check the patches image saved regularly how the patch based training proceeds. The files has a column per batch-instance, and 6 rows showing the quantities from the paper:

  • I_C content patch
  • I_M mixed template patch on highest scale
  • I_G parametric generation component
  • I blended patch
  • \alpha blending mask
  • A mixing matrix

License

Please make sure to cite/acknowledge our paper, if you use any of the contained code in your own projects or publication.

The MIT License (MIT)

Copyright © 2018 Zalando SE, https://tech.zalando.com

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Owner
Zalando Research
Repositories of the research branch of Zalando SE
Zalando Research
Distributed Evolutionary Algorithms in Python

DEAP DEAP is a novel evolutionary computation framework for rapid prototyping and testing of ideas. It seeks to make algorithms explicit and data stru

Distributed Evolutionary Algorithms in Python 4.9k Jan 05, 2023
Project to deploy a machine learning model based on Titanic dataset from Kaggle

kaggle_titanic_deploy Project to deploy a machine learning model based on Titanic dataset from Kaggle In this project we used the Titanic dataset from

Vivian Yamassaki 8 May 23, 2022
Made in collaboration with Chris George for Art + ML Spring 2019.

Deepdream Eyes Made in collaboration with Chris George for Art + ML Spring 2019.

Francisco Cabrera 1 Jan 12, 2022
dirty_cat is a Python module for machine-learning on dirty categorical variables.

dirty_cat dirty_cat is a Python module for machine-learning on dirty categorical variables.

637 Dec 29, 2022
Predict profitability of trades based on indicator buy / sell signals

Predict profitability of trades based on indicator buy / sell signals Trade profitability analysis for trades based on various indicators signals: MAC

Tomasz Porzycki 1 Dec 15, 2021
QML: A Python Toolkit for Quantum Machine Learning

QML is a Python2/3-compatible toolkit for representation learning of properties of molecules and solids.

176 Dec 09, 2022
CVXPY is a Python-embedded modeling language for convex optimization problems.

CVXPY The CVXPY documentation is at cvxpy.org. We are building a CVXPY community on Discord. Join the conversation! For issues and long-form discussio

4.3k Jan 08, 2023
Client - 🔥 A tool for visualizing and tracking your machine learning experiments

Weights and Biases Use W&B to build better models faster. Track and visualize all the pieces of your machine learning pipeline, from datasets to produ

Weights & Biases 5.2k Jan 03, 2023
Distributed scikit-learn meta-estimators in PySpark

sk-dist: Distributed scikit-learn meta-estimators in PySpark What is it? sk-dist is a Python package for machine learning built on top of scikit-learn

Ibotta 282 Dec 09, 2022
Projeto: Machine Learning: Linguagens de Programacao 2004-2001

Projeto: Machine Learning: Linguagens de Programacao 2004-2001 Projeto de Data Science e Machine Learning de análise de linguagens de programação de 2

Victor Hugo Negrisoli 0 Jun 29, 2021
A project based example of Data pipelines, ML workflow management, API endpoints and Monitoring.

MLOps template with examples for Data pipelines, ML workflow management, API development and Monitoring.

Utsav 33 Dec 03, 2022
cleanlab is the data-centric ML ops package for machine learning with noisy labels.

cleanlab is the data-centric ML ops package for machine learning with noisy labels. cleanlab cleans labels and supports finding, quantifying, and lear

Cleanlab 51 Nov 28, 2022
Fast Fourier Transform-accelerated Interpolation-based t-SNE (FIt-SNE)

FFT-accelerated Interpolation-based t-SNE (FIt-SNE) Introduction t-Stochastic Neighborhood Embedding (t-SNE) is a highly successful method for dimensi

Kluger Lab 547 Dec 21, 2022
Kubeflow is a machine learning (ML) toolkit that is dedicated to making deployments of ML workflows on Kubernetes simple, portable, and scalable.

SDK: Overview of the Kubeflow pipelines service Kubeflow is a machine learning (ML) toolkit that is dedicated to making deployments of ML workflows on

Kubeflow 3.1k Jan 06, 2023
Fit interpretable models. Explain blackbox machine learning.

InterpretML - Alpha Release In the beginning machines learned in darkness, and data scientists struggled in the void to explain them. Let there be lig

InterpretML 5.2k Jan 09, 2023
Bottleneck a collection of fast, NaN-aware NumPy array functions written in C.

Bottleneck Bottleneck is a collection of fast, NaN-aware NumPy array functions written in C. As one example, to check if a np.array has any NaNs using

Python for Data 835 Dec 27, 2022
Coursera Machine Learning - Python code

Coursera Machine Learning This repository contains python implementations of certain exercises from the course by Andrew Ng. For a number of assignmen

Jordi Warmenhoven 859 Dec 10, 2022
Highly interpretable classifiers for scikit learn, producing easily understood decision rules instead of black box models

Highly interpretable, sklearn-compatible classifier based on decision rules This is a scikit-learn compatible wrapper for the Bayesian Rule List class

Tamas Madl 482 Nov 19, 2022
ArviZ is a Python package for exploratory analysis of Bayesian models

ArviZ (pronounced "AR-vees") is a Python package for exploratory analysis of Bayesian models. Includes functions for posterior analysis, data storage, model checking, comparison and diagnostics

ArviZ 1.3k Jan 05, 2023
Python Machine Learning Jupyter Notebooks (ML website)

Python Machine Learning Jupyter Notebooks (ML website) Dr. Tirthajyoti Sarkar, Fremont, California (Please feel free to connect on LinkedIn here) Also

Tirthajyoti Sarkar 2.6k Jan 03, 2023