Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Last update: Dec 24, 2022

Related tags

Overview

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Abstract

We propose a new generative model for 3D garment deformations that enables us to learn, for the first time, a data-driven method for virtual try-on that effectively addresses garment-body collisions. In contrast to existing methods that require an undesirable postprocessing step to fix garment-body interpenetrations at test time, our approach directly outputs 3D garment configurations that do not collide with the underlying body. Key to our success is a new canonical space for garments that removes pose-and-shape deformations already captured by a new diffused human body model, which extrapolates body surface properties such as skinning weights and blendshapes to any 3D point. We leverage this representation to train a generative model with a novel self-supervised collision term that learns to reliably solve garment-body interpenetrations. We extensively evaluate and compare our results with recently proposed data-driven methods, and show that our method is the first to successfully address garment-body contact in unseen body shapes and motions, without compromising realism and detail.

Running the model

Requirements: python3.8, tensorflow-2.2.1, numpy-1.18.5, scipy-1.7.1, chumpy-0.70

Project structure:

vto-garment-collisions
│
└───assets 
|    └─ images    
|    └─ meshes    
|    └─ CMU       # Not included, see instructions
|    └─ SMPL      # Not included, see instructions
| 
└───rendering     # Code to render meshes 
|
└───src           # Code to run the model
| 
└───trained_models      
|    └─ diffused_body  # Networks of the diffused body model (Not included, see instructions)
|    └─ tshirt         # Networks of tshirt model (Not included, see instructions)
│
└───run_model.py

Download trained models

Download models of the diffused human body: https://github.com/isantesteban/vto-garment-collisions/releases/download/trained-models/trained_models_diffused_body.zip
Download models of the garment: https://github.com/isantesteban/vto-garment-collisions/releases/download/tshirt-trained-models/trained_models_tshirt.zip
Create trained_models directory and extract trained_models_diffused_body.zip and trained_models_tshirt.zip there.

Download human model

Sign in into https://smpl.is.tue.mpg.de
Download SMPL version 1.0.0 for Python 2.7 (10 shape PCs)
Extract SMPL_python_v.1.0.0.zip and copy smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl in assets/SMPL

Download animation sequences

Sign in into https://amass.is.tue.mpg.de
Download the body data for the CMU motions (SMPL+H model)
Extract CMU.tar.bz2 in assets/CMU:

tar -C assets/ -xf ~/Downloads/CMU.tar.bz2 CMU/

Generate garment animation

To generate the deformed garment meshes for a given sequence:

python run_model.py assets/CMU/07/07_02_poses.npz --export_dir results/07_02

Rendering

Requirements: blender-2.93, ffmpeg

To render the meshes:

blender --background rendering/scene.blend --python rendering/render.py --path results/07_02

Citation

If you find this repository useful please cite our work:

@article {santesteban2021garmentcollisions,
    journal = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    title = {{Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On}},
    author = {Santesteban, Igor and Thuerey, Nils and Otaduy, Miguel A and Casas, Dan},
    year = {2021}
}

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

AI Virtual Calculator: This is a simple virtual calculator that works with gestures using OpenCV. We will use our hand in the air to click on the calc

1 Jan 13, 2022

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

SoC-Winter Checkout some cool self-projects you can try your hands on to curb your boredom this December! These are short projects that you can do you

29 Nov 8, 2022

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models Codes for this paper The Lottery Tickets Hypo

59 Dec 28, 2022

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Patch-Rotation(PatchRot) Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models Submitted to Neurips2021 To

4 Jul 12, 2021

The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

Lexa-Benchmark Codebase for the self-supervised goal reaching benchmark introduced in 'Discovering and Achieving Goals via World Models'. Setup Create

1 Oct 14, 2021

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Core-tuning This repository is the official implementation of ``Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regular

18 Dec 17, 2022

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Related tags

Overview

Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On

Abstract

Running the model

Download trained models

Download human model

Download animation sequences

Generate garment animation

Rendering

Citation

You might also like...

AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

The self-supervised goal reaching benchmark introduced in Discovering and Achieving Goals via World Models

This repository is the official implementation of Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (NeurIPS21).

Try out deep learning models online on Google Colab

Users can free try their models on SIDD dataset based on this code

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

Releases(dress-trained-models)

dress-trained-models(Mar 28, 2022)

tshirt-trained-models(Nov 4, 2021)

trained-models(Nov 4, 2021)

Owner

Interpretable-contrastive-word-mover-s-embedding

This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking".

A static analysis library for computing graph representations of Python programs suitable for use with graph neural networks.

WTTE-RNN a framework for churn and time to event prediction

NeoPlay is the project dedicated to ESport events.

Dense Unsupervised Learning for Video Segmentation (NeurIPS*2021)

The official implementation for "FQ-ViT: Fully Quantized Vision Transformer without Retraining".

Unofficial PyTorch implementation of MobileViT.

covid question answering datasets and fine tuned models

SE-MSCNN: A Lightweight Multi-scaled Fusion Network for Sleep Apnea Detection Using Single-Lead ECG Signals

Official PyTorch implementation of SyntaSpeech (IJCAI 2022)

Neural Fixed-Point Acceleration for Convex Optimization

Code for "Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation". [AAAI 2021]

(JMLR' 19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

Code & Models for Temporal Segment Networks (TSN) in ECCV 2016

💊 A 3D Generative Model for Structure-Based Drug Design (NeurIPS 2021)

Fuzzer for Linux Kernel Drivers

The first public PyTorch implementation of Attentive Recurrent Comparators

Synthetic structured data generators

Using Convolutional Neural Networks (CNN) for Semantic Segmentation of Breast Cancer Lesions (BRCA)