StyleGAN2 Webtoon / Anime Style Toonify

Overview

StyleGAN2 Webtoon / Anime Style Toonify

Korea Webtoon or Japanese Anime Character
Stylegan2 base high Quality 1024x1024 / 512x512 Generate and Transfer

Korean language
Stylegan2 를 이용해 고화질 반사실화 및 Webtoon / anime 스타일 캐릭터 생성 및 변환을 시도해 보았고 몇번의 시행착오 끝에 아래와 같이 만들 수 있었습니다.
사람 얼굴 사진 Pretrained Model 에서 webtoon / anime 이미지 4백장정도로 Transfer Learning 과 Network blending 을 이용하였습니다.
기존에 나온 것들은 서양 카툰 디즈니적이거나 반사실화적이지 않거나 동양적인 예쁜 케릭터가 잘 나오지 않아서 사실화같으면서 동양적 예쁘고 멋진 webtoon / anime 캐릭터가 나오는데 목적을 두었습니다.
새로운 동양적이고 예쁜 고화질 캐릭터 생성도 잘 하지만 사람 얼굴 사진을 project 후 나온 latent vectors 를 이용해 이를 새로운 훈련된 모델로 generate 하여 photo face to toon/anime face 로 변환하였습니다.
아직 미숙한 점도 있지만 단순 생성, 변환만이 아니고 latent vectors 를 이용하여 얼굴 움직이기, 웃기, 화내기 등 표정 등도 변경 생성 가능하였습니다.

이후에는 반대로 webtoon / anime 캐릭터 이미지를 사실적 photo로 변경하는 것과 얼굴뿐만 아니라 전체 몸을 생성, webtoon / anime 전체 화면을 생성해 보는 것을 해 보려 합니다. 시간이 되는대로...
더 나아가서는 DALL-E, CLIP 를 이용하거나 이런 방식을 이용해 글을 통해 toon/anime 를 생성하는 것을 진행해보고 싶습니다.
웹툰/애니메이션을 좋아하나 그림 솜씨는 없어도 AI를 이용해서 원하는 캐릭터나 toon/anime 을 만들고 이를 움직이고 말이나 글을 통해서 만들수 있지 않을까. 말로 AI 에게 지시해서 만들어가는 tonn/anime 을 상상해 봅니다.


English Language
I tried creating and converting high-definition reflections and Webtoon/anime style characters using Stylegan2, and after several trials and errors, I was able to create it as follows.
Transfer learning and network blending were used with about 400 webtoon / anime images with the human face photo Pretrained Model.
The existing ones are Western Cartoon Disney, not reflective, or Asian pretty characters aren't well out, so the purpose is to come up with Asian pretty and cool webtoon/anime character.
I am good at creating new Asian and pretty high-quality characters, but I used latent vectors that came out after projecting a human face photo and generated it as a new trained model and converted it to photo face to toon/anime face.
Although there are still some inexperienced things, it was possible to change expressions such as facial movements, laughs, and anger using latent vectors as well as simple creation and transformation.

Afterwards, I will try to change the webtoon / anime character image to a realistic photo, create not only the face but also the entire body, and create the webtoon / anime full screen. As time comes...
Furthermore, I would like to proceed with creating toon/anime through text writing using DALL-E, CLIP, or using this method.
I like webtoon/animation, but I don’t have any drawing skills.
Wouldn't it be possible to create a character or toon/anime you want using AI, move it, and create it through words or writings? I imagine a toon/anime created by instructing AI in words.

Compare to Others

Toonify compare image

Generate toon images

Toonify compare image

Transfer Korea Star Photos to toon Images

Toonify compare image

Edit Face 1(Latent direction move)

Toonify Latent direction

Edit Face 2(Latent direction move)

Toonify Latent direction


Requirements

See StyleGAN2 Requirements

Usage

  1. align image

Copy real face photos in "data_test" directory. It will save face croped and aligned images.

python align_images.py data_test data_img_aligned
  1. project image

It will generate *.npy files and projected_image files in "data_project_gen" directory. Make "pretrained_networks" directory. Download pretrain network "https://drive.google.com/file/d/1-04v78_pI59M0IvhcKxsm3YhK2-plnbj/view?usp=drivesdk" and save in "pretrained_networks" directory.

python project_images.py data_img_aligned data_project_gen --num-steps=1500 --network-pkl="pretrained_networks/generator_star-stylegan2-config-f.pkl" --tmp-dir="./tmp"
  1. toonify

It will generate toonified image files in "data_project_gen" directory and It will add "_toon" to filename. Download pretrain network "https://drive.google.com/file/d/1ENBelwQO-_HWFJk0cBzFvi86wPuYwxSz/view?usp=sharing" and save in "pretrained_networks" directory. Download pretrain network "https://drive.google.com/file/d/1AO42jOsgAdWC1L4lgZzk2-FarniPeilp/view?usp=sharing" and save in "pretrained_networks" directory.

Semi-realistic Toon

python toonify_lnw.py --image_dir=data_project_gen --filename=_toon1.jpg --blendednet="pretrained_networks/wonwizard-bl-anistar-s30r64.pkl"

Drawing Toon

python toonify_lnw.py --image_dir=data_project_gen --filename=_toon2.jpg --blendednet="pretrained_networks/wonwizard-bl-anistar-s120r128.pkl"

Based on nvidia StyleGAN2.
개인적으로만 사용가능하며 허가없이 대외적으로 사용할 수 없음.
These are for personal use only and cannot be used externally without my permission.

Owner
AI Research Android & Wizard Dreamer , Middle aged man
Learning Correspondence from the Cycle-consistency of Time (CVPR 2019)

TimeCycle Code for Learning Correspondence from the Cycle-consistency of Time (CVPR 2019, Oral). The code is developed based on the PyTorch framework,

Xiaolong Wang 706 Nov 29, 2022
Capsule endoscopy detection DACON challenge

capsule_endoscopy_detection (DACON Challenge) Overview Yolov5, Yolor, mmdetection기반의 모델을 사용 (총 11개 모델 앙상블) 모든 모델은 학습 시 Pretrained Weight을 yolov5, yolo

MAILAB 11 Nov 25, 2022
Principled Detection of Out-of-Distribution Examples in Neural Networks

ODIN: Out-of-Distribution Detector for Neural Networks This is a PyTorch implementation for detecting out-of-distribution examples in neural networks.

189 Nov 29, 2022
A pytorch-based deep learning framework for multi-modal 2D/3D medical image segmentation

A 3D multi-modal medical image segmentation library in PyTorch We strongly believe in open and reproducible deep learning research. Our goal is to imp

Adaloglou Nikolas 1.2k Dec 27, 2022
Demonstrates how to divide a DL model into multiple IR model files (division) and introduce a simplest way to implement a custom layer works with OpenVINO IR models.

Demonstration of OpenVINO techniques - Model-division and a simplest-way to support custom layers Description: Model Optimizer in Intel(r) OpenVINO(tm

Yasunori Shimura 12 Nov 09, 2022
Code for 'Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning' (AAAI 2022)

Blockwise Sequential Model Learning Code for 'Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning' (AAAI 2022) For ins

2 Jun 17, 2022
The second project in Python course on FCC

Assignment Write a function named add_time that takes in two required parameters and one optional parameter: a start time in the 12-hour clock format

Denise T 1 Dec 13, 2021
Framework for Spectral Clustering on the Sparse Coefficients of Learned Dictionaries

Dictionary Learning for Clustering on Hyperspectral Images Overview Framework for Spectral Clustering on the Sparse Coefficients of Learned Dictionari

Joshua Bruton 6 Oct 25, 2022
Source for the paper "Universal Activation Function for machine learning"

Universal Activation Function Tensorflow and Pytorch source code for the paper Yuen, Brosnan, Minh Tu Hoang, Xiaodai Dong, and Tao Lu. "Universal acti

4 Dec 03, 2022
Original Pytorch Implementation of FLAME: Facial Landmark Heatmap Activated Multimodal Gaze Estimation

FLAME Original Pytorch Implementation of FLAME: Facial Landmark Heatmap Activated Multimodal Gaze Estimation, accepted at the 17th IEEE Internation Co

Neelabh Sinha 19 Dec 17, 2022
PyTorch implementation of the Crafting Better Contrastive Views for Siamese Representation Learning

Crafting Better Contrastive Views for Siamese Representation Learning This is the official PyTorch implementation of the ContrastiveCrop paper: @artic

249 Dec 28, 2022
Least Square Calibration for Peer Reviews

Least Square Calibration for Peer Reviews Requirements gurobipy - for solving convex programs GPy - for Bayesian baseline numpy pandas To generate p

Sigma <a href=[email protected]"> 1 Nov 01, 2021
Iran Open Source Hackathon

Iran Open Source Hackathon is an open-source hackathon (duh) with the aim of encouraging participation in open-source contribution amongst Iranian dev

OSS Hackathon 121 Dec 25, 2022
The source code of the paper "Understanding Graph Neural Networks from Graph Signal Denoising Perspectives"

GSDN-F and GSDN-EF This repository provides a reference implementation of GSDN-F and GSDN-EF as described in the paper "Understanding Graph Neural Net

Guoji Fu 18 Nov 14, 2022
Reinforcement Learning for Portfolio Management

qtrader Reinforcement Learning for Portfolio Management Why Reinforcement Learning? Learns the optimal action, rather than models the market. Adaptive

Angelos Filos 406 Jan 01, 2023
Source code for the ACL-IJCNLP 2021 paper entitled "T-DNA: Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation" by Shizhe Diao et al.

T-DNA Source code for the ACL-IJCNLP 2021 paper entitled Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adapta

shizhediao 17 Dec 22, 2022
PyTorch implementation of Off-policy Learning in Two-stage Recommender Systems

Off-Policy-2-Stage This repo provides a PyTorch implementation of the MovieLens experiments for the following paper: Off-policy Learning in Two-stage

Jiaqi Ma 25 Dec 12, 2022
ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees

ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees This repository is the official implementation of the empirica

Kuan-Lin (Jason) Chen 2 Oct 02, 2022
A parametric soroban written with CADQuery.

A parametric soroban written in CADQuery The purpose of this project is to demonstrate how "code CAD" can be intuitive to learn. See soroban.py for a

Lee 4 Aug 13, 2022
Code for A Volumetric Transformer for Accurate 3D Tumor Segmentation

VT-UNet This repo contains the supported pytorch code and configuration files to reproduce 3D medical image segmentaion results of VT-UNet. Environmen

Himashi Amanda Peiris 114 Dec 20, 2022