A denoising diffusion probabilistic model (DDPM) tailored for conditional generation of protein distograms

Last update: Nov 23, 2022

Overview

Denoising Diffusion Probabilistic Model for Proteins

Implementation of Denoising Diffusion Probabilistic Model in Pytorch. It is a new approach to generative modeling that may have the potential to rival GANs. It uses denoising score matching to estimate the gradient of the data distribution, followed by Langevin sampling to sample from the true distribution. This implementation was transcribed from the official Tensorflow version here.

This specific repository will be using a heavily modifying version of the U-net for learning on protein structure, with eventual conditioning from MSA Transformers attention heads.

** at around 40k iterations **

Install

$ pip install ddpm-proteins

Training

We are using weights & biases for experimental tracking

First you need to login

$ wandb login

Then

$ python train.py

Edit train.py to whatever for your research desires

Todo

condition on mask
condition on MSA transformers (with caching of tensors in specified directory by protein id)
reach for size 384
all-attention network with uformer https://arxiv.org/abs/2106.03106 (with 1d + 2d conv kernels)
add all improvements from https://arxiv.org/abs/2105.05233 and https://cascaded-diffusion.github.io/

Usage

import torch
from ddpm_proteins import Unet, GaussianDiffusion

model = Unet(
    dim = 64,
    dim_mults = (1, 2, 4, 8)
)

diffusion = GaussianDiffusion(
    model,
    image_size = 128,
    timesteps = 1000,   # number of steps
    loss_type = 'l1'    # L1 or L2
)

training_images = torch.randn(8, 3, 128, 128)
loss = diffusion(training_images)
loss.backward()
# after a lot of training

sampled_images = diffusion.sample(batch_size = 4)
sampled_images.shape # (4, 3, 128, 128)

Or, if you simply want to pass in a folder name and the desired image dimensions, you can use the Trainer class to easily train a model.

from ddpm_proteins import Unet, GaussianDiffusion, Trainer

model = Unet(
    dim = 64,
    dim_mults = (1, 2, 4, 8)
).cuda()

diffusion = GaussianDiffusion(
    model,
    image_size = 128,
    timesteps = 1000,   # number of steps
    loss_type = 'l1'    # L1 or L2
).cuda()

trainer = Trainer(
    diffusion,
    'path/to/your/images',
    train_batch_size = 32,
    train_lr = 2e-5,
    train_num_steps = 700000,         # total training steps
    gradient_accumulate_every = 2,    # gradient accumulation steps
    ema_decay = 0.995,                # exponential moving average decay
    fp16 = True                       # turn on mixed precision training with apex
)

trainer.train()

Samples and model checkpoints will be logged to ./results periodically

Citations

@misc{ho2020denoising,
    title   = {Denoising Diffusion Probabilistic Models},
    author  = {Jonathan Ho and Ajay Jain and Pieter Abbeel},
    year    = {2020},
    eprint  = {2006.11239},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

@inproceedings{anonymous2021improved,
    title   = {Improved Denoising Diffusion Probabilistic Models},
    author  = {Anonymous},
    booktitle = {Submitted to International Conference on Learning Representations},
    year    = {2021},
    url     = {https://openreview.net/forum?id=-NEXDKk8gZ},
    note    = {under review}
}

@article{Rao2021.02.12.430858,
    author  = {Rao, Roshan and Liu, Jason and Verkuil, Robert and Meier, Joshua and Canny, John F. and Abbeel, Pieter and Sercu, Tom and Rives, Alexander},
    title   = {MSA Transformer},
    year    = {2021},
    publisher = {Cold Spring Harbor Laboratory},
    URL     = {https://www.biorxiv.org/content/early/2021/02/13/2021.02.12.430858},
    journal = {bioRxiv}
}

QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing

QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing Environment Tested on Ubuntu 14.04 64bit and 16.04 64bit Installation # disabl

581 Dec 30, 2022

Official PyTorch implementation for FastDPM, a fast sampling algorithm for diffusion probabilistic models

Official PyTorch implementation for "On Fast Sampling of Diffusion Probabilistic Models". FastDPM generation on CIFAR-10, CelebA, and LSUN datasets. S

68 Dec 26, 2022

Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

2.9k Jan 4, 2023

Implementation of GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation (ICLR 2022).

GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation [OpenReview] [arXiv] [Code] The official implementation of GeoDiff: A Geome

155 Dec 26, 2022

Implementation and replication of ProGen, Language Modeling for Protein Generation, in Jax

ProGen - (wip) Implementation and replication of ProGen, Language Modeling for Protein Generation, in Pytorch and Jax (the weights will be made easily

71 Dec 1, 2022

Replication attempt for the Protein Folding Model

RGN2-Replica (WIP) To eventually become an unofficial working Pytorch implementation of RGN2, an state of the art model for MSA-less Protein Folding f

36 Nov 29, 2022

Implementation of a protein autoregressive language model, but with autoregressive infilling objective (editing subsequences capability)

Protein GLM (wip) Implementation of a protein autoregressive language model, but with autoregressive infilling objective (editing subsequences capabil

17 May 6, 2022

McGill Physics Hackathon 2021: Reaction-Diffusion Models for the Generation of Biological Patterns

DiffuseAnimals: Reaction-Diffusion Models for the Generation of Biological Patterns Introduction Reaction-diffusion equations can be utilized in order

2 Mar 7, 2022

StudioGAN is a Pytorch library providing implementations of representative Generative Adversarial Networks (GANs) for conditional/unconditional image generation.

3k Jan 8, 2023

A denoising diffusion probabilistic model (DDPM) tailored for conditional generation of protein distograms

Related tags

Overview

Denoising Diffusion Probabilistic Model for Proteins

Install

Training

Todo

Usage

Citations

You might also like...

QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing

Official PyTorch implementation for FastDPM, a fast sampling algorithm for diffusion probabilistic models

Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

Implementation of GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation (ICLR 2022).

Implementation and replication of ProGen, Language Modeling for Protein Generation, in Jax

Replication attempt for the Protein Folding Model

Implementation of a protein autoregressive language model, but with autoregressive infilling objective (editing subsequences capability)

McGill Physics Hackathon 2021: Reaction-Diffusion Models for the Generation of Biological Patterns

StudioGAN is a Pytorch library providing implementations of representative Generative Adversarial Networks (GANs) for conditional/unconditional image generation.

Releases(0.0.11)

0.0.11(Apr 20, 2022)

0.0.10(Aug 24, 2021)

0.0.9(Jun 22, 2021)

0.0.8(Jun 17, 2021)

0.0.7(Jun 17, 2021)

0.0.6(Jun 15, 2021)

0.0.5(Jun 14, 2021)

0.0.4(Jun 14, 2021)

0.0.2(Jun 14, 2021)

0.0.1b(Jun 14, 2021)

Owner

Phil Wang

Official code for "Mean Shift for Self-Supervised Learning"

Beyond imagenet attack (accepted by ICLR 2022) towards crafting adversarial examples for black-box domains.

《Rethinking Sptil Dimensions of Vision Trnsformers》(2021)

SplineConv implementation for Paddle.

Official repository for MixFaceNets: Extremely Efficient Face Recognition Networks

Official Pytorch implementation for 2021 ICCV paper "Learning Motion Priors for 4D Human Body Capture in 3D Scenes" and trained models / data

The FIRST GANs-based omics-to-omics translation framework

Patch SVDD for Image anomaly detection

Wordle Env: A Daily Word Environment for Reinforcement Learning

Source code for Task-Aware Variational Adversarial Active Learning

Programming with Neural Surrogates of Programs

MRI reconstruction (e.g., QSM) using deep learning methods

chainladder - Property and Casualty Loss Reserving in Python

Modeling Temporal Concept Receptive Field Dynamically for Untrimmed Video Analysis

Code accompanying the paper "Wasserstein GAN"

Machine learning library for fast and efficient Gaussian mixture models

CVPR2022 paper "Dense Learning based Semi-Supervised Object Detection"

Ensembling Off-the-shelf Models for GAN Training

Roach: End-to-End Urban Driving by Imitating a Reinforcement Learning Coach

Blender add-on: Add to Cameras menu: View → Camera, View → Add Camera, Camera → View, Previous Camera, Next Camera