The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Last update: Oct 19, 2022

Overview

Enformer TPU training script (wip)

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters, in an effort to migrate the model to pytorch.

This was pieced together from the Deepmind Enformer repository, the colab training notebook, as well as Basenji sequence augmentation code

It accounts for:

distributed TPU training
distributed datasets
distributed validation
gradient clipping
cross replica batchnorms
dataset augmentation

Training takes about 3 days on v3-64

Todo

fix script for differences in sequence length in basenji training data, which is ~130k vs ~190k bp as in paper

Citations

@article {Avsec2021.04.07.438649,
    author  = {Avsec, {\v Z}iga and Agarwal, Vikram and Visentin, Daniel and Ledsam, Joseph R. and Grabska-Barwinska, Agnieszka and Taylor, Kyle R. and Assael, Yannis and Jumper, John and Kohli, Pushmeet and Kelley, David R.},
    title   = {Effective gene expression prediction from sequence by integrating long-range interactions},
    elocation-id = {2021.04.07.438649},
    year    = {2021},
    doi     = {10.1101/2021.04.07.438649},
    publisher = {Cold Spring Harbor Laboratory},
    URL     = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649},
    eprint  = {https://www.biorxiv.org/content/early/2021/04/08/2021.04.07.438649.full.pdf},
    journal = {bioRxiv}
}

The full training script for Enformer (Tensorflow Sonnet) on TPU clusters

Related tags

Overview

Enformer TPU training script (wip)

Todo

Citations

Owner

Phil Wang

This repository provides a PyTorch implementation and model weights for HCSC (Hierarchical Contrastive Selective Coding)

Adversarial Texture Optimization from RGB-D Scans (CVPR 2020).

Exploring Simple Siamese Representation Learning

Blender Add-On for slicing meshes with planes

The Official Repository for "Generalized OOD Detection: A Survey"

Code for the Convolutional Vision Transformer (ConViT)

PyTorch code for JEREX: Joint Entity-Level Relation Extractor

Implementation of Deep Deterministic Policy Gradiet Algorithm in Tensorflow

Re-implement CycleGAN in Tensorlayer

Repo for the Tutorials of Day1-Day3 of the Nordic Probabilistic AI School 2021 (https://probabilistic.ai/)

Image based Human Fall Detection

Using LSTM write Tang poetry

Information-Theoretic Multi-Objective Bayesian Optimization with Continuous Approximations

A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets

An implementation of Geoffrey Hinton's paper "How to represent part-whole hierarchies in a neural network" in Pytorch.

Ankou: Guiding Grey-box Fuzzing towards Combinatorial Difference

Code for the paper Progressive Pose Attention for Person Image Generation in CVPR19 (Oral).

Immortal tracker

Learned model to estimate number of distinct values (NDV) of a population using a small sample.

A way to store images in YAML.