The code for two papers: Feedback Transformer and Expire-Span.

Last update: Dec 25, 2022

Related tags

Deep Learning transformer-sequential

Overview

transformer-sequential

This repo contains the code for two papers:

Feedback Transformer
Expire-Span

The training code is structured for long sequential modeling with Transformer-like architectures.

Requirements

You will need a CUDA-enabled GPU to run the code.

Setup

Run the following:

pip install -r requirements.txt

Feedback Transformer

Introduced in Addressing Some Limitations of Transformers with Feedback Memory.

Running Experiments from the Paper

enwik8

Model	Params	Valid	Test
Feedback Transformer	77M	0.984	0.962

Numbers are Bits-Per-Character

bash experiments/feedback/enwik8.sh

Algorithmic

Model	3 Variable	5 Variable
Transformer	33.7	37.5
Feedback Transformer	99.1	92.6

Numbers are % Accuracy on Test

bash experiments/feedback/algorithmic_3var.sh
bash experiments/feedback/algorithmic_5var.sh

Expire-Span

Introduced in Not All Memories are Created Equal: Learning to Expire.

Running Experiments from the Paper

enwik8

Model	Params	Valid	Test
Expire-Span 12L	38M	1.014	0.994

Numbers are Bits-Per-Character

bash experiments/expire_span/enwik8.sh

Object Collision

Model	Maximum Span	Test Error (%)
Expire-Span	16k	52.2
Expire-Span	32k	36.7
Expire-Span	64k	26.7

bash experiments/expire_span/object_collision_16k.sh
bash experiments/expire_span/object_collision_32k.sh
bash experiments/expire_span/object_collision_64k.sh

License

The code is licensed under CC-BY-NC license. See the LICENSE file for more details.

The code for two papers: Feedback Transformer and Expire-Span.

Related tags

Overview

transformer-sequential

Requirements

Setup

Feedback Transformer

Running Experiments from the Paper

enwik8

Algorithmic

Expire-Span

Running Experiments from the Paper

enwik8

Object Collision

License

Owner

Facebook Research

The final project of "Applying AI to 2D Medical Imaging Data" of "AI for Healthcare" nanodegree - Udacity.

Analysing poker data from home games with friends

Code for the paper BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks

Code for the paper "Generative design of breakwaters usign deep convolutional neural network as a surrogate model"

PHOTONAI is a high level python API for designing and optimizing machine learning pipelines.

Official PyTorch implementation of SyntaSpeech (IJCAI 2022)

Code for "NeRS: Neural Reflectance Surfaces for Sparse-View 3D Reconstruction in the Wild," in NeurIPS 2021

This is a Keras implementation of a CNN for estimating age, gender and mask from a camera.

Safe Local Motion Planning with Self-Supervised Freespace Forecasting, CVPR 2021

Code associated with the paper "Towards Understanding the Data Dependency of Mixup-style Training".

Code from Daniel Lemire, A Better Alternative to Piecewise Linear Time Series Segmentation

Code basis for the paper "Camera Condition Monitoring and Readjustment by means of Noise and Blur" (2021)

Official codebase for Pretrained Transformers as Universal Computation Engines.

Evaluating saliency methods on artificial data with different background types

Source code and data in paper "MDFEND: Multi-domain Fake News Detection (CIKM'21)"

Marine debris detection with commercial satellite imagery and deep learning.

Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Adaptation through prediction: multisensory active inference torque control

The codes reproduce the figures and statistics in the paper, "Controlling for multiple covariates," by Mark Tygert.

code for paper "Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?"