Long Expressive Memory (LEM)

Last update: Dec 17, 2022

Overview

Long Expressive Memory for Sequence Modeling

This repository contains the implementation to reproduce the numerical experiments of the paper Long Expressive Memory for Sequence Modeling

Requirements

To setup the environment, install the requirements using python 3.7.10 with

pip install -r requirements.txt

Experiments

This repository contains the codes to reproduce the results of the following experiments for the proposed LEM:

Adding Task
EigenWorms
FitzHugh-Nagumo
Google12 (V2)
Heart rate prediction
noisy CIFAR-10
Permuted Sequential MNIST
PennTree Bank (word- and char-level)
Sequential MNIST

All datasets get either synthetically generated or downloaded automatically, providing a full data pipline for each experiment.
Note that the PTB experiments are based on the open-source tensorflow project LAnguage Modelling Benchmarks (lamb) for tuning and testing Tensorflow language models. In order to reproduce these experiments, please follow the steps explained in the README of the PTB directory.

Results

The results of LEM for each of the experiments are:

Experiment	Result
EigenWorms	92.3% mean test accuracy
FitzHugh-Nagumo	0.002 test L^2 loss
Google12 (V2)	95.7% test accuracy
Heart rate prediction	0.85 test L^2 loss
noisy CIFAR-10	60.5% test accuracy
Permuted Sequential MNIST	96.6% test accuracy
PennTree Bank char-level (single layer)	1.25 test bits-per-character
PennTree Bank word-level (single layer)	72.8 test perplexity
Sequential MNIST	99.5% test accuracy

Long Expressive Memory (LEM)

Related tags

Overview

Long Expressive Memory for Sequence Modeling

Requirements

Experiments

Results

Owner

Konstantin Rusch

EMNLP 2021 Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Implementation of Neonatal Seizure Detection using EEG signals for deploying on edge devices including Raspberry Pi.

The aim of the game, as in the original one, is to find a specific image from a group of different images of a person's face

Autoencoder - Reducing the Dimensionality of Data with Neural Network

MWPToolkit is a PyTorch-based toolkit for Math Word Problem (MWP) solving.

Use your Philips Hue lights as Racing Flags. Works with Assetto Corsa, Assetto Corsa Competizione and iRacing.

Background-Click Supervision for Temporal Action Localization

Identifying a Training-Set Attack’s Target Using Renormalized Influence Estimation

Meta Language-Specific Layers in Multilingual Language Models

Orthogonal Over-Parameterized Training

Transfer SemanticKITTI labeles into other dataset/sensor formats.

Solutions and questions for AoC2021. Merry christmas!

3rd Place Solution for ICCV 2021 Workshop SSLAD Track 3A - Continual Learning Classification Challenge

A python library for highly configurable transformers - easing model architecture search and experimentation.

A repo with study material, exercises, examples, etc for Devnet SPAUTO

Deploy optimized transformer based models on Nvidia Triton server

GARCH and Multivariate LSTM forecasting models for Bitcoin realized volatility with potential applications in crypto options trading, hedging, portfolio management, and risk management

ViewFormer: NeRF-free Neural Rendering from Few Images Using Transformers

A benchmark dataset for emulating atmospheric radiative transfer in weather and climate models with machine learning (NeurIPS 2021 Datasets and Benchmarks Track)

The official implementation of Theme Transformer