Long Expressive Memory for Sequence Modeling
This repository contains the implementation to reproduce the numerical experiments of the paper Long Expressive Memory for Sequence Modeling
Requirements
To setup the environment, install the requirements using python 3.7.10 with
pip install -r requirements.txt
Experiments
This repository contains the codes to reproduce the results of the following experiments for the proposed LEM:
- Adding Task
- EigenWorms
- FitzHugh-Nagumo
- Google12 (V2)
- Heart rate prediction
- noisy CIFAR-10
- Permuted Sequential MNIST
- PennTree Bank (word- and char-level)
- Sequential MNIST
All datasets get either synthetically generated or downloaded automatically, providing a full data pipline for each experiment.
Note that the PTB experiments are based on the open-source tensorflow project LAnguage Modelling Benchmarks (lamb) for tuning and testing Tensorflow language models. In order to reproduce these experiments, please follow the steps explained in the README of the PTB directory.
Results
The results of LEM for each of the experiments are:
| Experiment | Result |
| EigenWorms | 92.3% mean test accuracy |
| FitzHugh-Nagumo | 0.002 test L^2 loss |
| Google12 (V2) | 95.7% test accuracy |
| Heart rate prediction | 0.85 test L^2 loss |
| noisy CIFAR-10 | 60.5% test accuracy |
| Permuted Sequential MNIST | 96.6% test accuracy |
| PennTree Bank char-level (single layer) | 1.25 test bits-per-character |
| PennTree Bank word-level (single layer) | 72.8 test perplexity |
| Sequential MNIST | 99.5% test accuracy |
176 Dec 28, 2022
223 Oct 18, 2022
105 Dec 17, 2022
49 Dec 11, 2022
320 Dec 28, 2022
69 Dec 30, 2022
182 Dec 23, 2022
672 Jan 01, 2023
6.2k Jan 01, 2023
49 Dec 22, 2022
95 Dec 29, 2022
1 Oct 08, 2021
125 Dec 23, 2022
10 Dec 22, 2022
385 Dec 29, 2022
8 Jan 02, 2023
11 Dec 03, 2022
42 Dec 28, 2022
108 Dec 09, 2022
395 Dec 30, 2022