Official code repository of the paper Linear Transformers Are Secretly Fast Weight Programmers.

Last update: Dec 19, 2022

Overview

Linear Transformers Are Secretly Fast Weight Programmers

This repository contains the code accompanying the paper Linear Transformers Are Secretly Fast Weight Programmers which is published at ICML'21. It also contains the logs of all synthetic experiments.

Synthetic Experiments

Requirements

$ cat req.txt 
jupyter==1.0.0
pandas==1.0.1
seaborn==0.10.0
torch==1.6.0
matplotlib==3.1.3
numpy==1.17.2

pip3 install -r req.txt

Rerun Experiments

Logs are provided in the synthetic/logs folder. The files in that folder are a result of running the following commands:

Setting 1 (capacity):

python3 main.py --begin=20 --end=600 --step=20 --attn_name=softmax --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=linear --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=2 --update_rule=sum

python3 main.py --begin=20 --end=600 --step=20 --attn_name=dpfp --attn_arg=3 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=64 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=128 --update_rule=sum
python3 main.py --begin=20 --end=600 --step=20 --attn_name=favor --attn_arg=512 --update_rule=sum

Setting 2 (update rule):

python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=sum --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=tanh --update_rule=fwm --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=1 --update_rule=fwm --replace

python3 main.py --begin=20 --end=200 --step=20 --attn_name=dpfp --attn_arg=2 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=linear --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=favor --attn_arg=64 --update_rule=ours --replace
python3 main.py --begin=20 --end=200 --step=20 --attn_name=favor --attn_arg=128 --update_rule=ours --replace

Generate figures from the logs using the following notebooks:

synthetic/setting1_generate_figure.ipynb
synthetic/setting2_generate_figure.ipynb

Language Modelling & Machine Translation

The toolkit and scripts for language modeling experiments can be found at IDSIA/lmtool-fwms.

For machine translation experiments, we ported the different attention functions implemented in the language modeling toolkit to the multi-head attention implementation in FAIRSEQ.

Citation

@inproceedings{schlag2021linear,
      title={Linear Transformers Are Secretly Fast Weight Programmers}, 
      author={Imanol Schlag and Kazuki Irie and J\"urgen Schmidhuber},
      booktitle={Proc. Int. Conf. on Machine Learning (ICML)},
      address = {Virtual only},
      month = jul,
      year={2021}
}

Official code repository of the paper Linear Transformers Are Secretly Fast Weight Programmers.

Related tags

Overview

Linear Transformers Are Secretly Fast Weight Programmers

Synthetic Experiments

Requirements

Rerun Experiments

Language Modelling & Machine Translation

Citation

Owner

Imanol Schlag

Various capabilities for static malware analysis.

Research code for ECCV 2020 paper "UNITER: UNiversal Image-TExt Representation Learning"

Toward Model Interpretability in Medical NLP

Automatic privilege escalation for misconfigured capabilities, sudo and suid binaries

Tutorial to pretrain & fine-tune a 🤗 Flax T5 model on a TPUv3-8 with GCP

Random-Word-Generator - Generates meaningful words from dictionary with given no. of letters and words.

neural network based speaker embedder

Score-Based Point Cloud Denoising (ICCV'21)

A Facebook Messenger Chatbot using NLP

This code is the implementation of Text Emotion Recognition (TER) with linguistic features

A python script that will use hydra to get user and password to login to ssh, ftp, and telnet

Modified GPT using average pooling to reduce the softmax attention memory constraints.

topic modeling on unstructured data in Space news articles retrieved from the Guardian (UK) newspaper using API

💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants

Code for Discovering Topics in Long-tailed Corpora with Causal Intervention.

💫 Industrial-strength Natural Language Processing (NLP) in Python

Official code of our work, Unified Pre-training for Program Understanding and Generation [NAACL 2021].

GAP-text2SQL: Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Japanese Long-Unit-Word Tokenizer with RemBertTokenizerFast of Transformers

🤗🖼️ HuggingPics: Fine-tune Vision Transformers for anything using images found on the web.