Fast Transformer - Pytorch
Implementation of Fast Transformer in Pytorch. This only work as an encoder.
Install
$ pip install fast-transformer-pytorch
Usage
import torch
from fast_transformer_pytorch import FastTransformer
model = FastTransformer(
num_tokens = 20000,
dim = 512,
depth = 2,
max_seq_len = 4096,
absolute_pos_emb = True # default uses relative positional encoding, but if that isn't working, then turn on absolute positional embedding by setting this to True
)
x = torch.randint(0, 20000, (1, 4096))
mask = torch.ones(1, 4096).bool()
logits = model(x, mask = mask) # (1, 4096, 20000)
Citations
@misc{wu2021fastformer,
title = {Fastformer: Additive Attention is All You Need},
author = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},
year = {2021},
eprint = {2108.09084},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}
4 Jan 20, 2022
633 Dec 21, 2022
97 Nov 21, 2022
24.1k Jan 5, 2023
6.2k Dec 31, 2022
1.6k Dec 27, 2022
2 Jan 09, 2022
14 May 24, 2022
21 Dec 11, 2022
8 Dec 17, 2021
163 Dec 31, 2022
597 Dec 18, 2022
290 Dec 20, 2022
101 Jan 03, 2023
922 Dec 31, 2022
1 Nov 19, 2021
13 Jul 30, 2022
49 Dec 30, 2022
1 Oct 10, 2022
2 Nov 29, 2022
4 Jul 08, 2022
27 Jul 09, 2022
172 Dec 13, 2022
1 Jan 27, 2022