Fast Transformer - Pytorch
Implementation of Fast Transformer in Pytorch. This only work as an encoder.
Install
$ pip install fast-transformer-pytorch
Usage
import torch
from fast_transformer_pytorch import FastTransformer
model = FastTransformer(
num_tokens = 20000,
dim = 512,
depth = 2,
max_seq_len = 4096,
absolute_pos_emb = True # default uses relative positional encoding, but if that isn't working, then turn on absolute positional embedding by setting this to True
)
x = torch.randint(0, 20000, (1, 4096))
mask = torch.ones(1, 4096).bool()
logits = model(x, mask = mask) # (1, 4096, 20000)
Citations
@misc{wu2021fastformer,
title = {Fastformer: Additive Attention is All You Need},
author = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},
year = {2021},
eprint = {2108.09084},
archivePrefix = {arXiv},
primaryClass = {cs.CL}
}
4 Jan 20, 2022
633 Dec 21, 2022
97 Nov 21, 2022
24.1k Jan 5, 2023
6.2k Dec 31, 2022
1.6k Dec 27, 2022
11 Sep 19, 2022
458 Jan 02, 2023
47 Dec 15, 2022
1.1k Dec 16, 2022
9 Nov 09, 2021
231 Dec 30, 2022
1 Jan 06, 2022
14 Dec 12, 2022
6 Nov 10, 2022
4 Aug 20, 2022
319 Dec 20, 2022
2 May 14, 2022
2 Jan 12, 2022
45 Oct 29, 2022
73 Nov 28, 2022
754 Jan 09, 2023
1 Dec 17, 2021
1.1k Dec 17, 2022
395 Nov 21, 2022