Implementation of Fast Transformer in Pytorch

Last update: Dec 27, 2022

Overview

Fast Transformer - Pytorch

Implementation of Fast Transformer in Pytorch. This only work as an encoder.

Install

$ pip install fast-transformer-pytorch

Usage

import torch
from fast_transformer_pytorch import FastTransformer

model = FastTransformer(
    num_tokens = 20000,
    dim = 512,
    depth = 2,
    max_seq_len = 4096,
    absolute_pos_emb = True   # default uses relative positional encoding, but if that isn't working, then turn on absolute positional embedding by setting this to True
)

x = torch.randint(0, 20000, (1, 4096))
mask = torch.ones(1, 4096).bool()

logits = model(x, mask = mask) # (1, 4096, 20000)

Citations

@misc{wu2021fastformer,
    title   = {Fastformer: Additive Attention is All You Need}, 
    author  = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},
    year    = {2021},
    eprint  = {2108.09084},
    archivePrefix = {arXiv},
    primaryClass = {cs.CL}
}

A Transformer Implementation that is easy to understand and customizable.

Simple Transformer I've written a series of articles on the transformer architecture and language models on Medium. This repository contains an implem

4 Jan 20, 2022

Fast topic modeling platform

The state-of-the-art platform for topic modeling. Full Documentation User Mailing List Download Releases User survey What is BigARTM? BigARTM is a pow

633 Dec 21, 2022

Easy, fast, effective, and automatic g-code compression!

Getting to the meat of g-code. Easy, fast, effective, and automatic g-code compression! MeatPack nearly doubles the effective data rate of a standard

97 Nov 21, 2022

Library for fast text representation and classification.

fastText fastText is a library for efficient learning of word representations and sentence classification. Table of contents Resources Models Suppleme

24.1k Jan 5, 2023

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Main features: Train new vocabularies and tok

6.2k Dec 31, 2022

✨Fast Coreference Resolution in spaCy with Neural Networks

✨ NeuralCoref 4.0: Coreference Resolution in spaCy with Neural Networks. NeuralCoref is a pipeline extension for spaCy 2.1+ which annotates and resolv

2.6k Jan 4, 2023

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

(Framework for Adapting Representation Models) What is it? FARM makes Transfer Learning with BERT & Co simple, fast and enterprise-ready. It's built u

1.6k Dec 27, 2022

Library for fast text representation and classification.

fastText fastText is a library for efficient learning of word representations and sentence classification. Table of contents Resources Models Suppleme

22.2k Feb 18, 2021

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Main features: Train new vocabularies and tok

4.3k Feb 18, 2021

Releases(0.0.4)

0.0.4(Aug 25, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Aug 24, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Aug 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Aug 23, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

Collection of useful (to me) python scripts for interacting with napari

Napari scripts A collection of napari related tools in various state of disrepair/functionality. Browse_LIF_widget.py This module can be imported, for

5 Aug 15, 2022

Smart discord chatbot integrated with Dialogflow to manage different classrooms and assist in teaching!

smart-school-chatbot Smart discord chatbot integrated with Dialogflow to interact with students naturally and manage different classes in a school. De

5 Oct 24, 2022

ADCS cert template modification and ACL enumeration

Purpose This tool is designed to aid an operator in modifying ADCS certificate templates so that a created vulnerable state can be leveraged for privi

78 Dec 12, 2022

Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention

Sinkhorn Transformer This is a reproduction of the work outlined in Sparse Sinkhorn Attention, with additional enhancements. It includes a parameteriz

217 Nov 25, 2022

null

CP-Cluster Confidence Propagation Cluster aims to replace NMS-based methods as a better box fusion framework in 2D/3D Object detection, Instance Segme

41 Dec 08, 2022

本插件是pcrjjc插件的重置版，可以独立于后端api运行

pcrjjc2 本插件是pcrjjc重置版，不需要使用其他后端api，但是需要自行配置客户端本项目基于AGPL v3协议开源，由于项目特殊性，禁止基于本项目的任何商业行为配置方法环境需求：.net framework 4.5及以上 jre8 别忘了装jre8 别忘了装jre8 别忘了装jre8

132 Dec 26, 2022

Gpt2-WebAPI - The objective of this API is to provide the 3 best possible responses to sentences that the user would input via http GET request as a parameter

This repository is a modification of: https://github.com/openai/gpt-2 for our sp

1 Jan 06, 2022

Implementation of Fast Transformer in Pytorch

Related tags

Overview

Fast Transformer - Pytorch

Install

Usage

Citations

You might also like...

A Transformer Implementation that is easy to understand and customizable.

Fast topic modeling platform

Easy, fast, effective, and automatic g-code compression!

Library for fast text representation and classification.

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

✨Fast Coreference Resolution in spaCy with Neural Networks

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

Library for fast text representation and classification.

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Releases(0.0.4)

0.0.4(Aug 25, 2021)

0.0.3(Aug 24, 2021)

0.0.2(Aug 23, 2021)

0.0.1(Aug 23, 2021)

Owner

Phil Wang

Collection of useful (to me) python scripts for interacting with napari

Smart discord chatbot integrated with Dialogflow to manage different classrooms and assist in teaching!

ADCS cert template modification and ACL enumeration

Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention

null

本插件是pcrjjc插件的重置版，可以独立于后端api运行

Gpt2-WebAPI - The objective of this API is to provide the 3 best possible responses to sentences that the user would input via http GET request as a parameter

A simple visual front end to the Maya UE4 RBF plugin delivered with MetaHumans

Tools and data for measuring the popularity & growth of various programming languages.

AllenNLP integration for Shiba: Japanese CANINE model

Command Line Text-To-Speech using Google TTS

CCF BDCI 2020 房产行业聊天问答匹配赛道 A榜47/2985

PyABSA - Open & Efficient for Framework for Aspect-based Sentiment Analysis

Pytorch implementation of Tacotron

A simple Speech Emotion Recognition (SER) API created using Flask and running in a Docker container.

Code voor mijn Master project omtrent VideoBERT

YACLC - Yet Another Chinese Learner Corpus

Arabic speech recognition, classification and text-to-speech.

🚀Clone a voice in 5 seconds to generate arbitrary speech in real-time

🏖 Easy training and deployment of seq2seq models.