Offical implementation of Shunted Self-Attention via Multi-Scale Token Aggregation

Last update: Dec 27, 2022

Related tags

Overview

Shunted Transformer

This is the offical implementation of Shunted Self-Attention via Multi-Scale Token Aggregation by Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, Xinchao Wang

Training from scratch

Training

bash dist_train.sh

Model Zoo

The checkpoints can be found at Goolge Drive, Baidu Pan (code:hazr) (Checkpoints of the large models are coming soon.)

Method	Size	[email protected]	#Params (M)
Shunted-T	224	79.8	11.5
Shunted-S	224	82.9	22.4
Shunted-B	224	84.0	39.6

Citation

@misc{ren2021shunted,
      title={Shunted Self-Attention via Multi-Scale Token Aggregation}, 
      author={Sucheng Ren and Daquan Zhou and Shengfeng He and Jiashi Feng and Xinchao Wang},
      year={2021},
      eprint={2111.15193},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Owner

First-year master student at SCUT

GitHub Repository

VIsually-Pivoted Audio and(N) Text

VIP-ANT: VIsually-Pivoted Audio and(N) Text Code for the paper Connecting the Dots between Audio and Text without Parallel Data through Visual Knowled

16 Nov 04, 2022

This is the codebase for the ICLR 2021 paper Trajectory Prediction using Equivariant Continuous Convolution

Trajectory Prediction using Equivariant Continuous Convolution (ECCO) This is the codebase for the ICLR 2021 paper Trajectory Prediction using Equivar

45 Jul 22, 2022

Boundary-aware Transformers for Skin Lesion Segmentation

Boundary-aware Transformers for Skin Lesion Segmentation Introduction This is an official release of the paper Boundary-aware Transformers for Skin Le

79 Dec 16, 2022

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

LESA Introduction This repository contains the official implementation of Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Cont

20 Dec 31, 2021

IRON Kaggle project done while doing IRONHACK Bootcamp where we had to analyze and use a Machine Learning Project to predict future sales

IRON Kaggle project done while doing IRONHACK Bootcamp where we had to analyze and use a Machine Learning Project to predict future sales. In this case, we ended up using XGBoost because it was the o

1 Jan 04, 2022

Implementation of Squeezenet in pytorch, pretrained models on Cifar 10 data to come

Pytorch Squeeznet Pytorch implementation of Squeezenet model as described in https://arxiv.org/abs/1602.07360 on cifar-10 Data. The definition of Sque

86 Oct 28, 2022

A PyTorch implementation of QANet.

QANet-pytorch NOTICE I'm very busy these months. I'll return to this repo in about 10 days. Introduction An implementation of QANet with PyTorch. Any

343 Nov 03, 2022

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

AugMix Introduction We propose AugMix, a data processing technique that mixes augmented images and enforces consistent embeddings of the augmented ima

876 Dec 17, 2022

Pyeventbus: a publish/subscribe event bus

pyeventbus pyeventbus is a publish/subscribe event bus for Python 2.7. simplifies the communication between python classes decouples event senders and

15 Apr 21, 2022

Create Own QR code with Python

Create-Own-QR-code Create Own QR code with Python SO guys in here, you have to install pyqrcode 2. open CMD and type python -m pip install pyqrcode

10 Jul 13, 2022

Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized C

139 Jan 04, 2023

Offical implementation of Shunted Self-Attention via Multi-Scale Token Aggregation

Related tags

Overview

Shunted Transformer

Training from scratch

Training

Model Zoo

Citation

Owner

VIsually-Pivoted Audio and(N) Text

This is the codebase for the ICLR 2021 paper Trajectory Prediction using Equivariant Continuous Convolution

Boundary-aware Transformers for Skin Lesion Segmentation

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

IRON Kaggle project done while doing IRONHACK Bootcamp where we had to analyze and use a Machine Learning Project to predict future sales

Implementation of Squeezenet in pytorch, pretrained models on Cifar 10 data to come

A PyTorch implementation of QANet.

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Pyeventbus: a publish/subscribe event bus

Create Own QR code with Python

Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

Pytorch Implementation of Auto-Compressing Subset Pruning for Semantic Image Segmentation

[ICLR'19] Trellis Networks for Sequence Modeling

Code for "Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo"

Virtual hand gesture mouse using a webcam

Code for the submitted paper Surrogate-based cross-correlation for particle image velocimetry

An executor that performs image segmentation on fashion items

Automatic differentiation with weighted finite-state transducers.

Implementation of paper "Self-supervised Learning on Graphs:Deep Insights and New Directions"

Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.