This is the official repository for our paper: ''Pruning Self-attentions into Convolutional Layers in Single Path''.

Last update: Dec 26, 2022

Related tags

Overview

Pruning Self-attentions into Convolutional Layers in Single Path

This is the official repository for our paper: Pruning Self-attentions into Convolutional Layers in Single Path by Haoyu He, Jing liu, Zizheng Pan, Jianfei Cai, Jing Zhang, Dacheng Tao and Bohan Zhuang.

Introduction:

To reduce the massive computational resource consumption for ViTs and add convolutional inductive bias, our SPViT prunes pre-trained ViT models into accurate and compact hybrid models by pruning self-attentions into convolutional layers. Thanks to the proposed weight-sharing scheme between self-attention and convolutional layers that cast the search problem as finding which subset of parameters to use, our SPViT has significantly reduced search cost.

Getting started:

In this repository, we provide code for pruning two representative ViT models.

SPViT-DeiT that prunes DeiT. Please see SPViT_DeiT/README.md for details.
SPViT-Swin that prunes Swin. Please see SPViT_Swin/README.md for details.

If you find our paper useful, please consider cite:

@article{he2021Pruning,
  title={Pruning Self-attentions into Convolutional Layersin Single Path},
  author={He, Haoyu and Liu, Jing and Pan, Zizheng and Cai, Jianfei and Zhang, Jing and Tao, Dacheng and Zhuang, Bohan},
  journal={arXiv preprint arXiv:2111.11802},
  year={2021}
}

This repo contains the official implementations of EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis This repo contains the official implementations of EigenDamage: Structured Prunin

107 Apr 20, 2022

Official pytorch code for "APP: Anytime Progressive Pruning"

APP: Anytime Progressive Pruning Diganta Misra1,2,3, Bharat Runwal2,4, Tianlong Chen5, Zhangyang Wang5, Irina Rish1,3 1 Mila - Quebec AI Institute,2 L

12 Nov 22, 2022

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes Implementation of CoSMA: Convolutional Semi-Regular Mesh Autoencoder arXiv p

10 Oct 11, 2022

Official Repository for the ICCV 2021 paper "PixelSynth: Generating a 3D-Consistent Experience from a Single Image"

PixelSynth: Generating a 3D-Consistent Experience from a Single Image (ICCV 2021) Chris Rockwell, David F. Fouhey, and Justin Johnson [Project Website

95 Nov 22, 2022

The repository offers the official implementation of our paper in PyTorch.

Cloth Interactive Transformer (CIT) Cloth Interactive Transformer for Virtual Try-On Bin Ren1, Hao Tang1, Fanyang Meng2, Runwei Ding3, Ling Shao4, Phi

49 Dec 1, 2022

The official repository for our paper "The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers". We significantly improve the systematic generalization of transformer models on a variety of datasets using simple tricks and careful considerations.

Codebase for training transformers on systematic generalization datasets. The official repository for our EMNLP 2021 paper The Devil is in the Detail:

57 Nov 21, 2022

PyTorch Implementation of the SuRP algorithm by the authors of the AISTATS 2022 paper "An Information-Theoretic Justification for Model Pruning"

PyTorch Implementation of the SuRP algorithm by the authors of the AISTATS 2022 paper "An Information-Theoretic Justification for Model Pruning".

8 Dec 8, 2022

Code for our ICASSP 2021 paper: SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks (paper) By Qing-Long Zhang and Yu-Bin Yang [State Key Laboratory for Novel Software T

199 Jan 8, 2023

PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network"

HAN PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network" This repository is for HAN introduced in the

140 Nov 23, 2022

This is the official repository for our paper: ''Pruning Self-attentions into Convolutional Layers in Single Path''.

Related tags

Overview

Pruning Self-attentions into Convolutional Layers in Single Path

Introduction:

Getting started:

You might also like...

This repo contains the official implementations of EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

Official pytorch code for "APP: Anytime Progressive Pruning"

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

Official Repository for the ICCV 2021 paper "PixelSynth: Generating a 3D-Consistent Experience from a Single Image"

The repository offers the official implementation of our paper in PyTorch.

The official repository for our paper "The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers". We significantly improve the systematic generalization of transformer models on a variety of datasets using simple tricks and careful considerations.

PyTorch Implementation of the SuRP algorithm by the authors of the AISTATS 2022 paper "An Information-Theoretic Justification for Model Pruning"

Code for our ICASSP 2021 paper: SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network"

Releases(1.0)

1.0(Dec 4, 2021)

Owner

Zhuang AI Group

Colab notebook and additional materials for Python-driven analysis of redlining data in Philadelphia

Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks (MAPDN)

Official implementation of the ICCV 2021 paper: "The Power of Points for Modeling Humans in Clothing".

Revisiting Temporal Alignment for Video Restoration

Transformer part of 12th place solution in Riiid! Answer Correctness Prediction

Labelbox is the fastest way to annotate data to build and ship artificial intelligence applications

Example how to deploy deep learning model with aiohttp.

Block-wisely Supervised Neural Architecture Search with Knowledge Distillation (CVPR 2020)

This is my research project for the Irving Center for Cancer Dynamics/Azizi Lab, Columbia University.

The project covers common metrics for super-resolution performance evaluation.

SAFL: A Self-Attention Scene Text Recognizer with Focal Loss

PyTorch reimplementation of the Smooth ReLU activation function proposed in the paper "Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations" [arXiv 2022].

LSTC: Boosting Atomic Action Detection with Long-Short-Term Context

This codebase is the official implementation of Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS2021, Spotlight)

FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes

Fuzzing the Kernel Using Unicornafl and AFL++

USAD - UnSupervised Anomaly Detection on multivariate time series

Predict bus arrival time using VertexAI and Nvidia's Jetson Nano

Official implementation of CATs: Cost Aggregation Transformers for Visual Correspondence NeurIPS'21

Cave Generation using metaballs in Blender. Originally created by sdfgeoff, Edited by Myself (Archie Jaskowicz).