Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Last update: Dec 25, 2022

Overview

Hourglass Transformer - Pytorch (wip)

Implementation of Hourglass Transformer, in Pytorch. It will also contain some of my own ideas about how to make it work better.

Citations

@misc{nawrot2021hierarchical,
    title   = {Hierarchical Transformers Are More Efficient Language Models}, 
    author  = {Piotr Nawrot and Szymon Tworkowski and Michał Tyrolski and Łukasz Kaiser and Yuhuai Wu and Christian Szegedy and Henryk Michalewski},
    year    = {2021},
    eprint  = {2110.13711},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Larger Google Sat2Map dataset This dataset extends the aerial ⟷ Maps dataset used in pix2pix (Isola et al., CVPR17). The provide script download_sat2m

34 Dec 28, 2022

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

GD-Thief Red Team tool for exfiltrating files from a target's Google Drive that you(the attacker) has access to, via the Google Drive API. This includ

39 Dec 27, 2022

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

google-drive-to-sqlite Create a SQLite database containing metadata from Google

140 Dec 4, 2022

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

BasicRL: easy and fundamental codes for deep reinforcement learning BasicRL is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up. It is

12 Apr 28, 2022

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

1.4k Dec 30, 2022

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

ImageProcessingTransformer Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

61 Jan 1, 2023

Transformer - Transformer in PyTorch

Transformer 完成进度 Embeddings and PositionalEncoding with example. MultiHeadAttent

1 Jan 6, 2022

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Deep Daze mist over green hills shattered plates on the grass cosmic love and attention a time traveler in the crowd life during the plague meditative

4.4k Jan 3, 2023

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

coax is built on top of JAX, but it doesn't have an explicit dependence on the jax python package. The reason is that your version of jaxlib will depend on your CUDA version.

128 Dec 27, 2022

Comments

No option for rotary positional embeddings

For the ImageNet64 generation experiments, the original authors used rotary positional embeddings. I think it would be worth adding this feature to the repository.

I can prepare a pull request that solves this~

opened by vvvm23 4
Issue with an example

@lucidrains Just Awesome to see this super fast launch !

The third example under "For Images...." throws the following error .,

Then , when I gave the inputs with num_tokens = 20000, max_seq_len = 1024, I got the following runtime error

opened by nsankar 1

Releases(0.0.6)

0.0.6(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Nov 9, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

MADT: Offline Pre-trained Multi-Agent Decision Transformer

MADT: Offline Pre-trained Multi-Agent Decision Transformer A link to our paper can be found on Arxiv. Overview Official codebase for Offline Pre-train

51 Dec 21, 2022

Code for: https://berkeleyautomation.github.io/bags/

DeformableRavens Code for the paper Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks. Here is the

121 Dec 30, 2022

Training a Resilient Q-Network against Observational Interference, Causal Inference Q-Networks

Obs-Causal-Q-Network AAAI 2022 - Training a Resilient Q-Network against Observational Interference Preprint | Slides | Colab Demo | Environment Setup

23 Nov 21, 2022

Compute descriptors for 3D point cloud registration using a multi scale sparse voxel architecture

MS-SVConv : 3D Point Cloud Registration with Multi-Scale Architecture and Self-supervised Fine-tuning Compute features for 3D point cloud registration

42 Jul 25, 2022

BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation This is a demo implementation of BYOL for Audio (BYOL-A), a self-sup

160 Jan 04, 2023

Local Multi-Head Channel Self-Attention for FER2013

LHC-Net Local Multi-Head Channel Self-Attention This repository is intended to provide a quick implementation of the LHC-Net and to replicate the resu

12 Jan 04, 2023

Files for a tutorial to train SegNet for road scenes using the CamVid dataset

SegNet and Bayesian SegNet Tutorial This repository contains all the files for you to complete the 'Getting Started with SegNet' and the 'Bayesian Seg

800 Dec 31, 2022

An implementation of EWC with PyTorch

EWC.pytorch An implementation of Elastic Weight Consolidation (EWC), proposed in James Kirkpatrick et al. Overcoming catastrophic forgetting in neural

166 Dec 22, 2022

Generate image analogies using neural matching and blending

neural image analogies This is basically an implementation of this "Image Analogies" paper, In our case, we use feature maps from VGG16. The patch mat

3.5k Jan 08, 2023

Official implementation of the paper Visual Parser: Representing Part-whole Hierarchies with Transformers

Visual Parser (ViP) This is the official implementation of the paper Visual Parser: Representing Part-whole Hierarchies with Transformers. Key Feature

117 Dec 11, 2022

PyTorch ,ONNX and TensorRT implementation of YOLOv4

4.2k Jan 01, 2023

Official PyTorch implementation of "BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation" (NeurIPS 2021)

BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation Official PyTorch implementation of the NeurIPS 2021 paper Mingcong Liu, Qiang

462 Dec 29, 2022

Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI

Related tags

Overview

Hourglass Transformer - Pytorch (wip)

Citations

You might also like...

A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

Google-drive-to-sqlite - Create a SQLite database containing metadata from Google Drive

BasicRL: easy and fundamental codes for deep reinforcement learning。It is an improvement on rainbow-is-all-you-need and OpenAI Spinning Up.

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

Transformer - Transformer in PyTorch

Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Plug-n-Play Reinforcement Learning in Python with OpenAI Gym and JAX

Comments

No option for rotary positional embeddings

Issue with an example

Releases(0.0.6)

0.0.6(Nov 9, 2021)

0.0.5(Nov 9, 2021)

0.0.4(Nov 9, 2021)

0.0.3(Nov 9, 2021)

0.0.2(Nov 9, 2021)

0.0.1(Nov 9, 2021)

Owner

Phil Wang

MADT: Offline Pre-trained Multi-Agent Decision Transformer

Code for: https://berkeleyautomation.github.io/bags/

Training a Resilient Q-Network against Observational Interference, Causal Inference Q-Networks

Compute descriptors for 3D point cloud registration using a multi scale sparse voxel architecture

BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation

Local Multi-Head Channel Self-Attention for FER2013

Files for a tutorial to train SegNet for road scenes using the CamVid dataset

An implementation of EWC with PyTorch

Generate image analogies using neural matching and blending

Official implementation of the paper Visual Parser: Representing Part-whole Hierarchies with Transformers

PyTorch ,ONNX and TensorRT implementation of YOLOv4

Official PyTorch implementation of "BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation" (NeurIPS 2021)

Official implementation of the paper Momentum Capsule Networks (MoCapsNet)

Doods2 - API for detecting objects in images and video streams using Tensorflow

Fiddle is a Python-first configuration library particularly well suited to ML applications.

HuSpaCy: industrial-strength Hungarian natural language processing

Code for visualizing the loss landscape of neural nets

HyDiff: Hybrid Differential Software Analysis

Code used for the results in the paper "ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning"

[ICCV2021] Official code for "Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition"