A GPT, made only of MLPs, in Jax

Last update: Sep 27, 2022

Overview

MLP GPT - Jax (wip)

A GPT, made only of MLPs, in Jax. The specific MLP to be used are gMLPs with the Spatial Gating Units.

Install

$ pip install mlp-gpt-jax

Usage

from jax import random, numpy as np
from mlp_gpt_jax import MLPGpt

gpt = MLPGpt(
    num_tokens = 20000,
    dim = 512,
    depth = 6,
    seq_len = 512
)

key    = random.PRNGKey(0)
seq    = random.randint(key, (512,), 0, 20000)

params = gpt.init(key, seq)
logits = gpt.apply(params, seq) # (512, 20000)

Citations

@misc{liu2021pay,
    title   = {Pay Attention to MLPs}, 
    author  = {Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le},
    year    = {2021},
    eprint  = {2105.08050},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Storium GPT-2 Models This is the official repository for the GPT-2 models described in the EMNLP 2020 paper [STORIUM: A Dataset and Evaluation Platfor

27 Dec 20, 2022

Training data extraction on GPT-2

Training data extraction from GPT-2 This repository contains code for extracting training data from GPT-2, following the approach outlined in the foll

62 Dec 7, 2022

Simple Text-Generator with OpenAI gpt-2 Pytorch Implementation

GPT2-Pytorch with Text-Generator Better Language Models and Their Implications Our model, called GPT-2 (a successor to GPT), was trained simply to pre

775 Jan 8, 2023

ChatBot-Pytorch - A GPT-2 ChatBot implemented using Pytorch and Huggingface-transformers

ChatBot-Pytorch A GPT-2 ChatBot implemented using Pytorch and Huggingface-transf

42 Dec 9, 2022

AI-Bot - 一个基于watermelon改造的OpenAI-GPT-2的智能机器人

AI-Bot 一个基于watermelon改造的OpenAI-GPT-2的智能机器人在Binder上直接运行测试目前有两种实现方式 TF2的GPT-2 TF

9 Nov 16, 2022

Building Ellee — A GPT-3 and Computer Vision Powered Talking Robotic Teddy Bear With Human Level Conversation Intelligence

Using an object detection and facial recognition system built on MobileNetSSDV2 and Dlib and running on an NVIDIA Jetson Nano, a GPT-3 model, Google Speech Recognition, Amazon Polly and servo motors, I built Ellee - a robotic teddy bear who can move her head and converse naturally.

24 Oct 26, 2022

MAGMA - a GPT-style multimodal model that can understand any combination of images and language

MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning Authors repo (alphabetical) Constantin (CoEich), Mayukh (Mayukh

331 Jan 3, 2023

FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX.

FedJAX: Federated learning with JAX What is FedJAX? FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX. FedJAX priori

208 Dec 14, 2022

Flax is a neural network ecosystem for JAX that is designed for flexibility.

Flax: A neural network library and ecosystem for JAX designed for flexibility Overview | Quick install | What does Flax look like? | Documentation See

3.9k Jan 2, 2023

Comments

mistake in parameter initialization

floor division will always return 0 :(

https://github.com/lucidrains/mlp-gpt-jax/blob/c8a6d7738562e44d3c0b3018c83ae577f7931e78/mlp_gpt_jax/mlp_gpt_jax.py#L75

opened by guyd1995 1

Releases(0.0.19)

0.0.19(Jun 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.18(Jun 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.17(Jun 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.16(Jun 3, 2021)

Source code(tar.gz)
Source code(zip)
0.0.15(Jun 3, 2021)

Source code(tar.gz)
Source code(zip)
0.0.14(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.12(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.11(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.10(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.9(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.8(May 29, 2021)

Source code(tar.gz)
Source code(zip)
0.0.7(May 27, 2021)

Source code(tar.gz)
Source code(zip)
0.0.6(May 26, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(May 25, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(May 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(May 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(May 21, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(May 21, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention

GitHub Repository

This is the 3D Implementation of 《Inconsistency-aware Uncertainty Estimation for Semi-supervised Medical Image Segmentation》

CoraNet This is the 3D Implementation of 《Inconsistency-aware Uncertainty Estimation for Semi-supervised Medical Image Segmentation》 Environment pytor

25 Nov 08, 2022

SpineAI Bilsky Grading With Python

SpineAI-Bilsky-Grading SpineAI Paper with Code 📫 Contact Address correspondence to J.T.P.D.H. (e-mail: james_hallinan AT nuhs.edu.sg) Disclaimer This

[email protected]"> 2 Dec 16, 2021

Attendance Monitoring with Face Recognition using Python

Attendance Monitoring with Face Recognition using Python A python GUI integrated attendance system using face recognition to take attendance. In this

2 Jun 21, 2022

A library for preparing, training, and evaluating scalable deep learning hybrid recommender systems using PyTorch.

collie Collie is a library for preparing, training, and evaluating implicit deep learning hybrid recommender systems, named after the Border Collie do

96 Dec 29, 2022

Implementation of accepted AAAI 2021 paper: Deep Unsupervised Image Hashing by Maximizing Bit Entropy

Deep Unsupervised Image Hashing by Maximizing Bit Entropy This is the PyTorch implementation of accepted AAAI 2021 paper: Deep Unsupervised Image Hash

62 Dec 30, 2022

Image Matching Evaluation

Image Matching Evaluation (IME) IME provides to test any feature matching algorithm on datasets containing ground-truth homographies. Also, one can re

32 Nov 17, 2022

RIM: Reliable Influence-based Active Learning on Graphs.

RIM: Reliable Influence-based Active Learning on Graphs. This repository is the official implementation of RIM. Requirements To install requirements:

4 Aug 29, 2022

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language This repository contains the code, model, and deployment config

16 Oct 23, 2022

In this project we combine techniques from neural voice cloning and musical instrument synthesis to achieve good results from as little as 16 seconds of target data.

Neural Instrument Cloning In this project we combine techniques from neural voice cloning and musical instrument synthesis to achieve good results fro

127 Dec 23, 2022

TensorFlow (Python) implementation of DeepTCN model for multivariate time series forecasting.

DeepTCN TensorFlow TensorFlow (Python) implementation of multivariate time series forecasting model introduced in Chen, Y., Kang, Y., Chen, Y., & Wang

21 Dec 19, 2022

Code for "ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on", accepted at WACV 2021 Generation of Human Behavior Workshop.

ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on [ Paper ] [ Project Page ] This repository contains the code fo

97 Dec 13, 2022

A GPT, made only of MLPs, in Jax

Related tags

Overview

MLP GPT - Jax (wip)

Install

Usage

Citations

You might also like...

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Training data extraction on GPT-2

Simple Text-Generator with OpenAI gpt-2 Pytorch Implementation

ChatBot-Pytorch - A GPT-2 ChatBot implemented using Pytorch and Huggingface-transformers

AI-Bot - 一个基于watermelon改造的OpenAI-GPT-2的智能机器人

Building Ellee — A GPT-3 and Computer Vision Powered Talking Robotic Teddy Bear With Human Level Conversation Intelligence

MAGMA - a GPT-style multimodal model that can understand any combination of images and language

FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX.

Flax is a neural network ecosystem for JAX that is designed for flexibility.

Comments

mistake in parameter initialization

Releases(0.0.19)

0.0.19(Jun 23, 2021)

0.0.18(Jun 22, 2021)

0.0.17(Jun 22, 2021)

0.0.16(Jun 3, 2021)

0.0.15(Jun 3, 2021)

0.0.14(Jun 2, 2021)

0.0.12(Jun 2, 2021)

0.0.11(Jun 2, 2021)

0.0.10(Jun 2, 2021)

0.0.9(Jun 2, 2021)

0.0.8(May 29, 2021)

0.0.7(May 27, 2021)

0.0.6(May 26, 2021)

0.0.5(May 25, 2021)

0.0.4(May 23, 2021)

0.0.3(May 22, 2021)

0.0.2(May 21, 2021)

0.0.1(May 21, 2021)

Owner

Phil Wang

This is the 3D Implementation of 《Inconsistency-aware Uncertainty Estimation for Semi-supervised Medical Image Segmentation》

SpineAI Bilsky Grading With Python

Attendance Monitoring with Face Recognition using Python

A library for preparing, training, and evaluating scalable deep learning hybrid recommender systems using PyTorch.

Implementation of accepted AAAI 2021 paper: Deep Unsupervised Image Hashing by Maximizing Bit Entropy

Image Matching Evaluation

RIM: Reliable Influence-based Active Learning on Graphs.

Sign-to-Speech for Sign Language Understanding: A case study of Nigerian Sign Language

In this project we combine techniques from neural voice cloning and musical instrument synthesis to achieve good results from as little as 16 seconds of target data.

TensorFlow (Python) implementation of DeepTCN model for multivariate time series forecasting.

Code for "ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on", accepted at WACV 2021 Generation of Human Behavior Workshop.

HairCLIP: Design Your Hair by Text and Reference Image

Automatic labeling, conversion of different data set formats, sample size statistics, model cascade

Code for the paper "MASTER: Multi-Aspect Non-local Network for Scene Text Recognition" (Pattern Recognition 2021)

Vector Quantized Diffusion Model for Text-to-Image Synthesis

Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation".

Deploy tensorflow graphs for fast evaluation and export to tensorflow-less environments running numpy.

Official source code of Fast Point Transformer, CVPR 2022

"Learning and Analyzing Generation Order for Undirected Sequence Models" in Findings of EMNLP, 2021

Backdoor Attack through Frequency Domain