A GPT, made only of MLPs, in Jax

Last update: Sep 27, 2022

Overview

MLP GPT - Jax (wip)

A GPT, made only of MLPs, in Jax. The specific MLP to be used are gMLPs with the Spatial Gating Units.

Install

$ pip install mlp-gpt-jax

Usage

from jax import random, numpy as np
from mlp_gpt_jax import MLPGpt

gpt = MLPGpt(
    num_tokens = 20000,
    dim = 512,
    depth = 6,
    seq_len = 512
)

key    = random.PRNGKey(0)
seq    = random.randint(key, (512,), 0, 20000)

params = gpt.init(key, seq)
logits = gpt.apply(params, seq) # (512, 20000)

Citations

@misc{liu2021pay,
    title   = {Pay Attention to MLPs}, 
    author  = {Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le},
    year    = {2021},
    eprint  = {2105.08050},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG}
}

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Storium GPT-2 Models This is the official repository for the GPT-2 models described in the EMNLP 2020 paper [STORIUM: A Dataset and Evaluation Platfor

27 Dec 20, 2022

Training data extraction on GPT-2

Training data extraction from GPT-2 This repository contains code for extracting training data from GPT-2, following the approach outlined in the foll

62 Dec 7, 2022

Simple Text-Generator with OpenAI gpt-2 Pytorch Implementation

GPT2-Pytorch with Text-Generator Better Language Models and Their Implications Our model, called GPT-2 (a successor to GPT), was trained simply to pre

775 Jan 8, 2023

ChatBot-Pytorch - A GPT-2 ChatBot implemented using Pytorch and Huggingface-transformers

ChatBot-Pytorch A GPT-2 ChatBot implemented using Pytorch and Huggingface-transf

42 Dec 9, 2022

AI-Bot - 一个基于watermelon改造的OpenAI-GPT-2的智能机器人

AI-Bot 一个基于watermelon改造的OpenAI-GPT-2的智能机器人在Binder上直接运行测试目前有两种实现方式 TF2的GPT-2 TF

9 Nov 16, 2022

Building Ellee — A GPT-3 and Computer Vision Powered Talking Robotic Teddy Bear With Human Level Conversation Intelligence

Using an object detection and facial recognition system built on MobileNetSSDV2 and Dlib and running on an NVIDIA Jetson Nano, a GPT-3 model, Google Speech Recognition, Amazon Polly and servo motors, I built Ellee - a robotic teddy bear who can move her head and converse naturally.

24 Oct 26, 2022

MAGMA - a GPT-style multimodal model that can understand any combination of images and language

MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning Authors repo (alphabetical) Constantin (CoEich), Mayukh (Mayukh

331 Jan 3, 2023

FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX.

FedJAX: Federated learning with JAX What is FedJAX? FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX. FedJAX priori

208 Dec 14, 2022

Flax is a neural network ecosystem for JAX that is designed for flexibility.

Flax: A neural network library and ecosystem for JAX designed for flexibility Overview | Quick install | What does Flax look like? | Documentation See

3.9k Jan 2, 2023

Comments

mistake in parameter initialization

floor division will always return 0 :(

https://github.com/lucidrains/mlp-gpt-jax/blob/c8a6d7738562e44d3c0b3018c83ae577f7931e78/mlp_gpt_jax/mlp_gpt_jax.py#L75

opened by guyd1995 1

Releases(0.0.19)

0.0.19(Jun 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.18(Jun 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.17(Jun 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.16(Jun 3, 2021)

Source code(tar.gz)
Source code(zip)
0.0.15(Jun 3, 2021)

Source code(tar.gz)
Source code(zip)
0.0.14(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.12(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.11(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.10(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.9(Jun 2, 2021)

Source code(tar.gz)
Source code(zip)
0.0.8(May 29, 2021)

Source code(tar.gz)
Source code(zip)
0.0.7(May 27, 2021)

Source code(tar.gz)
Source code(zip)
0.0.6(May 26, 2021)

Source code(tar.gz)
Source code(zip)
0.0.5(May 25, 2021)

Source code(tar.gz)
Source code(zip)
0.0.4(May 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(May 22, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(May 21, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(May 21, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention

GitHub Repository

Source code for "FastBERT: a Self-distilling BERT with Adaptive Inference Time".

FastBERT Source code for "FastBERT: a Self-distilling BERT with Adaptive Inference Time". Good News 2021/10/29 - Code: Code of FastPLM is released on

584 Jan 02, 2023

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

SoC-Winter Checkout some cool self-projects you can try your hands on to curb your boredom this December! These are short projects that you can do you

29 Nov 08, 2022

利用yolov5和TensorRT从0到1实现目标检测的模型训练到模型部署全过程

写在前面利用TensorRT加速推理速度是以时间换取精度的做法，意味着在推理速度上升的同时将会有精度的下降，不过不用太担心，精度下降微乎其微。此外，要有NVIDIA显卡，经测试，CUDA10.2可以支持20系列显卡及以下，30系列显卡需要CUDA11.x的支持，并且目前有bug。默认你已经完成了

6 Jul 28, 2022

Flower classification model that classifies flowers in 10 classes made using transfer learning (~85% accuracy).

flower-classification-inceptionV3 Flower classification model that classifies flowers in 10 classes. Training and validation are done using a pre-anot

1 Dec 12, 2021

Image-to-image translation with conditional adversarial nets

pix2pix Project | Arxiv | PyTorch Torch implementation for learning a mapping from input images to output images, for example: Image-to-Image Translat

9.3k Jan 08, 2023

A basic duplicate image detection service using perceptual image hash functions and nearest neighbor search, implemented using faiss, fastapi, and imagehash

Duplicate Image Detection Getting Started Install dependencies pip install -r requirements.txt Run service python main.py Testing Test with pytest How

21 Nov 11, 2022

A GPT, made only of MLPs, in Jax

Related tags

Overview

MLP GPT - Jax (wip)

Install

Usage

Citations

You might also like...

Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Training data extraction on GPT-2

Simple Text-Generator with OpenAI gpt-2 Pytorch Implementation

ChatBot-Pytorch - A GPT-2 ChatBot implemented using Pytorch and Huggingface-transformers

AI-Bot - 一个基于watermelon改造的OpenAI-GPT-2的智能机器人

Building Ellee — A GPT-3 and Computer Vision Powered Talking Robotic Teddy Bear With Human Level Conversation Intelligence

MAGMA - a GPT-style multimodal model that can understand any combination of images and language

FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX.

Flax is a neural network ecosystem for JAX that is designed for flexibility.

Comments

mistake in parameter initialization

Releases(0.0.19)

0.0.19(Jun 23, 2021)

0.0.18(Jun 22, 2021)

0.0.17(Jun 22, 2021)

0.0.16(Jun 3, 2021)

0.0.15(Jun 3, 2021)

0.0.14(Jun 2, 2021)

0.0.12(Jun 2, 2021)

0.0.11(Jun 2, 2021)

0.0.10(Jun 2, 2021)

0.0.9(Jun 2, 2021)

0.0.8(May 29, 2021)

0.0.7(May 27, 2021)

0.0.6(May 26, 2021)

0.0.5(May 25, 2021)

0.0.4(May 23, 2021)

0.0.3(May 22, 2021)

0.0.2(May 21, 2021)

0.0.1(May 21, 2021)

Owner

Phil Wang

Source code for "FastBERT: a Self-distilling BERT with Adaptive Inference Time".

Checkout some cool self-projects you can try your hands on to curb your boredom this December!

利用yolov5和TensorRT从0到1实现目标检测的模型训练到模型部署全过程

Flower classification model that classifies flowers in 10 classes made using transfer learning (~85% accuracy).

Image-to-image translation with conditional adversarial nets

A basic duplicate image detection service using perceptual image hash functions and nearest neighbor search, implemented using faiss, fastapi, and imagehash

CRLT: A Unified Contrastive Learning Toolkit for Unsupervised Text Representation Learning

PyTorch implementation of Wide Residual Networks with 1-bit weights by McDonnell (ICLR 2018)

Objax Apache-2Objax (🥉19 · ⭐ 580) - Objax is a machine learning framework that provides an Object.. Apache-2 jax

AirCode: A Robust Object Encoding Method

Easy way to add GoogleMaps to Flask applications. maintainer: @getcake

Continual Learning of Electronic Health Records (EHR).

Code for "FPS-Net: A convolutional fusion network for large-scale LiDAR point cloud segmentation".

Differentiable Simulation of Soft Multi-body Systems

A python library for implementing a recommender system

Bling's Object detection tool

《Truly shift-invariant convolutional neural networks》(2021)

Code for the TIP 2021 Paper "Salient Object Detection with Purificatory Mechanism and Structural Similarity Loss"

Simple object detection app with streamlit

VisionKG: Vision Knowledge Graph