Source code for "Efficient Training of BERT by Progressively Stacking"

Overview

Introduction

This repository is the code to reproduce the result of Efficient Training of BERT by Progressively Stacking. The code is based on Fairseq.

Requirements and Installation

  • PyTorch >= 1.0.0
  • For training new models, you'll also need an NVIDIA GPU and NCCL
  • Python version 3.7

After PyTorch is installed, you can install requirements with:

pip install -r requirements.txt

Getting Started

Step 1:

bash install.sh

This script downloads:

  1. Moses Decoder
  2. Subword NMT
  3. Fast BPE (In the next steps, we use Subword NMT instead of Fast BPE. Recommended if you want to generate your own dictionary on a large-scale dataset.)

These library will do cleaning, tokenization, and BPE encoding for GLUE data in step 3. They will also be helpful if you want to make your own corpus for BERT training or if you want to test our model on your own tasks.

Step 2:

bash reproduce_bert.sh

This script runs progressive stacking and train a BERT. The code is tested on 4 Tesla P40 GPUs (24GB Gmem). For different hardware, you probably need to change the maximum number of tokens per batch (by changing max-tokens and update-freq).

Step 3:

bash reproduce_glue.sh

This script fine-tunes the BERT trained in step 2. The script chooses the checkpoint trained for 400K steps, which is the same as the stacking model in our paper.

Cite

@InProceedings{pmlr-v97-gong19a,
  title = 	 {Efficient Training of {BERT} by Progressively Stacking},
  author = 	 {Gong, Linyuan and He, Di and Li, Zhuohan and Qin, Tao and Wang, Liwei and Liu, Tieyan},
  booktitle = 	 {Proceedings of the 36th International Conference on Machine Learning},
  pages = 	 {2337--2346},
  year = 	 {2019},
  editor = 	 {Chaudhuri, Kamalika and Salakhutdinov, Ruslan},
  volume = 	 {97},
  series = 	 {Proceedings of Machine Learning Research},
  address = 	 {Long Beach, California, USA},
  month = 	 {09--15 Jun},
  publisher = 	 {PMLR},
  pdf = 	 {http://proceedings.mlr.press/v97/gong19a/gong19a.pdf},
  url = 	 {http://proceedings.mlr.press/v97/gong19a.html},
}
Owner
Gong Linyuan
Gong Linyuan
The simple way of using Imgur.

PyImgur The simple way of using Imgur. You can upload images, download images, read comments, update your albums, message people and more. In fact, yo

Andreas Damgaard Pedersen 120 Dec 06, 2022
Unofficial API wrapper for seedr.cc

Seedr API Unofficial API wrapper for seedr.cc Inspired by theabbie's seedr-api Powered by @harp_tech (Telegram) How to use You can install lib via git

Anjana Madu 49 Oct 24, 2022
TwitterBot-ImageCollector - Twitter bot that collects images from likes saves the image

TwitterBot-ImageCollector Bot de Twitter que recolecta imagenes a partir de los

Gx3 Studios 4 Jun 01, 2022
Presentation and code files for the talk at PyCon Indonesia

pycon-indonesia Presentation and code files for the talk at PyCon Indonesia. Files used for the PyCon Indonesia presentation. [Directory Includes:] Be

Neeraj Pandey 2 Dec 04, 2021
Modern, privacy-friendly, and detailed web analytics that works without cookies or JS.

Modern, privacy-friendly, and cookie-free web analytics. Getting started » Screenshots • Features • Office Hours Motivation There are a lot of web ana

R. Miles McCain 2.1k Jan 03, 2023
Customizable and open-sourced bot for a few private servers

MarlBot A private bot for controlling monkeys and turtles. Why does this bot exist? The bot exists as a general-purpose community bot for a select few

KR 1 Jan 18, 2022
Wrapper around the Mega API

python-mega Overview Wrapper around the Mega API. Based on the work of Julien Marchand. Installation Install using pip, including any optional package

Juan Riaza 104 Nov 26, 2022
CryptoBar - A simple MenuBar app that shows the price of 3 cryptocurrencies

CryptoBar A very simple MenuBar app that shows the price of the following crypto

4 Jul 04, 2022
A decentralized messaging daemon built on top of the Kademlia routing protocol.

parakeet-message A decentralized messaging daemon built on top of the Kademlia routing protocol. Now that you are done laughing... pictures what is it

Jonathan Abbott 3 Apr 23, 2022
Analytics platform for Telegram Channels

Tele-Report Analytics platform for Telegram Channels 🚧 👷 Getting Started 1- Install redis and postgreSQL (it would be more generic in future, like u

2 Oct 11, 2022
Paginator for Dis-Snek Python Discord API wrapper

snek-paginator Paginator for Dis-Snek Python Discord API wrapper Installation: pip install -U snek-paginator Basic Example: from dis_snek.client impo

1 Nov 04, 2021
8300-account-nuker - A simple accoutn nuker or can use it full closing dm and leaving server

8300 ACCOUNT NUKER VERISON: its just simple accoutn nuker or can use it full clo

†† 5 Jan 26, 2022
A Powerfull Userbot Telegram PandaX_Userbot, Vc Music Userbot + Bot Manager based Telethon

Support ☑ CREDITS THANKS YOU VERRY MUCH FOR ALL Telethon Pyrogram TeamUltroid TeamUserge CatUserbot pytgcalls Dan Lainnya

22 Dec 25, 2022
Upload comma-delimited files to biglocalnews.org in your GitHub Action

Upload comma-delimited files to biglocalnews.org in your GitHub Action Inputs api-key: Your biglocalnews.org API token. project-id: The identifier of

biglocalnews 1 Apr 20, 2022
An advanced Filter Bot with nearly unlimitted filters

Telegram MTProto API Framework for Python Documentation • Releases • Community Pyrogram from pyrogram import Client, filters app = Client("my_account

Pyrogram 3.2k Jan 05, 2023
A simple Telegram bot that converts a phone number to a direct whatsapp chat link

Open in WhatsApp I was using a great app to open a whatsapp chat with a given number directly without saving that number in my contact list, but I fel

Pathfinder 19 Dec 24, 2022
Discord bot that manages expiration of roles with subscriptions!

Discord bot that manages expiration of roles with subscriptions!

Chakeaw__ 3 Apr 28, 2022
Cord Python API Client

Cord Python API Client The data programming platform for AI 💻 Features Minimal low-level Python client that allows you to interact with Cord's API Su

Cord 52 Nov 25, 2022
Create CDK projects with projen

The Projenator: I'll be back! Description This is a CDKv2 project that takes the grind out of setting up new cdk projects/implementations by using aut

Andrew 2 Dec 11, 2021