Source code for "Efficient Training of BERT by Progressively Stacking"

Overview

Introduction

This repository is the code to reproduce the result of Efficient Training of BERT by Progressively Stacking. The code is based on Fairseq.

Requirements and Installation

  • PyTorch >= 1.0.0
  • For training new models, you'll also need an NVIDIA GPU and NCCL
  • Python version 3.7

After PyTorch is installed, you can install requirements with:

pip install -r requirements.txt

Getting Started

Step 1:

bash install.sh

This script downloads:

  1. Moses Decoder
  2. Subword NMT
  3. Fast BPE (In the next steps, we use Subword NMT instead of Fast BPE. Recommended if you want to generate your own dictionary on a large-scale dataset.)

These library will do cleaning, tokenization, and BPE encoding for GLUE data in step 3. They will also be helpful if you want to make your own corpus for BERT training or if you want to test our model on your own tasks.

Step 2:

bash reproduce_bert.sh

This script runs progressive stacking and train a BERT. The code is tested on 4 Tesla P40 GPUs (24GB Gmem). For different hardware, you probably need to change the maximum number of tokens per batch (by changing max-tokens and update-freq).

Step 3:

bash reproduce_glue.sh

This script fine-tunes the BERT trained in step 2. The script chooses the checkpoint trained for 400K steps, which is the same as the stacking model in our paper.

Cite

@InProceedings{pmlr-v97-gong19a,
  title = 	 {Efficient Training of {BERT} by Progressively Stacking},
  author = 	 {Gong, Linyuan and He, Di and Li, Zhuohan and Qin, Tao and Wang, Liwei and Liu, Tieyan},
  booktitle = 	 {Proceedings of the 36th International Conference on Machine Learning},
  pages = 	 {2337--2346},
  year = 	 {2019},
  editor = 	 {Chaudhuri, Kamalika and Salakhutdinov, Ruslan},
  volume = 	 {97},
  series = 	 {Proceedings of Machine Learning Research},
  address = 	 {Long Beach, California, USA},
  month = 	 {09--15 Jun},
  publisher = 	 {PMLR},
  pdf = 	 {http://proceedings.mlr.press/v97/gong19a/gong19a.pdf},
  url = 	 {http://proceedings.mlr.press/v97/gong19a.html},
}
Owner
Gong Linyuan
Gong Linyuan
Status-embed - Cool open source profile embed for Discord

Current Status : Incomplete Status Embed Status Embed is an awesome open source

Ritabrata Das 2 Feb 17, 2022
Collection of script to manage WLED devices

Collection of script to manage WLED devices

Daniel Poelzleithner 4 Sep 26, 2022
YouTube-Discord-Bot - Discord Bot to Search YouTube

YouTube Bot Info YouTube Bot is a discord bot where you can search for anything

Riceblades11 10 Mar 05, 2022
The system to host your files on the Discord application

Distorage The system to host your files on the Discord application Documentation Documentation Distorage How to use the package You can install it wit

6 Jun 27, 2022
Python library for using SMS.ir web services

smsir smsir is a Python library for using SMS web services www.sms.ir Installation Use the package manager pip to install smsir. pip install smsir Usa

mohammad reza 2 Oct 14, 2022
Luna Rush Auto Clicker Bot

Luna Rush Auto Clicker Bot Se o aplicativo lhe ajudar de alguma forma, uma doação para ajudar a pagar a conta de luz sempre é bem vinda ;) Wallet Smar

Walter Discher Cechinel 29 Dec 20, 2022
pokemon-colorscripts compatible for mac

Pokemon colorscripts some scripts to print out images of pokemons to terminal. Inspired by DT's colorscripts compilation Description Prints out colore

43 Jan 06, 2023
WhatSender is a python package that allows you to send Whatsapp messages at a given time.

WhatSender is a python package that allows you to send Whatsapp messages at a given time.

IdoBarel 0 Apr 21, 2022
Simulation artifacts, core components and configuration files to integrate AWS DeepRacer device with ROS Navigation stack.

AWS DeepRacer Overview The AWS DeepRacer Evo vehicle is a 1/18th scale Wi-Fi enabled 4-wheel ackermann steering platform that features two RGB cameras

AWS DeepRacer 31 Nov 21, 2022
Bot to notify when vaccine appointments are available

Vaccine Watch Bot to notify when vaccine appointments are available. Supports checking Hy-Vee, Walgreens, CVS, Walmart, Cosentino's stores (KC), and B

Peter Carnesciali 37 Aug 13, 2022
a Disqus alternative

Isso – a commenting server similar to Disqus Isso – Ich schrei sonst – is a lightweight commenting server written in Python and JavaScript. It aims to

Martin Zimmermann 4.7k Jan 02, 2023
An async python wrapper to interact with the Steam API and its CMs

steam.py A modern, easy to use, and async ready package to interact with the Steam API. Heavily inspired by discord.py and borrowing functionality fro

James Hilton-Balfe 90 Dec 15, 2022
Example-bot-discord - Example bot discord xD

example-python-bot-discord Clone this repository Grab a token on Discord's devel

Amitminer 1 Mar 14, 2022
Tinyman Python SDK

tinyman-py-sdk Tinyman Python SDK Design Goal This SDK is designed for automated interaction with the Tinyman AMM. It will be most useful for develope

Tinyman 113 Dec 30, 2022
A Python API For Questionnaire

Инструкция по разворачиванию приложения Окружение проекта: python 3.8 Django 2.2.10 djangorestframework Склонируйте репозиторий с помощью git: git clo

2 Feb 14, 2022
A discord bot written in python

arch-bot A discord bot written in python prefix: . help: .help Installation Requirements A discord bot token Your user id Python installed. For window

3 Jan 10, 2022
Erhalten Sie wichtige Warnmeldungen des Bevölkerungsschutzes für Gefahrenlagen wie zum Beispiel Gefahrstoffausbreitung oder Unwetter per Programmierschnittstelle.

nina-api Erhalten Sie wichtige Warnmeldungen des Bevölkerungsschutzes für Gefahrenlagen wie zum Beispiel Gefahrstoffausbreitung oder Unwetter per Prog

Bundesstelle für Open Data 68 Dec 19, 2022
Semplice pagina di informazione per sapere se e quando è uscito Joypad, il podcast a tema videoludico di Matteo Bordone (Corri!), Francesco Fossetti (Salta!) e Alessandro Zampini (Spara! per finta).

È uscito Joypad? Semplice pagina di informazione per sapere se e quando è uscito Joypad, il podcast a tema videoludico di Matteo Bordone (Corri!), Fra

Paolo Donadeo 32 Jan 02, 2023
Running Performance Calculator

Running Performance Calculator 👉 Have you ever wondered if you ran 10km at 2000

Davide Liu 6 Oct 26, 2022
An interactive and multi-function Telegram bot, made especially for Telegram groups.

PyKorone An interaction and fun bot for Telegram groups, having some useful and other useless commands. Created as an experiment and learning bot but

Amano Team 17 Nov 12, 2022