BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels

DaReCzech is a dataset for text relevance ranking in Czech

Transfer-Learn is an open-source and well-documented library for Transfer Learning.

Low Complexity Channel estimation with Neural Network Solutions

Flower - A Friendly Federated Learning Framework

Learning to Initialize Neural Networks for Stable and Efficient Training

PyStan, a Python interface to Stan, a platform for statistical modeling. Documentation: https://pystan.readthedocs.io

Code for the paper Language as a Cognitive Tool to Imagine Goals in Curiosity Driven Exploration

BboxToolkit is a tiny library of special bounding boxes.

Repo for "TableParser: Automatic Table Parsing with Weak Supervision from Spreadsheets" at [email protected]

Artifacts for paper "MMO: Meta Multi-Objectivization for Software Configuration Tuning"

Human motion synthesis using Unity3D

MAUS: A Dataset for Mental Workload Assessment Using Wearable Sensor - Baseline system

labelpix is a graphical image labeling interface for drawing bounding boxes

The implementation code for "DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction"

This repository contains the code to replicate the analysis from the paper "Moving On - Investigating Inventors' Ethnic Origins Using Supervised Learning"

Open source implementation of "A Self-Supervised Descriptor for Image Copy Detection" (SSCD).

A simple and extensible library to create Bayesian Neural Network layers on PyTorch.

Unofficial PyTorch Implementation of AHDRNet (CVPR 2019)

Anderson Acceleration for Deep Learning