BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Avalanche RL: an End-to-End Library for Continual Reinforcement Learning

Full body anonymization - Realistic Full-Body Anonymization with Surface-Guided GANs

Composable transformations of Python+NumPy programsComposable transformations of Python+NumPy programs

This is the official github repository of the Met dataset

Lingvo is a framework for building neural networks in Tensorflow, particularly sequence models.

Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

PyTorch implementation of EigenGAN

AnimationKit: AI Upscaling & Interpolation using Real-ESRGAN+RIFE

Simple Linear 2nd ODE Solver GUI - A 2nd constant coefficient linear ODE solver with simple GUI using euler's method

Code for SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (ACL'2020).

This Jupyter notebook shows one way to implement a simple first-order low-pass filter on sampled data in discrete time.

Implementation of ToeplitzLDA for spatiotemporal stationary time series data.

Official implementation of CATs: Cost Aggregation Transformers for Visual Correspondence NeurIPS'21

A simple Python library for stochastic graphical ecological models

C3D is a modified version of BVLC caffe to support 3D ConvNets.

The `rtdl` library + The official implementation of the paper

PyTorch - Python + Nim

ICLR 2021 i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning

Unofficial pytorch implementation of 'Image Inpainting for Irregular Holes Using Partial Convolutions'

Evaluating deep transfer learning for whole-brain cognitive decoding