BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

The fastest way to visualize GradCAM with your Keras models.

Delving into Localization Errors for Monocular 3D Object Detection, CVPR'2021

Unofficial Alias-Free GAN implementation. Based on rosinality's version with expanded training and inference options.

🤗 Push your spaCy pipelines to the Hugging Face Hub

A curated list and survey of awesome Vision Transformers.

Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Convolutional Neural Network for Text Classification in Tensorflow

YOLOv5 🚀 is a family of object detection architectures and models pretrained on the COCO dataset

Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators..

TCTrack: Temporal Contexts for Aerial Tracking (CVPR2022)

This is the reference implementation for "Coresets via Bilevel Optimization for Continual Learning and Streaming"

PyAF is an Open Source Python library for Automatic Time Series Forecasting built on top of popular pydata modules.

CS506-Spring2022 - Code and Slides for Boston University CS 506

Meaningful titles for tabs and PDF downloads! Also supports tab search.

Residual Pathway Priors for Soft Equivariance Constraints

For medical image segmentation

Paaster is a secure by default end-to-end encrypted pastebin built with the objective of simplicity.

Code for "OctField: Hierarchical Implicit Functions for 3D Modeling (NeurIPS 2021)"

i3DMM: Deep Implicit 3D Morphable Model of Human Heads