BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Synthetic LiDAR sequential point cloud dataset with point-wise annotations

Ego4d dataset repository. Download the dataset, visualize, extract features & example usage of the dataset

Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

QT Py Media Knob using rotary encoder & neopixel ring

This repository provides some of the code implemented and the data used for the work proposed in "A Cluster-Based Trip Prediction Graph Neural Network Model for Bike Sharing Systems".

Library extending Jupyter notebooks to integrate with Apache TinkerPop and RDF SPARQL.

Repository relating to the CVPR21 paper TimeLens: Event-based Video Frame Interpolation

structured-generative-modeling

Code for Understanding Pooling in Graph Neural Networks

Mercury: easily convert Python notebook to web app and share with others

An University Project of Quera Web Crawling.

One-line your code easily but still with the fun of doing so!

U-2-Net: U Square Net - Modified for paired image training of style transfer

Pytorch code for "DPFM: Deep Partial Functional Maps" - 3DV 2021 (Oral)

My tensorflow implementation of "A neural conversational model", a Deep learning based chatbot

Attention-based CNN-LSTM and XGBoost hybrid model for stock prediction

Real-time ground filtering algorithm of cloud points acquired using Terrestrial Laser Scanner (TLS)

Code for Domain Adaptive Video Segmentation via Temporal Consistency Regularization in ICCV 2021

Code for the published paper : Learning to recognize rare traffic sign

C3D is a modified version of BVLC caffe to support 3D ConvNets.