BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Codes for our paper "SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge" (EMNLP 2020)

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

Code for paper Decoupled Dynamic Spatial-Temporal Graph Neural Network for Traffic Forecasting

[CVPR 2022 Oral] Rethinking Minimal Sufficient Representation in Contrastive Learning

competitions-v2

Corruption Invariant Learning for Re-identification

The repo of Feedback Networks, CVPR17

Proximal Backpropagation - a neural network training algorithm that takes implicit instead of explicit gradient steps

dyld_shared_cache processing / Single-Image loading for BinaryNinja

Few-shot Neural Architecture Search

HNECV: Heterogeneous Network Embedding via Cloud model and Variational inference

DPT: Deformable Patch-based Transformer for Visual Recognition (ACM MM2021)

Codes for our IJCAI21 paper: Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization

The most simple and minimalistic navigation dashboard.

基于Paddlepaddle复现yolov5，支持PaddleDetection接口

Official Pytorch implementation of MixMo framework

BlockUnexpectedPackets - Preventing BungeeCord CPU overload due to Layer 7 DDoS attacks by scanning BungeeCord's logs

K-Nearest Neighbor in Pytorch

As a part of the HAKE project, includes the reproduced SOTA models and the corresponding HAKE-enhanced versions (CVPR2020).

This repository compare a selfie with images from identity documents and response if the selfie match.