BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

GNPy: Optical Route Planning and DWDM Network Optimization

Learning Chinese Character style with conditional GAN

Language models are open knowledge graphs ( non official implementation )

a practicable framework used in Deep Learning. So far UDL only provide DCFNet implementation for the ICCV paper (Dynamic Cross Feature Fusion for Remote Sensing Pansharpening)

Stochastic Normalizing Flows

Efficient training of deep recommenders on cloud.

[ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Face Transformer for Recognition

Deep Image Search is an AI-based image search engine that includes deep transfor learning features Extraction and tree-based vectorized search.

Source code for "Progressive Transformers for End-to-End Sign Language Production" (ECCV 2020)

The code repository for "RCNet: Reverse Feature Pyramid and Cross-scale Shift Network for Object Detection" (ACM MM'21)

This is official implementaion of paper "Token Shift Transformer for Video Classification".

🐸STT integration examples

DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models

Towards End-to-end Video-based Eye Tracking

Implementation of "RaScaNet: Learning Tiny Models by Raster-Scanning Image" from CVPR 2021.

Official Implementation of DE-CondDETR and DELA-CondDETR in "Towards Data-Efficient Detection Transformers"

Add-on for importing and auto setup of character creator 3 character exports.

Digitalizing-Prescription-Image - PIRDS - Prescription Image Recognition and Digitalizing System is a OCR make with Tensorflow