Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

Event-forecasting - Event Forecasting Algorithms With Python

FinEAS: Financial Embedding Analysis of Sentiment 📈

Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"

2D Human Pose estimation using transformers. Implementation in Pytorch

(Py)TOD: Tensor-based Outlier Detection, A General GPU-Accelerated Framework

Official repository of ICCV21 paper "Viewpoint Invariant Dense Matching for Visual Geolocalization"

Pytorch implementation of the popular Improv RNN model originally proposed by the Magenta team.

CVPR 2020 oral paper: Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax.

HiFT: Hierarchical Feature Transformer for Aerial Tracking (ICCV2021)

Which Style Makes Me Attractive? Interpretable Control Discovery and Counterfactual Explanation on StyleGAN

Implementation of light baking system for ray tracing based on Activision's UberBake

PyTorch implementation for Graph Contrastive Learning with Augmentations

PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021.

Empirical Study of Transformers for Source Code & A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code

A script depending on VASP output for calculating Fermi-Softness.

RITA is a family of autoregressive protein models, developed by LightOn in collaboration with the OATML group at Oxford and the Debora Marks Lab at Harvard.

Optimizing synthesizer parameters using gradient approximation

IDA file loader for UF2, created for the DEFCON 29 hardware badge

[NeurIPS 2019] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

NumPy로 구현한 딥러닝 라이브러리입니다. (자동 미분 지원)