Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

Data and analysis code for an MS on SK VOC genomes phenotyping/neutralisation assays

An efficient toolkit for Face Stylization based on the paper "AgileGAN: Stylizing Portraits by Inversion-Consistent Transfer Learning"

Official code for paper "Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight"

SweiNet is an uncertainty-quantifying shear wave speed (SWS) estimator for ultrasound shear wave elasticity (SWE) imaging.

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

Code for You Only Cut Once: Boosting Data Augmentation with a Single Cut

Vit-ImageClassification - Pytorch ViT for Image classification on the CIFAR10 dataset

ktrain is a Python library that makes deep learning and AI more accessible and easier to apply

Minimisation of a negative log likelihood fit to extract the lifetime of the D^0 meson (MNLL2ELDM)

The official pytorch implemention of the CVPR paper "Temporal Modulation Network for Controllable Space-Time Video Super-Resolution".

The Self-Supervised Learner can be used to train a classifier with fewer labeled examples needed using self-supervised learning.

Awesome Remote Sensing Toolkit based on PaddlePaddle.

Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Experiments and examples converting Transformers to ONNX

Research on Tabular Deep Learning (Python package & papers)

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Example-custom-ml-block-keras - Custom Keras ML block example for Edge Impulse

This repository contains the source code for the paper Tutorial on amortized optimization for learning to optimize over continuous domains by Brandon Amos

FLAVR is a fast, flow-free frame interpolation method capable of single shot multi-frame prediction

PyTorch implementation for NED. It can be used to manipulate the facial emotions of actors in videos based on emotion labels or reference styles.