Meta Language-Specific Layers in Multilingual Language Models

This repo contains the source codes for our paper

On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Zirui Wang, Zachary C. Lipton, Yulia Tsvetkov

EMNLP 2020

Introduction

This repo contains code to train multilingual language models (XLM) that (1) contain language-specific layers, and (2) meta-learn these layers through gradient of gradient.

Language-specific layers are served as meta parameters, optimized using an iterative procedure. The goal is to remedy negative transfer in multilingual models through a meta training objective. Please see our paper for details.

Dependencies

Python 3
XLM
NumPy
PyTorch

Usage

The code is based on the official implementation of XLM. This repo only contains files that we modified from the original codebase. To train a model, please merge code with the source code of XLM, and then follow the standard preprocessing and training instructions there.

Meta Language-Specific Layers in Multilingual Language Models

Related tags

Overview

Meta Language-Specific Layers in Multilingual Language Models

Introduction

Dependencies

Usage

Owner

Zirui Wang

Large-Scale Unsupervised Object Discovery

[MedIA2021]MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning

Unsupervised Image Generation with Infinite Generative Adversarial Networks

MT3: Multi-Task Multitrack Music Transcription

Convert weight file.pth to weight file.blob

Technical Analysis library in pandas for backtesting algotrading and quantitative analysis

This repository contains implementations and illustrative code to accompany DeepMind publications

Simply enable or disable your Nvidia dGPU

LoFTR:Detector-Free Local Feature Matching with Transformers CVPR 2021

A library for preparing, training, and evaluating scalable deep learning hybrid recommender systems using PyTorch.

Code for BMVC2021 paper "Boundary Guided Context Aggregation for Semantic Segmentation"

An expansion for RDKit to read all types of files in one line

Pseudo-Visual Speech Denoising

Code for "Intra-hour Photovoltaic Generation Forecasting based on Multi-source Data and Deep Learning Methods."

Pytorch code for our paper Beyond ImageNet Attack: Towards Crafting Adversarial Examples for Black-box Domains)

Label Hallucination for Few-Shot Classification

SemiNAS: Semi-Supervised Neural Architecture Search

A novel framework to automatically learn high-quality scanning of non-planar, complex anisotropic appearance.

Repo 4 basic seminar §How to make human machine readable"

Real time Human Detection Counting