Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

MolRep: A Deep Representation Learning Library for Molecular Property Prediction

The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

YOLOv5 Series Multi-backbone, Pruning and quantization Compression Tool Box.

Python scripts for performing lane detection using the LSTR model in ONNX

DCA - Official Python implementation of Delaunay Component Analysis algorithm

Second-Order Neural ODE Optimizer, NeurIPS 2021 spotlight

CS583: Deep Learning

Fully Convolutional Refined Auto Encoding Generative Adversarial Networks for 3D Multi Object Scenes

某学校选课系统GIF验证码数据集 + Baseline模型 + 上下游相关工具

Solutions and questions for AoC2021. Merry christmas!

1st place solution to the Satellite Image Change Detection Challenge hosted by SenseTime

Pytorch implementation of "A simple neural network module for relational reasoning" (Relational Networks)

Pre-Training 3D Point Cloud Transformers with Masked Point Modeling

Neighborhood Reconstructing Autoencoders

PyTorch code for JEREX: Joint Entity-Level Relation Extractor

Fight Recognition from Still Images in the Wild @ WACVW2022, Real-world Surveillance Workshop

Official implementation of Long-Short Transformer in PyTorch.

Cossim - Sharpened Cosine Distance implementation in PyTorch

Project code for weakly supervised 3D object detectors using wide-baseline multi-view traffic camera data: WIBAM.

AMTML-KD: Adaptive Multi-teacher Multi-level Knowledge Distillation