Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

Classification of EEG data using Deep Learning

Voila - Voilà turns Jupyter notebooks into standalone web applications

A Pytorch loader for MVTecAD dataset.

Development of IP code based on VIPs and AADM

Denoising Normalizing Flow

PyTorch implementation for "HyperSPNs: Compact and Expressive Probabilistic Circuits", NeurIPS 2021

Extension to fastai for volumetric medical data

Self-Supervised Learning with Kernel Dependence Maximization

Locally cache assets that are normally streamed in POPULATION: ONE

yolox_backbone is a deep-learning library and is a collection of YOLOX Backbone models.

A simple library that implements CLIP guided loss in PyTorch.

The audio-video synchronization of MKV Container Format is exploited to achieve data hiding

DANet for Tabular data classification/ regression.

A object detecting neural network powered by the yolo architecture and leveraging the PyTorch framework and associated libraries.

Hippocampal segmentation using the UNet network for each axis

Script for getting information in discord

Out-of-distribution detection using the pNML regret. NeurIPS2021

Impelmentation for paper Feature Generation and Hypothesis Verification for Reliable Face Anti-Spoofing

Generate Contextual Directory Wordlist For Target Org

Official implementation of CATs: Cost Aggregation Transformers for Visual Correspondence NeurIPS'21