Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together

Last update: Nov 07, 2022

Related tags

Text Data & NLP SpeechMix

Overview

SpeechMix

Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together.

Introduction

For the same input:

from datasets import load_dataset
import soundfile as sf


# define function to read in sound file
def map_to_array(batch):
    speech, _ = sf.read(batch["file"])
    batch["speech"] = speech
    return batch


# load dummy dataset and read soundfiles
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_to_array)

transcript = ds['text'][0]
speech = ds["speech"][0]

Speech encoder NLP decoder

model = SpeechMixED("facebook/wav2vec2-base-960h", "facebook/bart-large")

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Speech encoder NLP decoder only fine-tune on cross attention/projection/decoder embedding

model = SpeechMixED("facebook/wav2vec2-base-960h", "facebook/bart-large", ftl=True)

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Speech encoder NLP encoder decoder

model = SpeechMixEED("facebook/wav2vec2-base-960h", "facebook/bart-large")

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Speech encoder NLP encoder decoder only fine-tune on layer norm and attention

model = SpeechMixEED("facebook/wav2vec2-base-960h", "facebook/bart-large", lna=True)

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Speech encoder NLP encoder decoder only fine-tune on speech encoder

model = SpeechMixEED("facebook/wav2vec2-base-960h", "facebook/bart-large", fne=True)

transcript_tensor = model.tokenizer(transcript, return_tensors="pt").input_ids
speech_tensor = model.processor(speech, return_tensors="pt").input_values

model(speech_tensor, transcript_tensor)

Installation

pip install

pip install speechmix

Build from source

git clone and cd into this project.

pip install -e .

Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together

Related tags

Overview

SpeechMix

Introduction

Speech encoder NLP decoder

Speech encoder NLP decoder only fine-tune on cross attention/projection/decoder embedding

Speech encoder NLP encoder decoder

Speech encoder NLP encoder decoder only fine-tune on layer norm and attention

Speech encoder NLP encoder decoder only fine-tune on speech encoder

Installation

pip install

Build from source

Owner

Eric Lam

SAINT PyTorch implementation

skweak: A software toolkit for weak supervision applied to NLP tasks

HiFi DeepVariant + WhatsHap workflowHiFi DeepVariant + WhatsHap workflow

Artificial Conversational Entity for queries in Eulogio "Amang" Rodriguez Institute of Science and Technology (EARIST)

Repository for Project Insight: NLP as a Service

BPEmb is a collection of pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE) and trained on Wikipedia.

A PyTorch implementation of VIOLET

A framework for evaluating Knowledge Graph Embedding Models in a fine-grained manner.

Deeply Supervised, Layer-wise Prediction-aware (DSLP) Transformer for Non-autoregressive Neural Machine Translation

AI and Machine Learning workflows on Anthos Bare Metal.

GSoC'2021 | TensorFlow implementation of Wav2Vec2

Finally, some decent sample sentences

CCF BDCI BERT系统调优赛题baseline（Pytorch版本）

JaQuAD: Japanese Question Answering Dataset

Sentiment-Analysis and EDA on the IMDB Movie Review Dataset

[ICLR'19] Trellis Networks for Sequence Modeling

wxPython app for converting encodings, modifying and fixing SRT files

This repository contains the official release of the model "BanglaBERT" and associated downstream finetuning code and datasets introduced in the paper titled "BanglaBERT: Combating Embedding Barrier in Multilingual Models for Low-Resource Language Understanding".

Universal Adversarial Triggers for Attacking and Analyzing NLP (EMNLP 2019)

DELTA is a deep learning based natural language and speech processing platform.