I really like the DeBERTa-v3 models and the monolingual models work very well for me. Weirdly enough, the multilingual model uploaded on the huggingface hub does not seem to work. I have a code for training multilingual models on XNLI, and the training normally works well (e.g. no issue with microsoft/Multilingual-MiniLM-L12-H384), but when I apply the exact same code to mDeBERTa, the model does not seem to learn anything. I don't get an error message, but the training results look like this:

I've double checked by running the exact same code on multilingual-minilm and the training works, which makes me think that it's not an issue in the code (wrongly formatting the input data or something like that), but something went wrong when uploading mDeBERTa to the huggingface hub? Accuracy of exactly random 0.3333, 0 training loss at epoch 2 and NaN validation loss maybe indicates that the data is running through the model, but some parameters are not updating or something like that?

My environment is google colab; Transformers==4.12.5

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base")

Gives me an error ValueError: This tokenizer cannot be instantiated. Please make sure you have sentencepiece installed in order to use this tokenizer. But sentencepiece is already installed

Also tried

!pip install deberta
from DeBERTa import deberta
vocab_path, vocab_type = deberta.load_vocab(pretrained_id='base-v3')
tokenizer = deberta.tokenizers[vocab_type](vocab_path)

this gives me TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

Please help, how can I use the tokenizer for deberta-base-v3?

Hello,

It seems like some of the weights were renamed/shaped in the V2 model releases and I couldn't quite figure out how to map them to the old structure

# it seemed like 
pos_q_proj => query_proj
v_bias => value_proj

but I couldn't match

deberta.encoder.layer.44.attention.self.key_proj.weight', 'deberta.encoder.layer.44.attention.self.key_proj.bias
=>
deberta.encoder.layer.44.attention.self.q_bias', 'deberta.encoder.layer.44.attention.self.value_proj', 'deberta.encoder.layer.44.attention.self.in_proj.weight', 'deberta.encoder.layer.44.attention.self.pos_proj.weight

That was for huggingface, but I couldn't figure it out in this repo either.

Could someone upload the v2 model file?

This error occurs when running the setup_glue_data() function inside any Glue Task script, like qqp_large.sh. The error resides on the original script download_glue_data.py, probably the token is no longer valid, but this directly affects this repo making it not possible to reproduce the results with the Glue Tasks.

Error log:

Downloading and extracting QQP...
Traceback (most recent call last):
  File "<stdin>", line 172, in <module>
  File "<stdin>", line 168, in main
  File "<stdin>", line 57, in download_and_extract
  File "/usr/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Hello,

I would like to fine-tune mDEBERTaV3 for the genre classification task, and compare it to XML-RoBERTa and some other similar models, but the training gives very low results (macro, micro f1: 0.003, 0.035; high running loss: 3.0456) and the confusion matrix shows that the model predicts one class to all instances (different class in different runs). Training other models (XML-RoBERTa, SloBERTa, BERTić etc.) with the same setting (only the model type and model name are changed for each model, otherwise the code and dataset is the same) works without any problems.

Here are the hyperparameters:

from simpletransformers.classification import ClassificationModel

model_args ={"overwrite_output_dir": True,
             "num_train_epochs": 90,
             "labels_list": LABELS,
             "learning_rate": 1e-5,
             "train_batch_size": 32,
             "no_cache": True,
             "no_save": True,
             "max_seq_length": 300,
             "save_steps": -1
             }

debertav3_model = ClassificationModel(
        "debertav2", "microsoft/mdeberta-v3-base",
        num_labels=21,
        use_cuda=True,
        args=model_args
    )

The training is performed without being stopped by an error, but there occur some warning messages that might have something to do with the low performance:

1. When loading the pre-trained model:

Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

This makes me think that it might be the problem with the model type, but changing it to "deberta-v2", "debertav3", or "deberta" results in an error.

2. When training:

/opt/conda/lib/python3.7/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:1313: UserWarning: This overload of nonzero is deprecated:
	nonzero()
Consider using one of the following signatures instead:
	nonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)
  label_index = (labels >= 0).nonzero()
/opt/conda/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:123: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)

I'm working on Kaggle, and using the following versions: pytorch>=1.6, cudatoolkit=11.0, simpletransformers==0.63.3, torch==1.6.0+cu101

Thank you very much in advance for your help!

Hello ! Thank you for sharing a great piece of work.

I was wondering whether the MLM pre-training codeis for training DeBERTa v3 or v2 ? (or v1)

Regards

Thanks for sharing the repo. However, I could not access the per-trained base and large models from the below paths.

https://github.com/microsoft/DeBERTa/releases/download/v0.1/base.zip https://github.com/microsoft/DeBERTa/releases/download/v0.1/large.zip

On HF, deberta-v3-large is 800mb: https://huggingface.co/microsoft/deberta-v3-large

But after even a few steps of MLM training the saved model is 1.6gb: https://colab.research.google.com/drive/1PG4PKYnye_F1We2i7VccQ4nYn_XTHhKP?usp=sharing

This seems true of many other finetuned versions of DeBERTaV3 on HF (for both base and large size). It also doesn't seem specific to MLM: https://huggingface.co/navteca/nli-deberta-v3-large https://huggingface.co/cross-encoder/nli-deberta-v3-base/tree/main

Any idea why this is -- is it something to do with V3 itself? And does anyone know if the model size can be reduced again after traning?

Thanks!

The results of debertav3 small on the mnli validation set, in the paper, reported as 88.2/87.9, look different from those reported in open source: https://huggingface.co/mrm8488/deberta-v3-small-finetuned-mnli (reported as 87.46)

Thank you very, very much for DeBERTa! I'm using it as a 0-shot relation extractor, and it's works extremely well. I was wondering if you're planning to release the V3 models fine-tuned on MNLI like previous versions on Hugging Face. Thank you again!

I do: from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v2-xxlarge") model = AutoModel.from_pretrained("microsoft/deberta-v2-xxlarge")

But always the same error occurs: config_class = CONFIG_MAPPING[config_dict["model_type"]] KeyError: 'deberta-v2'

Appreciate your help!

Hi,

Awesome repo! Just started looking into DeBERTa-based models and as I was reading through the README I noticed a few typos that could be fixed. Let me know what you think.

Hope this help :)

https://huggingface.co/microsoft/deberta-v2-xlarge/tree/main

If I run : tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-v2-xlarge')

get bug: ValueError: Couldn't instantiate the backend tokenizer from one of: (1) a tokenizers library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

I am trying to use deberta in google colab, getting this:

AssertionError Traceback (most recent call last) in ----> 1 m = deberta.DeBERTa(pre_trained="large")

2 frames /usr/local/lib/python3.7/dist-packages/DeBERTa/deberta/deberta.py in key_match(key, s) 141 def key_match(key, s): 142 c = [k for k in s if key in k] --> 143 assert len(c)==1, c 144 return c[0] 145 current = self.state_dict()

AssertionError: []

Any ideas?