Hi there, I've run a guide code and it doesn't work. I'm getting an error on the following line, decoder_sess, encoder_sess, tokenizer = get_encoder_decoder_tokenizer()

text is a text from Wikipedia about cars.

onnxt5==0.1.4 protobuf==3.6.0 python==3.7

<extra_id_0> the company<extra_id_1> the company<extra_id_2>.<extra_id_3>.<extra_id_4>.<extra_id_5>.<extra_id_6>. <extra_id_7>.

Do I need some postprocessing? Or it is an issue?

Hello , i can't run the first exemple ,

from onnxt5 import GenerativeT5
from onnxt5.api import get_encoder_decoder_tokenizer

decoder_sess, encoder_sess, tokenizer = get_encoder_decoder_tokenizer()
generative_t5 = GenerativeT5(encoder_sess, decoder_sess, tokenizer, onnx=True)
prompt = 'translate English to French: I was a victim of a series of accidents.'

output_text, output_logits = generative_t5(prompt, max_length=100, temperature=0.)
 # output_text: "J'ai été victime d'une série d'accidents." 

the model begin calculation but before End, i have this error :

TypeError                                 Traceback (most recent call last)
<ipython-input-1-257f12b63043> in <module>
      5 prompt = 'translate English to French: I was a victim of a series of accidents.'
      6 
----> 7 output_text, output_logits = generative_t5(prompt, max_length=16, temperature=0.)
      8 # output_text: "J'ai été victime d'une série d'accidents."

~\Anaconda3\envs\onnxt5\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~\Anaconda3\envs\onnxt5\lib\site-packages\onnxt5\models.py in forward(self, prompt, max_length, temperature, repetition_penalty, top_k, top_p, max_context_length)
    145                 new_tokens.append(next_token)
    146 
--> 147             return self.tokenizer.decode(new_tokens), new_logits

~\Anaconda3\envs\onnxt5\lib\site-packages\transformers\tokenization_utils_base.py in decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
   3000             skip_special_tokens=skip_special_tokens,
   3001             clean_up_tokenization_spaces=clean_up_tokenization_spaces,
-> 3002             **kwargs,
   3003         )
   3004 

~\Anaconda3\envs\onnxt5\lib\site-packages\transformers\tokenization_utils.py in _decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, spaces_between_special_tokens)
    730         spaces_between_special_tokens: bool = True,
    731     ) -> str:
--> 732         filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)
    733 
    734         # To avoid mixing byte-level and unicode for byte-level BPT

~\Anaconda3\envs\onnxt5\lib\site-packages\transformers\tokenization_utils.py in convert_ids_to_tokens(self, ids, skip_special_tokens)
    708         tokens = []
    709         for index in ids:
--> 710             index = int(index)
    711             if skip_special_tokens and index in self.all_special_ids:
    712                 continue

TypeError: int() argument must be a string, a bytes-like object or a number, not 'list

`

and i have no idea how to find solution , if you have any solution !? thx !

@abelriboulot , @Ki6an , @brymck .
I have finetuned t5 model for paraphrasing task like this: Paraphrase with t5

I want to reduce inference time, so I exported finetuned t5 model using onnxt5, here I get time taken more in case where I use onnx model on gpu than pytorch model on gpu.

gpu: time taken = 0.2357314471155405 time taken = 0.24958523781970143 time taken = 0.20342689706012607 time taken = 0.5490081580355763 time taken = 0.10756197292357683

onnxt5-gpu time taken = 0.5277913622558117 time taken = 0.6335883080027997 time taken = 0.6975196991115808 time taken = 1.9159171842038631 time taken = 0.7938353712670505

Did I make mistake in exporting/loading model ? gpu code onnxt5-gpu code

This adds a progress bar using tqdm.

The files this library downloads are about 500 MB in size, so I'd like to have some feedback on what's happening. Originally I wasn't clear what was the cause of the delay when running get_encoder_decoder_tokenizer.

This adds a progress bar using tqdm.

The files this library downloads are about 500 MB in size, so I'd like to have some feedback on what's happening. Originally I wasn't clear what was the cause of the delay when running get_encoder_decoder_tokenizer.

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

Thanks for the repo!

I was having an error message come up when running the code after my initial install.

Small code example:

import os

import torch
from onnxt5 import GenerativeT5
from onnxt5.api import get_sess
from transformers import AutoTokenizer

model_dir = <path-to-tokenizer-and-onnx-files>
model_name = <name-of-model>

tokenizer = AutoTokenizer.from_pretrained(
    model_dir,
)

decoder_sess, encoder_sess = get_sess(
    os.path.join(model_dir, model_name)
)

model = GenerativeT5(
    encoder_sess,
    decoder_sess,
    tokenizer,
    onnx=True,
    cuda=torch.cuda.is_available(),
)

sentences = [
    "I has good grammar.",
    "I have bettr grammur."
]

corrected_sentences = [
    model(f"grammar: {sentence}",
          max_length=512,
          temperature=1,
          )[0]
    for sentence in sentences
]

The error

Traceback (most recent call last):
  File "/Users/jamiebrandon/Code/inferentia-test/onnx_example/compiled-t5-base-grammar-correction/code/inference.py", line 133, in <module>
    main()
  File "/Users/jamiebrandon/Code/inferentia-test/onnx_example/compiled-t5-base-grammar-correction/code/inference.py", line 125, in main
    prediction_output = predict_fn(input_data=input_tokens,
  File "/Users/jamiebrandon/Code/inferentia-test/onnx_example/compiled-t5-base-grammar-correction/code/inference.py", line 95, in predict_fn
    corrected_sentences = [model(f"grammar: {sentence}",
  File "/Users/jamiebrandon/Code/inferentia-test/onnx_example/compiled-t5-base-grammar-correction/code/inference.py", line 95, in <listcomp>
    corrected_sentences = [model(f"grammar: {sentence}",
  File "/Users/jamiebrandon/Code/inferentia-test/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/jamiebrandon/Code/inferentia-test/onnx_example/compiled-t5-base-grammar-correction/onnxt5/onnxt5/models.py", line 154, in forward
    return self.tokenizer.decode(new_tokens), new_logits
  File "/Users/jamiebrandon/Code/inferentia-test/venv/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 3367, in decode
    return self._decode(
  File "/Users/jamiebrandon/Code/inferentia-test/venv/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 548, in _decode
    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
TypeError: 'float' object cannot be interpreted as an integer

It seems the tensor for new tokens is of type float instead of long. Adding dtype=torch.long to the instantiation of the tensor resolved my issue, so I thought I'd share.

86%|████████▌ | 18/21 [00:00<00:00, 44.29it/s]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-f543e3365977> in <module>()
     27 # Generating text
     28 generative_t5 = GenerativeT5(encoder_sess, decoder_sess, tokenizer, onnx=True)
---> 29 generative_t5('translate English to French: I was a victim of a series of accidents.', 21, temperature=0.)[0]

3 frames
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_fast.py in _decode(self, token_ids, skip_special_tokens, clean_up_tokenization_spaces, **kwargs)
    505         if isinstance(token_ids, int):
    506             token_ids = [token_ids]
--> 507         text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
    508 
    509         if clean_up_tokenization_spaces:

TypeError: 'float' object cannot be interpreted as an integer

Any possible version conflicts that you know of?

How to suppress output? Setting verbosity logging level does nothing 5%|█████████▊ | 16/300 [00:01<00:18, 15.65it/s]

Recently, I use the chinese function of multilingual-t5 model to accomplish the Chinese NLG tasks. However, the inference speed might be slow, could this model be used for multilingual-t5? How can I do?