Hi guys,

Hope you are all well !

I was wondering if adaptnlp can handle multi-label classification with 1560 labels.

More precisely, I would like to apply it to paperswithcode dataset where labels are called tasks.

Refs:

• Repository
• Dataset

Thanks for any insights or inputs on that.

Cheers, X

Describe the bug Your demo Colab Notebook "Custom Fine-Tuning and Training with Transformer Models" doesn't work and generates the following error:

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

I'm experiencing a slowdown in NER performance using EasyTokenTagger and 'ner-ontonotes' after updating to release 0.20. Has there been any underlying changes to how the tagger object works?

Specifically, I am dealing with a very large chunk of text. Prior to this release, the NER tagging took around 15 seconds for this particular text. Now, it's taking 15+ minutes the first time but subsequent calls on that text are very quick. Is there some sort of caching or indexing that's being done now? I'd imagine this could create a lot of overhead for large chunks of text.

Describe the bug It happens when I want to learning_rate = finetuner.find_learning_rate(**learning_rate_finder_configs) in the tutorial. I have a big dataset with 200k rows and each of them has a text with around 200 words.

In your code when you instantiate the TextDataset, the line tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text)) takes an eternity for a text of 20 million words. Do you think it can be achieved in the better/faster way like by keeping the rows like they are ?

For the record: Time for 100 characters: 0.0003399848937988281s Time for 1000 characters: 0.00124359130859375s Time for 10 000 characters: 0.012135982513427734s Time for 100 000 characters: 0.2131056785583496s Time for 1 000 000 characters: 8.782422542572021s Time for 10 000 000 characters: 734.5397665500641s

Can't reach the end of the full TextDataset (109 610 928 characters).

To Reproduce Tutorial with a big dataset

Describe the bug Trying to freeze a LMFinetuner based on Camembert weights and get:

---

AttributeError Traceback (most recent call last) in 6 } 7 finetuner = LMFineTuner(**ft_configs) ----> 8 finetuner.freeze()

~/anaconda3/envs/pe_adaptnlp/lib/python3.8/site-packages/adaptnlp/transformers/finetuning.py in freeze(self) 1630 """Freeze last classification layer group only 1631 """ -> 1632 layers_len = len(list(self.model.cls.parameters())) 1633 self.freeze_to(-layers_len) 1634

~/anaconda3/envs/pe_adaptnlp/lib/python3.8/site-packages/torch/nn/modules/module.py in getattr(self, name) 573 if name in modules: 574 return modules[name] --> 575 raise AttributeError("'{}' object has no attribute '{}'".format( 576 type(self).name, name)) 577

AttributeError: 'CamembertForMaskedLM' object has no attribute 'cls'

To Reproduce

from adaptnlp import LMFineTuner
train_file = "path/to/train" 
valid_file = "path/to/valid"
ft_configs = {
              "train_data_file": train_file,
              "eval_data_file": valid_file,
              "model_type": "camembert",
              "model_name_or_path": "camembert-base",
             }
finetuner = LMFineTuner(**ft_configs)
finetuner.freeze()

Expected behavior No error

Desktop (please complete the following information):

• OS: Amazon Linux
• Browser Chrome

Cannot use pool option to generate embeddings (instead of the default rnn).

A snippet for the problem:

embedding_type='albert-xxlarge-v2'
embedding_methods=["pool"]
doc_embeddings = EasyDocumentEmbeddings(embedding_type, methods = embedding_methods)

This is the error I get:

  File "env/lib/python3.7/site-packages/adaptnlp/training.py", line 91, in __init__
   self._initial_setup(self.label_dict, **kwargs)
 File "env/lib/python3.7/site-packages/adaptnlp/training.py", line 97, in _initial_setup
   document_embeddings: DocumentRNNEmbeddings = self.encoder.rnn_embeddings
AttributeError: 'EasyDocumentEmbeddings' object has no attribute 'rnn_embeddings'

Expected behavior would be to successfully obtain an easy document embeddings object with no errors

Running on debian buster, python3.7

If someone could give me a fix or a workaround or if I'm using this incorrectly, then please let me know

Hi! I tried to follow the tutorial for training custom sequence classifier: https://novetta.github.io/adaptnlp/tutorial/training-sequence-classification.html The last step returns empty sentences while expected labels: sentences = classifier.tag_text(example_text, model_name_or_path=OUTPUT_DIR)

To Reproduce the behavior:

from adaptnlp import EasySequenceClassifier
from flair.data import Sentence

OUTPUT_DIR = "…/best-model.pt"    # my custom model
classifier = EasySequenceClassifier()

ex_text = "This is a good text example"
example_text=[Sentence(ex_text)]

sentences = classifier.tag_text(text=example_text, model_name_or_path=OUTPUT_DIR, mini_batch_size=1)
print("Label output:\n")
print(sentences)

Returns

2020-12-28 17:44:31,111 loading file .../best-model.pt
Label output:

None

Surprisingly labels got added to example_text print(example_text) Returns [Sentence: " This is a good text example " [− Tokens: 17 − Sentence-Labels: {'label': [0 (0.8812)]}]]

Proposed explanation/ contribution: I think I know the reason for unexpected behavior and will be happy to help. classifier.tag_text creates FlairSequenceClassifier classifier. FlairSequenceClassifier initiates flair.models.TextClassifier classifier and uses TextClassifier predict method within its own predict method. But flair.models.TextClassifier predict method returns None because the labels are directly added to the sentences. I can re-write FlairSequenceClassifier predict method to return Sentences with labels instead of None.

Sequence classification over REST API using any model except for en-sentiment fails with:

File "/usr/local/lib/python3.6/dist-packages/starlette/routing.py", line 41, in app response = await func(request) File "/usr/local/lib/python3.6/dist-packages/fastapi/routing.py", line 197, in app dependant=dependant, values=values, is_coroutine=is_coroutine File "/usr/local/lib/python3.6/dist-packages/fastapi/routing.py", line 147, in run_endpoint_function return await dependant.call(**values) File "./app/main.py", line 87, in sequence_classifier text=text, mini_batch_size=1, model_name_or_path=_SEQUENCE_CLASSIFICATION_MODEL File "/adaptnlp/adaptnlp/sequence_classification.py", line 285, in tag_text return classifier.predict(text=text, mini_batch_size=mini_batch_size, **kwargs,) File "/adaptnlp/adaptnlp/sequence_classification.py", line 140, in predict text_sent.add_label(label) TypeError: add_label() missing 1 required positional argument: 'value'

Reproducable with: docker run -itp 5000:5000 -e TOKEN_TAGGING_MODE='ner' \ -e TOKEN_TAGGING_MODEL='ner-ontonotes-fast' \ -e SEQUENCE_CLASSIFICATION_MODEL='nlptown/bert-base-multilingual-uncased-sentiment' \ achangnovetta/adaptnlp-rest:latest \ bash

There are a lot of ideas that may be floating for feature implementations, so this thread just provides a mini roadmap and environment to think about adaptnlp's progression.

Ideas can be stated freely in this thread and do not replace feature-request issue posts.

• [x] Tokenizer Start integrating tokenizers all across adaptnlp for speed and performance enhancements for training and inference.
• [x] Summarization Add NLP-task of summarization using document-level encoder based on transformer language models
• [x] GPU Multi-GPU and mixed-precision is prevalent in AdaptNLP, but its implementation can be improved and debugged ~~FastAPI Batch-Serving Improve on the concurrent calls with batch processing from the NLP models (maybe try to make it CPU and GPU agnostic for ease-of-use)~~ ~~Model Downloading Start structuring a way to download and potentially upload pre-trained NLP-task models~~

We probably should have a data API of some form, that ties into https://github.com/Novetta/adaptnlp/issues/128

Ideally it should simply prep a dataset for tokenization of a model, or tokenize the data itself.

For now we cover two inputs:

1. Individual texts
2. CSV

We should support something akin to fastai's get_y, but with decent defaults so that customization is available, but not needed.

Ideally something like:

dset = TaskDataset.from_df(
  df,  # Can be fname or dataframe
  get_x = ColReader('text'),
  get_y = ColReader('label'),
  splitter = RandomSplitter(),
  model = 'bert-base-uncased', # The name/type of downstream model
  task = "ner" # Or use a `Task.NER` namespace class
)

And further:

dset.dataloaders(bs=8, collate_fn=data_collator)

It reads extremely similar to the fastai API, but we do not use the fastai API, as for text doing it like this is a bit easier.

The highest level API would look like so:

dls = TaskDataLoaders.from_df(df, 'text', 'label', model='bert-base-uncased')

We should note the model used, and when integrating it with the tuning API if something is off with the model entered, we make note of that

Describe the bug A clear and concise description of what the bug is. I tried to run the code in the tutorial

from adaptnlp import EasyTokenTagger


## Example Text
example_text = "Novetta's headquarters is located in Mclean, Virginia."

## Load the token tagger module and tag text with the NER model 
tagger = EasyTokenTagger()
sentences = tagger.tag_text(text=example_text, model_name_or_path="ner")

## Output tagged token span results in Flair's Sentence object model
for sentence in sentences:
    for entity in sentence.get_spans("ner"):
        print(entity)

and it gave me the error:

...
  File "/home/rajiv/Documents/dev/python/nltk-trial/adaptnlp.py", line 2, in <module>
    from adaptnlp import EasyTokenTagger
ImportError: cannot import name 'EasyTokenTagger'

Desktop (please complete the following information):

• OS: Ubuntu
• Version: 20.04
• Python: 3.6.9

hi i want to classify texts: classifier = EasySequenceClassifier() hub = HFModelHub() hub.search_model_by_task('text-classification') model = hub.search_model_by_name('nlptown/bert-base', user_uploaded=True)[0]; sentence = classifier.tag_text(text=inputs, model_name_or_path=model, mini_batch_size=1)

Q1: how force to run it on CPU? Q2: now i have GPU but i can't success to run, my errors: ... FileNotFoundError: [Errno 2] No such file or directory: 'nlptown/bert-base-multilingual-uncased-sentiment' During handling of the above exception, another exception occurred: ... RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Training API will use fastai under the hood, and we'll make a few functions to build general datasets.

Tasks and sample datasets to use:

• [x] Language Models:
  • IMDB_SAMPLE text (fastai)
• [x] NER/Token Classification
  • Annotated Corpus for Named Entity Recognition
• [ ] Question/Answering
  • SQuAD dataset (HuggingFace Datasets)
• [x] Sequence Classification
  • IMDB_SAMPLE classification (fastai)
• [ ] Summarization
  • Amazon Fine Food Reviews
• [ ] Translation
  • English to French

Other Information

Task API's should have a simple user interface, IE high-level can only input specific options, while midlevel has access to the full fastai Learner params.

Example mid-level API I'm thinking about:

dls = some_build_data_thing()
tuner = QAFineTuner(dls, 'bert-base-cased')
tuner.tune(
  scheduler = 'fit_flat_cos',
  n_epochs = 3,
  lr = None,
  suggest_method = 'valley', # Triggers if lr is None
  additional_callbacks = []
)

And its high-level:

tuner = QAFineTuner.from_csv(
  question_column_name = "question",
  answer_column_name = "answer",
  model = "bert-base-cased"
)
tuner.tune(...)

We should automatically pull in proper metrics for each task, but users have the option to bring in their own as well and pass it to QAFineTuner (good defaults)

Tuners should also have a func like QAFineTuner.from_csv() to build the dataset in-house

I notices when we run any code snippet, it convert the text to vectors or some similar thing. For example in this code snippet

from adaptnlp import EasyQuestionAnswering 
from pprint import pprint

## Example Query and Context 
query = "What is the meaning of life?"
context = "Machine Learning is the meaning of life."
top_n = 5

## Load the QA module and run inference on results 
qa = EasyQuestionAnswering()
best_answer, best_n_answers = qa.predict_qa(query=query, context=context, n_best_size=top_n, mini_batch_size=1, model_name_or_path="distilbert-base-uncased-distilled-squad")

## Output top answer as well as top 5 answers
print(best_answer)
pprint(best_n_answers)

It convert both query and context to vectors first. What if we have very long context and we have a lot of queries, each time it will convert the context to vector. I think there should be a way to save context vector and re-use it instead of creating again and again.

• [x] HuggingFace raw embeddings over Flair
• [x] Try and integrate Callbacks for text generation and other classes that aren't using it

  Note: Didn't do this for text generation, more complex than its worth

• [x] Use fastrelease (with conda)
• [x] Improve test coverage
• [x] GH CI for testing Mac, Windows, and Linux, similar to how fastai has it setup
• [x] nbdev?
• [x] Windows support
• [x] Use Pipeline for inference

  Note: Pipeline is slower on many tasks that AdaptNLP covers, tests are in place to ensure that this is always true

• [ ] 1.0.0: Unified training framework for at least 4 NLP tasks