TextAttack 🐙 is a Python framework for adversarial attacks, data augmentation, and model training in NLP

Overview

TextAttack 🐙

Generating adversarial examples for NLP models

[TextAttack Documentation on ReadTheDocs]

AboutSetupUsageDesign

Github Runner Covergae Status PyPI version

TextAttack Demo GIF

About

TextAttack is a Python framework for adversarial attacks, data augmentation, and model training in NLP.

If you're looking for information about TextAttack's menagerie of pre-trained models, you might want the TextAttack Model Zoo page.

Slack Channel

For help and realtime updates related to TextAttack, please join the TextAttack Slack!

Why TextAttack?

There are lots of reasons to use TextAttack:

  1. Understand NLP models better by running different adversarial attacks on them and examining the output
  2. Research and develop different NLP adversarial attacks using the TextAttack framework and library of components
  3. Augment your dataset to increase model generalization and robustness downstream
  4. Train NLP models using just a single command (all downloads included!)

Setup

Installation

You should be running Python 3.6+ to use this package. A CUDA-compatible GPU is optional but will greatly improve code speed. TextAttack is available through pip:

pip install textattack

Once TextAttack is installed, you can run it via command-line (textattack ...) or via python module (python -m textattack ...).

Tip: TextAttack downloads files to ~/.cache/textattack/ by default. This includes pretrained models, dataset samples, and the configuration file config.yaml. To change the cache path, set the environment variable TA_CACHE_DIR. (for example: TA_CACHE_DIR=/tmp/ textattack attack ...).

Usage

Help: textattack --help

TextAttack's main features can all be accessed via the textattack command. Two very common commands are textattack attack , and textattack augment . You can see more information about all commands using

textattack --help 

or a specific command using, for example,

textattack attack --help

The examples/ folder includes scripts showing common TextAttack usage for training models, running attacks, and augmenting a CSV file.

The documentation website contains walkthroughs explaining basic usage of TextAttack, including building a custom transformation and a custom constraint..

Running Attacks: textattack attack --help

The easiest way to try out an attack is via the command-line interface, textattack attack.

Tip: If your machine has multiple GPUs, you can distribute the attack across them using the --parallel option. For some attacks, this can really help performance. (If you want to attack Keras models in parallel, please check out examples/attack/attack_keras_parallel.py instead)

Here are some concrete examples:

TextFooler on BERT trained on the MR sentiment classification dataset:

textattack attack --recipe textfooler --model bert-base-uncased-mr --num-examples 100

DeepWordBug on DistilBERT trained on the Quora Question Pairs paraphrase identification dataset:

textattack attack --model distilbert-base-uncased-cola --recipe deepwordbug --num-examples 100

Beam search with beam width 4 and word embedding transformation and untargeted goal function on an LSTM:

textattack attack --model lstm-mr --num-examples 20 \
 --search-method beam-search^beam_width=4 --transformation word-swap-embedding \
 --constraints repeat stopword max-words-perturbed^max_num_words=2 embedding^min_cos_sim=0.8 part-of-speech \
 --goal-function untargeted-classification

Tip: Instead of specifying a dataset and number of examples, you can pass --interactive to attack samples inputted by the user.

Attacks and Papers Implemented ("Attack Recipes"): textattack attack --recipe [recipe_name]

We include attack recipes which implement attacks from the literature. You can list attack recipes using textattack list attack-recipes.

To run an attack recipe: textattack attack --recipe [recipe_name]

TextAttack Overview

Attack Recipe Name Goal Function ConstraintsEnforced Transformation Search Method Main Idea

Attacks on classification tasks, like sentiment classification and entailment:
a2t Untargeted {Classification, Entailment} Percentage of words perturbed, Word embedding distance, DistilBERT sentence encoding cosine similarity, part-of-speech consistency Counter-fitted word embedding swap (or) BERT Masked Token Prediction Greedy-WIR (gradient) from (["Towards Improving Adversarial Training of NLP Models" (Yoo et al., 2021)](https://arxiv.org/abs/2109.00544))
alzantot Untargeted {Classification, Entailment} Percentage of words perturbed, Language Model perplexity, Word embedding distance Counter-fitted word embedding swap Genetic Algorithm from (["Generating Natural Language Adversarial Examples" (Alzantot et al., 2018)](https://arxiv.org/abs/1804.07998))
bae Untargeted Classification USE sentence encoding cosine similarity BERT Masked Token Prediction Greedy-WIR BERT masked language model transformation attack from (["BAE: BERT-based Adversarial Examples for Text Classification" (Garg & Ramakrishnan, 2019)](https://arxiv.org/abs/2004.01970)).
bert-attack Untargeted Classification USE sentence encoding cosine similarity, Maximum number of words perturbed BERT Masked Token Prediction (with subword expansion) Greedy-WIR (["BERT-ATTACK: Adversarial Attack Against BERT Using BERT" (Li et al., 2020)](https://arxiv.org/abs/2004.09984))
checklist {Untargeted, Targeted} Classification checklist distance contract, extend, and substitutes name entities Greedy-WIR Invariance testing implemented in CheckList . (["Beyond Accuracy: Behavioral Testing of NLP models with CheckList" (Ribeiro et al., 2020)](https://arxiv.org/abs/2005.04118))
clare Untargeted {Classification, Entailment} USE sentence encoding cosine similarity RoBERTa Masked Prediction for token swap, insert and merge Greedy ["Contextualized Perturbation for Textual Adversarial Attack" (Li et al., 2020)](https://arxiv.org/abs/2009.07502))
deepwordbug {Untargeted, Targeted} Classification Levenshtein edit distance {Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution} Greedy-WIR Greedy replace-1 scoring and multi-transformation character-swap attack (["Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers" (Gao et al., 2018)](https://arxiv.org/abs/1801.04354)
fast-alzantot Untargeted {Classification, Entailment} Percentage of words perturbed, Language Model perplexity, Word embedding distance Counter-fitted word embedding swap Genetic Algorithm Modified, faster version of the Alzantot et al. genetic algorithm, from (["Certified Robustness to Adversarial Word Substitutions" (Jia et al., 2019)](https://arxiv.org/abs/1909.00986))
hotflip (word swap) Untargeted Classification Word Embedding Cosine Similarity, Part-of-speech match, Number of words perturbed Gradient-Based Word Swap Beam search (["HotFlip: White-Box Adversarial Examples for Text Classification" (Ebrahimi et al., 2017)](https://arxiv.org/abs/1712.06751))
iga Untargeted {Classification, Entailment} Percentage of words perturbed, Word embedding distance Counter-fitted word embedding swap Genetic Algorithm Improved genetic algorithm -based word substitution from (["Natural Language Adversarial Attacks and Defenses in Word Level (Wang et al., 2019)"](https://arxiv.org/abs/1909.06723)
input-reduction Input Reduction Word deletion Greedy-WIR Greedy attack with word importance ranking , Reducing the input while maintaining the prediction through word importance ranking (["Pathologies of Neural Models Make Interpretation Difficult" (Feng et al., 2018)](https://arxiv.org/pdf/1804.07781.pdf))
kuleshov Untargeted Classification Thought vector encoding cosine similarity, Language model similarity probability Counter-fitted word embedding swap Greedy word swap (["Adversarial Examples for Natural Language Classification Problems" (Kuleshov et al., 2018)](https://openreview.net/pdf?id=r1QZ3zbAZ))
pruthi Untargeted Classification Minimum word length, Maximum number of words perturbed {Neighboring Character Swap, Character Deletion, Character Insertion, Keyboard-Based Character Swap} Greedy search simulates common typos (["Combating Adversarial Misspellings with Robust Word Recognition" (Pruthi et al., 2019)](https://arxiv.org/abs/1905.11268)
pso Untargeted Classification HowNet Word Swap Particle Swarm Optimization (["Word-level Textual Adversarial Attacking as Combinatorial Optimization" (Zang et al., 2020)](https://www.aclweb.org/anthology/2020.acl-main.540/))
pwws Untargeted Classification WordNet-based synonym swap Greedy-WIR (saliency) Greedy attack with word importance ranking based on word saliency and synonym swap scores (["Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency" (Ren et al., 2019)](https://www.aclweb.org/anthology/P19-1103/))
textbugger : (black-box) Untargeted Classification USE sentence encoding cosine similarity {Character Insertion, Character Deletion, Neighboring Character Swap, Character Substitution} Greedy-WIR ([(["TextBugger: Generating Adversarial Text Against Real-world Applications" (Li et al., 2018)](https://arxiv.org/abs/1812.05271)).
textfooler Untargeted {Classification, Entailment} Word Embedding Distance, Part-of-speech match, USE sentence encoding cosine similarity Counter-fitted word embedding swap Greedy-WIR Greedy attack with word importance ranking (["Is Bert Really Robust?" (Jin et al., 2019)](https://arxiv.org/abs/1907.11932))

Attacks on sequence-to-sequence models:
morpheus Minimum BLEU Score Inflection Word Swap Greedy search Greedy to replace words with their inflections with the goal of minimizing BLEU score (["It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations"](https://www.aclweb.org/anthology/2020.acl-main.263.pdf)
seq2sick :(black-box) Non-overlapping output Counter-fitted word embedding swap Greedy-WIR Greedy attack with goal of changing every word in the output translation. Currently implemented as black-box with plans to change to white-box as done in paper (["Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples" (Cheng et al., 2018)](https://arxiv.org/abs/1803.01128))

Recipe Usage Examples

Here are some examples of testing attacks from the literature from the command-line:

TextFooler against BERT fine-tuned on SST-2:

textattack attack --model bert-base-uncased-sst2 --recipe textfooler --num-examples 10

seq2sick (black-box) against T5 fine-tuned for English-German translation:

 textattack attack --model t5-en-de --recipe seq2sick --num-examples 100

Augmenting Text: textattack augment

Many of the components of TextAttack are useful for data augmentation. The textattack.Augmenter class uses a transformation and a list of constraints to augment data. We also offer built-in recipes for data augmentation:

  • wordnet augments text by replacing words with WordNet synonyms
  • embedding augments text by replacing words with neighbors in the counter-fitted embedding space, with a constraint to ensure their cosine similarity is at least 0.8
  • charswap augments text by substituting, deleting, inserting, and swapping adjacent characters
  • eda augments text with a combination of word insertions, substitutions and deletions.
  • checklist augments text by contraction/extension and by substituting names, locations, numbers.
  • clare augments text by replacing, inserting, and merging with a pre-trained masked language model.

Augmentation Command-Line Interface

The easiest way to use our data augmentation tools is with textattack augment . textattack augment takes an input CSV file and text column to augment, along with the number of words to change per augmentation and the number of augmentations per input example. It outputs a CSV in the same format with all the augmentation examples corresponding to the proper columns.

For example, given the following as examples.csv:

"text",label
"the rock is destined to be the 21st century's new conan and that he's going to make a splash even greater than arnold schwarzenegger , jean- claud van damme or steven segal.", 1
"the gorgeously elaborate continuation of 'the lord of the rings' trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .", 1
"take care of my cat offers a refreshingly different slice of asian cinema .", 1
"a technically well-made suspenser . . . but its abrupt drop in iq points as it races to the finish line proves simply too discouraging to let slide .", 0
"it's a mystery how the movie could be released in this condition .", 0

The command

textattack augment --input-csv examples.csv --output-csv output.csv  --input-column text --recipe embedding --pct-words-to-swap .1 --transformations-per-example 2 --exclude-original

will augment the text column by altering 10% of each example's words, generating twice as many augmentations as original inputs, and exclude the original inputs from the output CSV. (All of this will be saved to augment.csv by default.)

Tip: Just as running attacks interactively, you can also pass --interactive to augment samples inputted by the user to quickly try out different augmentation recipes!

After augmentation, here are the contents of augment.csv:

text,label
"the rock is destined to be the 21st century's newest conan and that he's gonna to make a splashing even stronger than arnold schwarzenegger , jean- claud van damme or steven segal.",1
"the rock is destined to be the 21tk century's novel conan and that he's going to make a splat even greater than arnold schwarzenegger , jean- claud van damme or stevens segal.",1
the gorgeously elaborate continuation of 'the lord of the rings' trilogy is so huge that a column of expression significant adequately describe co-writer/director pedro jackson's expanded vision of j . rs . r . tolkien's middle-earth .,1
the gorgeously elaborate continuation of 'the lordy of the piercings' trilogy is so huge that a column of mots cannot adequately describe co-novelist/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .,1
take care of my cat offerings a pleasantly several slice of asia cinema .,1
taking care of my cat offers a pleasantly different slice of asiatic kino .,1
a technically good-made suspenser . . . but its abrupt drop in iq points as it races to the finish bloodline proves straightforward too disheartening to let slide .,0
a technically well-made suspenser . . . but its abrupt drop in iq dot as it races to the finish line demonstrates simply too disheartening to leave slide .,0
it's a enigma how the film wo be releases in this condition .,0
it's a enigma how the filmmaking wo be publicized in this condition .,0

The 'embedding' augmentation recipe uses counterfitted embedding nearest-neighbors to augment data.

Augmentation Python Interface

In addition to the command-line interface, you can augment text dynamically by importing the Augmenter in your own code. All Augmenter objects implement augment and augment_many to generate augmentations of a string or a list of strings. Here's an example of how to use the EmbeddingAugmenter in a python script:

>>> from textattack.augmentation import EmbeddingAugmenter
>>> augmenter = EmbeddingAugmenter()
>>> s = 'What I cannot create, I do not understand.'
>>> augmenter.augment(s)
['What I notable create, I do not understand.', 'What I significant create, I do not understand.', 'What I cannot engender, I do not understand.', 'What I cannot creating, I do not understand.', 'What I cannot creations, I do not understand.', 'What I cannot create, I do not comprehend.', 'What I cannot create, I do not fathom.', 'What I cannot create, I do not understanding.', 'What I cannot create, I do not understands.', 'What I cannot create, I do not understood.', 'What I cannot create, I do not realise.']

You can also create your own augmenter from scratch by importing transformations/constraints from textattack.transformations and textattack.constraints. Here's an example that generates augmentations of a string using WordSwapRandomCharacterDeletion:

>>> from textattack.transformations import WordSwapRandomCharacterDeletion
>>> from textattack.transformations import CompositeTransformation
>>> from textattack.augmentation import Augmenter
>>> transformation = CompositeTransformation([WordSwapRandomCharacterDeletion()])
>>> augmenter = Augmenter(transformation=transformation, transformations_per_example=5)
>>> s = 'What I cannot create, I do not understand.'
>>> augmenter.augment(s)
['What I cannot creae, I do not understand.', 'What I cannot creat, I do not understand.', 'What I cannot create, I do not nderstand.', 'What I cannot create, I do nt understand.', 'Wht I cannot create, I do not understand.']

Training Models: textattack train

Our model training code is available via textattack train to help you train LSTMs, CNNs, and transformers models using TextAttack out-of-the-box. Datasets are automatically loaded using the datasets package.

Training Examples

Train our default LSTM for 50 epochs on the Yelp Polarity dataset:

textattack train --model-name-or-path lstm --dataset yelp_polarity  --epochs 50 --learning-rate 1e-5

Fine-Tune bert-base on the CoLA dataset for 5 epochs*:

textattack train --model-name-or-path bert-base-uncased --dataset glue^cola --per-device-train-batch-size 8 --epochs 5

To check datasets: textattack peek-dataset

To take a closer look at a dataset, use textattack peek-dataset. TextAttack will print some cursory statistics about the inputs and outputs from the dataset. For example,

textattack peek-dataset --dataset-from-huggingface snli

will show information about the SNLI dataset from the NLP package.

To list functional components: textattack list

There are lots of pieces in TextAttack, and it can be difficult to keep track of all of them. You can use textattack list to list components, for example, pretrained models (textattack list models) or available search methods (textattack list search-methods).

Design

Models

TextAttack is model-agnostic! You can use TextAttack to analyze any model that outputs IDs, tensors, or strings. To help users, TextAttack includes pre-trained models for different common NLP tasks. This makes it easier for users to get started with TextAttack. It also enables a more fair comparison of attacks from the literature.

Built-in Models and Datasets

TextAttack also comes built-in with models and datasets. Our command-line interface will automatically match the correct dataset to the correct model. We include 82 different (Oct 2020) pre-trained models for each of the nine GLUE tasks, as well as some common datasets for classification, translation, and summarization.

A list of available pretrained models and their validation accuracies is available at textattack/models/README.md. You can also view a full list of provided models & datasets via textattack attack --help.

Here's an example of using one of the built-in models (the SST-2 dataset is automatically loaded):

textattack attack --model roberta-base-sst2 --recipe textfooler --num-examples 10

HuggingFace support: transformers models and datasets datasets

We also provide built-in support for transformers pretrained models and datasets from the datasets package! Here's an example of loading and attacking a pre-trained model and dataset:

textattack attack --model-from-huggingface distilbert-base-uncased-finetuned-sst-2-english --dataset-from-huggingface glue^sst2 --recipe deepwordbug --num-examples 10

You can explore other pre-trained models using the --model-from-huggingface argument, or other datasets by changing --dataset-from-huggingface.

Loading a model or dataset from a file

You can easily try out an attack on a local model or dataset sample. To attack a pre-trained model, create a short file that loads them as variables model and tokenizer. The tokenizer must be able to transform string inputs to lists or tensors of IDs using a method called encode(). The model must take inputs via the __call__ method.

Custom Model from a file

To experiment with a model you've trained, you could create the following file and name it my_model.py:

model = load_your_model_with_custom_code() # replace this line with your model loading code
tokenizer = load_your_tokenizer_with_custom_code() # replace this line with your tokenizer loading code

Then, run an attack with the argument --model-from-file my_model.py. The model and tokenizer will be loaded automatically.

Custom Datasets

Dataset from a file

Loading a dataset from a file is very similar to loading a model from a file. A 'dataset' is any iterable of (input, output) pairs. The following example would load a sentiment classification dataset from file my_dataset.py:

dataset = [('Today was....', 1), ('This movie is...', 0), ...]

You can then run attacks on samples from this dataset by adding the argument --dataset-from-file my_dataset.py.

Dataset via AttackedText class

To allow for word replacement after a sequence has been tokenized, we include an AttackedText object which maintains both a list of tokens and the original text, with punctuation. We use this object in favor of a list of words or just raw text.

Dataset loading via other mechanism, see: here

Attacks and how to design a new attack

We formulate an attack as consisting of four components: a goal function which determines if the attack has succeeded, constraints defining which perturbations are valid, a transformation that generates potential modifications given an input, and a search method which traverses through the search space of possible perturbations. The attack attempts to perturb an input text such that the model output fulfills the goal function (i.e., indicating whether the attack is successful) and the perturbation adheres to the set of constraints (e.g., grammar constraint, semantic similarity constraint). A search method is used to find a sequence of transformations that produce a successful adversarial example.

This modular design unifies adversarial attack methods into one system, enables us to easily assemble attacks from the literature while re-using components that are shared across attacks. We provides clean, readable implementations of 16 adversarial attack recipes from the literature (see above table). For the first time, these attacks can be benchmarked, compared, and analyzed in a standardized setting.

TextAttack is model-agnostic - meaning it can run attacks on models implemented in any deep learning framework. Model objects must be able to take a string (or list of strings) and return an output that can be processed by the goal function. For example, machine translation models take a list of strings as input and produce a list of strings as output. Classification and entailment models return an array of scores. As long as the user's model meets this specification, the model is fit to use with TextAttack.

Goal Functions

A GoalFunction takes as input an AttackedText object, scores it, and determines whether the attack has succeeded, returning a GoalFunctionResult.

Constraints

A Constraint takes as input a current AttackedText, and a list of transformed AttackedTexts. For each transformed option, it returns a boolean representing whether the constraint is met.

Transformations

A Transformation takes as input an AttackedText and returns a list of possible transformed AttackedTexts. For example, a transformation might return all possible synonym replacements.

Search Methods

A SearchMethod takes as input an initial GoalFunctionResult and returns a final GoalFunctionResult The search is given access to the get_transformations function, which takes as input an AttackedText object and outputs a list of possible transformations filtered by meeting all of the attack’s constraints. A search consists of successive calls to get_transformations until the search succeeds (determined using get_goal_results) or is exhausted.

On Benchmarking Attacks

  • See our analysis paper: Searching for a Search Method: Benchmarking Search Algorithms for Generating NLP Adversarial Examples at EMNLP BlackBoxNLP.

  • As we emphasized in the above paper, we don't recommend to directly compare Attack Recipes out of the box.

  • This comment is due to that attack recipes in the recent literature used different ways or thresholds in setting up their constraints. Without the constraint space held constant, an increase in attack success rate could come from an improved search or transformation method or a less restrictive search space.

  • Our Github on benchmarking scripts and results: TextAttack-Search-Benchmark Github

On Quality of Generated Adversarial Examples in Natural Language

  • Our analysis Paper in EMNLP Findings
  • We analyze the generated adversarial examples of two state-of-the-art synonym substitution attacks. We find that their perturbations often do not preserve semantics, and 38% introduce grammatical errors. Human surveys reveal that to successfully preserve semantics, we need to significantly increase the minimum cosine similarities between the embeddings of swapped words and between the sentence encodings of original and perturbed sentences.With constraints adjusted to better preserve semantics and grammaticality, the attack success rate drops by over 70 percentage points.
  • Our Github on Reevaluation results: Reevaluating-NLP-Adversarial-Examples Github
  • As we have emphasized in this analysis paper, we recommend researchers and users to be EXTREMELY mindful on the quality of generated adversarial examples in natural language
  • We recommend the field to use human-evaluation derived thresholds for setting up constraints

Multi-lingual Support

Contributing to TextAttack

We welcome suggestions and contributions! Submit an issue or pull request and we will do our best to respond in a timely manner. TextAttack is currently in an "alpha" stage in which we are working to improve its capabilities and design.

See CONTRIBUTING.md for detailed information on contributing.

Citing TextAttack

If you use TextAttack for your research, please cite TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP.

@inproceedings{morris2020textattack,
  title={TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP},
  author={Morris, John and Lifland, Eli and Yoo, Jin Yong and Grigsby, Jake and Jin, Di and Qi, Yanjun},
  booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  pages={119--126},
  year={2020}
}
Comments
  • Addition of optional language/model parameters to constraints (and download method adjustment)

    Addition of optional language/model parameters to constraints (and download method adjustment)

    What does this PR do?

    Summary

    This PR adds optional language/model parameters to the __init__ of multiple constraint. This enables usage of the constraints for other languages than English and even multilingual use cases. Also it contains some changes of resource download logic.

    Additions

    • Optional language/model parameters for multiple constraints (previously hard-coded configurations are set as default values)

    Changes

    • Download logic of resources has been changed (see download_from_s3 method)

    Checklist

    • [ X ] The title of your pull request should be a summary of its contribution.
    • [ X ] Please write detailed description of what parts have been newly added and what parts have been modified. Please also explain why certain changes were made.
    • [ X ] If your pull request addresses an issue, please mention the issue number in the pull request description to make sure they are linked (and people consulting the issue know you are working on it)
    • [ X ] To indicate a work in progress please mark it as a draft on Github.
    • [ X ] Make sure existing tests pass.
    • [ X ] Add relevant tests. No quality testing = no merge.
    • [ X ] All public methods must have informative docstrings that work nicely with sphinx. For new modules/files, please add/modify the appropriate .rst file in TextAttack/docs/apidoc.'
    opened by alexander-zap 17
  • How to accelerate attack in parallel through API

    How to accelerate attack in parallel through API

    Hi,

    I'm developing my project with TextAttack API. I wonder how can I parallel the attack to accelerate? Here is the current code I'm using:

        
        model_wrapper = textattack.models.wrappers.HuggingFaceModelWrapper(
            model, tokenizer, batch_size=args.model_batch_size
        )
        recipe = PWWSRen2019.build(model_wrapper)
        
        for idx, result in enumerate(recipe.attack_dataset(dataset)):
            
            print(("-" * 20), f"Result {idx+1}", ("-" * 20))
            print(result.__str__(color_method="ansi"))
            print()
            if idx > 100:
                break
    

    The code returns one result at one time in the loop. Because I want to check the overall performance of the whole dataset so I only care the accuracy. How can I make full use of the GPU memory and accelerate the process? It seems the batch_size of model_wrapper does not help.

    Thanks

    enhancement question 
    opened by ziqi-zhang 17
  • Add chinese version of readme

    Add chinese version of readme

    Thanks for releasing the powerful TextAttack🎉 For chinese fellows easier to get started with TextAttack, I translate the reademe to chinese. I try my best to adheres to 信、达、雅, the translation principles. But only the content is guaranteed to be informative. This chinese version may need some major review. Despite all this, it's still a good startpoint. Look forward to receiving your reply~

    opened by Opdoop 17
  •  Support for spaCy models

    Support for spaCy models

    Hi, I was asked by @jxmorris12 to create this issue in connection to my original question on Stackoverflow: https://stackoverflow.com/questions/61889477/using-spacy-models-with-allennlp-interpret-or-textattack/61897642?noredirect=1#comment109982631_61897642

    I tried to figure out how to use Textattack with spaCy models, but was unsuccessful. I'll be grateful for any help :)

    documentation question 
    opened by katarkor 17
  • Fix incorrect `__eq__` method of `AttackedText` in `textattack/shared/attacked_text.py`

    Fix incorrect `__eq__` method of `AttackedText` in `textattack/shared/attacked_text.py`

    What does this PR do?

    Summary

    fix incorrect __eq__ method of AttackedText in textattack/shared/attacked_text.py

    ~~Another thing is that I added DownloadConfig in textattack/datasets/huggingface_dataset.py to try to use a proxy for downloading things from huggingface. This is irrelevant to this pull request.~~

    Additions

    • in the __eq__ method of AttackedText, an additional check of equal number of dict items of attack_attrs is added. If this correction was not done, the __eq__ is incorrect, as illustrated in the following example
    from textattack.shared.attacked_text import AttackedText
    at1 = AttackedText("hehe", attack_attrs={"hehe":1})
    at2 = AttackedText("hehe", attack_attrs={"hehe":1, "haha":2})
    
    print(at1 == at2)
    print(at2 == at1)
    

    whose outcome would be

    True
    False
    

    Changes

    • NA

    Deletions

    • NA

    Checklist

    • [√] The title of your pull request should be a summary of its contribution.
    • [√] Please write detailed description of what parts have been newly added and what parts have been modified. Please also explain why certain changes were made.
    • [√] If your pull request addresses an issue, please mention the issue number in the pull request description to make sure they are linked (and people consulting the issue know you are working on it)
    • [√] To indicate a work in progress please mark it as a draft on Github.
    • [√] Make sure existing tests pass.
    • [√] Add relevant tests. No quality testing = no merge.
    • [√] All public methods must have informative docstrings that work nicely with sphinx. For new modules/files, please add/modify the appropriate .rst file in TextAttack/docs/apidoc.'
    opened by wenh06 16
  • Attack via API

    Attack via API

    What does this PR do?

    Summary

    Performing attacks using the TextAttack API is quite limiting compared to using the command line commands because the users have to write their own code to support logging, saving checkpoint, and parallel processing. This PR intends to introduce some new support for attacking using TextAttack's API so that what can be done in command line can also be done using the API. However, this version does break a lot of backward compatibilities.

    Additions

    • Added new dataclasses AttackArgs, AugmenterArgs, DatasetArgs, ModelArgs that are intended to represent arguments for specific features of TextAttack.
    • Added new module named textattack.Attacker.Attacker takes in Attack, textattack.datasets.Dataset, and AttackArgs objects to perform attacks on the datasets with the same features for logging and saving checkpoints as the command line version.
    • Added textattack.datasets.Dataset class. This takes in a list (or list-like) of tuples of (input, output) format. It is intended to the be the most basic class that other dataset classes extend from. The idea is to require all "dataset" inputs to be of type textattack.datasets.Dataset, so it's clear the users what they need to pass as a "dataset" (behavior, an arbitrary list counted as a "dataset" technically).

    Changes

    • Removed attack_dataset method from Attack class and removed all the generator-related functionalities. Generator design is neat, but is less readable and maintainable. The idea is to remove all dataset-level attack features from Attack class and make Attacker handle the dataset level attacks. Attack is solely responsible for attacking single examples.
    • Updated textattack.datasets.HuggingFaceDatasets to allow users to pass in their own datasets.Dataset objects (instead of just taking in name, subset, and split to load the dataset).
    • Updated modules in textattack.commands to work with the new APIs.
    opened by jinyongyoo 14
  • Enhance augment function

    Enhance augment function

    What does this PR do?

    Summary

    This PR introduces 2 new augmenter parameters, high_yield and fast_augment. The high_yield option was originally implemented in pull request #507 that still requires additional implementation before merging.

    When high_yield is set to True, every augmentation that fits the criteria of a successful transformation will be added to the final output. In most cases, the high-yield augmenter will generate far more augmentations than what users specify in transformations_per_example.

    When fast_augment is set to True, the augmenter terminate and return transformations_per_example number of transformations when the number of successful augmentations reaches transformations_per_example.

    This improves the running time of the augmenter but may cause skewness in returned augmentations (speed is improved via early stop).

    Additions

    • Added high_yield and fast_augment parameters in augmenter

    Changes

    • Changed the augment function, augmenter parser, and augment_command

    Checklist

      • [x] The title of your pull request should be a summary of its contribution.
      • [x] Please write a detailed description of what parts have been newly added and what parts have been modified. Please also explain why certain changes were made.
    • [ ] If your pull request addresses an issue, please mention the issue number in the pull request description to make sure they are linked (and people consulting the issue know you are working on it)
      • [x] To indicate a work in progress please mark it as a draft on Github.
      • [x] Make sure existing tests pass.
    • [ ] Add relevant tests. No quality testing = no merge.
    • [ ] All public methods must have informative docstrings that work nicely with sphinx. For new modules/files, please add/modify the appropriate .rst file in TextAttack/docs/apidoc.'
    augmentation 
    opened by Hanyu-Liu-123 12
  • ValueError: Unsupported dataset schema #449

    ValueError: Unsupported dataset schema #449

    I am running adversarial training on NLP models and I am getting an error " ValueError: Unsupported dataset schema ". When I run the following code: import textattack import transformers from textattack.datasets import HuggingFaceDataset

    model = transformers.AutoModelForSequenceClassification.from_pretrained("bert-base-uncased") tokenizer = transformers.AutoTokenizer.from_pretrained("bert-base-uncased") model_wrapper = textattack.models.wrappers.HuggingFaceModelWrapper(model, tokenizer)

    We only use DeepWordBugGao2018 to demonstration purposes. attack = textattack.attack_recipes.DeepWordBugGao2018.build(model_wrapper) train_dataset = HuggingFaceDataset('squad', split='train') eval_dataset = HuggingFaceDataset('squad', split='validation')

    Train for 3 epochs with 1 initial clean epochs, 1000 adversarial examples per epoch, learning rate of 5e-5, and effective batch size of 32 (8x4). training_args = textattack.TrainingArgs( num_epochs=3, num_clean_epochs=1, num_train_adv_examples=1000, learning_rate=5e-5, per_device_train_batch_size=8, gradient_accumulation_steps=4, log_to_tb=True, )

    trainer = textattack.Trainer( model_wrapper, "classification", attack,

    eval_dataset, training_args ) trainer.train() @jxmorris12

    bug 
    opened by marwanomar1 12
  • Unexpected behaviour when using customized word embedding

    Unexpected behaviour when using customized word embedding

    I was using TextFoolerJin2019 to attack the distillbert sentiment analysis model from huggingface transformer from transformers import pipeline p = pipeline("sentiment-analysis"), but i downloaded https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz and replaced the original default paragram cf embedding in the recipe(one word embedding distance constrait and one word swap embedding transformation). I was using ['this is a bad movie', 0] as the input data.

    The thing is in this vocab there are words like this: 'coulda_shoulda_woulda' and 'slovakia_slota_commented'. And among the attackedtexts, in some of them 'bad' is replaced with complex words like these two. When this happens, these long words got tokenized and got broken into multiple words. One consequence is the 'newly_modified_indices' don't correspond correctly to the original sentence anymore. Therefore, list index of bound error occurs when checking the contraint.

    To be specific, in this case where reference text is 'this is a bad movie' and attackedtext is 'this is a slovakia_slota_commented movie', in the code https://github.com/QData/TextAttack/blob/master/textattack/constraints/semantics/word_embedding_distance.py#L134, newly_modified_indices are 3,4,5 (representing slovakia, slota, commented after tokenization) and the reference_text has only length 5, therefore 5 will be out of bound.

    I understand this maybe an unexpected way to use textattack, but i think it goes back to my previous wish list that textattack comes with its own ExceptionResult for any unexpected exceptions.

    Also i think it's better to use defaultdict(dict) as the default for cos_sim_mat and mse_sim_mat, otherwise the statement in except block can fail:https://github.com/QData/TextAttack/blob/master/textattack/constraints/semantics/word_embedding_distance.py#L106 this test case fails

    c = WordEmbeddingDistance(min_cos_sim=0.5)
    c.cos_sim_mat = dict()
    c.get_cos_sim(1, 10)
    print(c.cos_sim_mat)
    

    but this one passes

    c = WordEmbeddingDistance(min_cos_sim=0.5)
    c.cos_sim_mat = defaultdict(dict)
    c.get_cos_sim(1, 10)
    print(c.cos_sim_mat)
    

    Thanks,

    opened by tsinggggg 12
  • Added hard label attack recipe

    Added hard label attack recipe

    What does this PR do?

    Closes #385

    Summary

    This PR adds Hard Label Attack attack recipe which uses search space reduction and population based optimization procedure to craft adversarial attacks using only the topmost predicted label.

    Additions

    • Added hard-label-attack recipe as textattack.attack_recipes.HardLabelAttackMaheshwary2021.
    • Added hard-label-attack search method as textattack.search_methods.HardLabelAttack.

    Changes

    • README.md has been updated that now includes the attack details.
    • textattack/attack_recipes/__init__.py has been updated that now includes the attack recipe.
    • textattack/commands/attack/attack_args.py has been updated that now includes the attack recipe name.
    • textattack/search_methods/__init__.py has been updated that now includes the search method.

    Deletions

    • No deletions

    Checklist

    • [x] The title of your pull request should be a summary of its contribution.
    • [x] Please write detailed description of what parts have been newly added and what parts have been modified. Please also explain why certain changes were made.
    • [x] If your pull request addresses an issue, please mention the issue number in the pull request description to make sure they are linked (and people consulting the issue know you are working on it)
    • [x] To indicate a work in progress please mark it as a draft on Github.
    • [x] Make sure existing tests pass.
    • [x] Add relevant tests. No quality testing = no merge.
    • [x] All public methods must have informative docstrings that work nicely with sphinx. For new modules/files, please add/modify the appropriate .rst file in TextAttack/docs/apidoc.'
    opened by RishabhMaheshwary 11
  • Parallel Attack RuntimeError: generator raised StopIteration

    Parallel Attack RuntimeError: generator raised StopIteration

    Uss recipe textbuggerattack imdb^test dataset, get RuntimeError: generator raised StopIteration The trackback is below:

      File "/u01/jh/Github/TextAttack/textattack/commands/attack/attack_command.py", line 47, in run
        run_parallel(args)
      File "/u01/jh/Github/TextAttack/textattack/commands/attack/run_attack_parallel.py", line 151, in run
        raise result
    RuntimeError: generator raised StopIteration
    

    I tried in python 3.7 with torch 1.7 and torch 1.6. Also python 3.6 with torch 1.6. This RuntimeError keeps show up in all these environments, although in different iterations.

    bug 
    opened by Opdoop 11
  • Setup issue: fatal error C1083: Cannot open compiler generated file: '': Invalid argument

    Setup issue: fatal error C1083: Cannot open compiler generated file: '': Invalid argument

    Hello,

    I'm tried to install textattack using pip package and also through the source option and I'm always getting the same error message. I've already tried to update pip, but I still get the same error message:

    Building wheels for collected packages: pycld2
      Building wheel for pycld2 (setup.py) ... error
      error: subprocess-exited-with-error
    
      × python setup.py bdist_wheel did not run successfully.
      │ exit code: 1
      ╰─> [41 lines of output]
          C:\some_folder\Python\Python39\lib\site-packages\setuptools\config\setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
            warnings.warn(msg, warning_class)
          running bdist_wheel
          The [wheel] section is deprecated. Use [bdist_wheel] instead.
          running build
          running build_py
          creating build
          creating build\lib.win-amd64-cpython-39
          creating build\lib.win-amd64-cpython-39\pycld2
          copying pycld2\__init__.py -> build\lib.win-amd64-cpython-39\pycld2
          running build_ext
          building 'pycld2._pycld2' extension
          creating build\temp.win-amd64-cpython-39
          creating build\temp.win-amd64-cpython-39\Release
          creating build\temp.win-amd64-cpython-39\Release\Users
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal
          "c:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\public -IC:\some_folder\Python\Python39\include -IC:\some_folder\Python\Python39\Include "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\ATLMFC\include" "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /EHsc /TpC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings\encodings.cc /Fobuild\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings\encodings.obj -w -O2 -m64 -fPIC      
          cl : Command line warning D9025 : overriding '/W3' with '/w'
          cl : Command line warning D9002 : ignoring unknown option '-m64'
          cl : Command line warning D9002 : ignoring unknown option '-fPIC'
          encodings.cc
          "c:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\public -IC:\some_folder\Python\Python39\include -IC:\some_folder\Python\Python39\Include "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\ATLMFC\include" "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /EHsc /TpC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings\pycldmodule.cc /Fobuild\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings\pycldmodule.obj -w -O2 -m64 -fPIC  
          cl : Command line warning D9025 : overriding '/W3' with '/w'
          cl : Command line warning D9002 : ignoring unknown option '-m64'
          cl : Command line warning D9002 : ignoring unknown option '-fPIC'
          pycldmodule.cc
          "c:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\public -IC:\some_folder\Python\Python39\include -IC:\some_folder\Python\Python39\Include "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\ATLMFC\include" "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /EHsc /TpC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal/cld2_generated_cjk_compatible.cc /Fobuild\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal/cld2_generated_cjk_compatible.obj -w -O2 -m64 -fPIC
          cl : Command line warning D9025 : overriding '/W3' with '/w'
          cl : Command line warning D9002 : ignoring unknown option '-m64'
          cl : Command line warning D9002 : ignoring unknown option '-fPIC'
          cld2_generated_cjk_compatible.cc
          C:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal\cld2_generated_cjk_compatible.cc : fatal error C1083: Cannot open compiler generated file: '': Invalid argument
          error: command 'c:\\some_folder\\Microsoft Visual Studio\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit code 1
          [end of output]
    
      note: This error originates from a subprocess, and is likely not a problem with pip.
      ERROR: Failed building wheel for pycld2
      Running setup.py clean for pycld2
    Failed to build pycld2
    Installing collected packages: pycld2, py4j, pptree, pinyin, overrides, mpld3, lru-dict, jieba, janome, docopt, zipp, xxhash, wrapt, urllib3, typing-extensions, threadpoolctl, terminaltables, tabulate, soupsieve, smart-open, six, regex, pyyaml, PySocks, pyparsing, pillow, packaging, numpy, num2words, networkx, multidict, more-itertools, lxml, kiwisolver, joblib, idna, future, ftfy, fsspec, frozenlist, fonttools, editdistance, dill, Cython, cycler, conllu, colorama, cloudpickle, charset-normalizer, certifi, attrs, async-timeout, yarl, tqdm, torch, segtok, scipy, requests, python-dateutil, pyarrow, multiprocess, lemminflect, langdetect, importlib-metadata, deprecated, contourpy, click, beautifulsoup4, anytree, aiosignal, wikipedia-api, simpful, scikit-learn, responses, pandas, OpenHowNet, nltk, miniful, matplotlib, language_tool_python, konoha, hyperopt, huggingface-hub, aiohttp, transformers, gdown, fst-pso, pyfume, datasets, bert-score, FuzzyTM, gensim, bpemb, flair, textattack
      Running setup.py install for pycld2 ... error
      error: subprocess-exited-with-error
    
      × Running setup.py install for pycld2 did not run successfully.
      │ exit code: 1
      ╰─> [42 lines of output]
          C:\some_folder\Python\Python39\lib\site-packages\setuptools\config\setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
            warnings.warn(msg, warning_class)
          running install
          C:\some_folder\Python\Python39\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
            warnings.warn(
          running build
          running build_py
          creating build
          creating build\lib.win-amd64-cpython-39
          creating build\lib.win-amd64-cpython-39\pycld2
          copying pycld2\__init__.py -> build\lib.win-amd64-cpython-39\pycld2
          running build_ext
          building 'pycld2._pycld2' extension
          creating build\temp.win-amd64-cpython-39
          creating build\temp.win-amd64-cpython-39\Release
          creating build\temp.win-amd64-cpython-39\Release\Users
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2
          creating build\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal
          "c:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\public -IC:\some_folder\Python\Python39\include -IC:\some_folder\Python\Python39\Include "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\ATLMFC\include" "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /EHsc /TpC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings\encodings.cc /Fobuild\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings\encodings.obj -w -O2 -m64 -fPIC      
          cl : Command line warning D9025 : overriding '/W3' with '/w'
          cl : Command line warning D9002 : ignoring unknown option '-m64'
          cl : Command line warning D9002 : ignoring unknown option '-fPIC'
          encodings.cc
          "c:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\public -IC:\some_folder\Python\Python39\include -IC:\some_folder\Python\Python39\Include "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\ATLMFC\include" "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /EHsc /TpC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings\pycldmodule.cc /Fobuild\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\bindings\pycldmodule.obj -w -O2 -m64 -fPIC  
          cl : Command line warning D9025 : overriding '/W3' with '/w'
          cl : Command line warning D9002 : ignoring unknown option '-m64'
          cl : Command line warning D9002 : ignoring unknown option '-fPIC'
          pycldmodule.cc
          "c:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal -IC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\public -IC:\some_folder\Python\Python39\include -IC:\some_folder\Python\Python39\Include "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\ATLMFC\include" "-Ic:\some_folder\Microsoft Visual Studio\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /EHsc /TpC:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal/cld2_generated_cjk_compatible.cc /Fobuild\temp.win-amd64-cpython-39\Release\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal/cld2_generated_cjk_compatible.obj -w -O2 -m64 -fPIC
          cl : Command line warning D9025 : overriding '/W3' with '/w'
          cl : Command line warning D9002 : ignoring unknown option '-m64'
          cl : Command line warning D9002 : ignoring unknown option '-fPIC'
          cld2_generated_cjk_compatible.cc
          C:\Users\xxx\AppData\Local\Temp\pip-install-zc43aqei\pycld2_842e29bfe00540bc8c314808a0cc68a9\cld2\internal\cld2_generated_cjk_compatible.cc : fatal error C1083: Cannot open compiler generated file: '': Invalid argument
          error: command 'c:\\some_folder\\Microsoft Visual Studio\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit code 1
          [end of output]
    
      note: This error originates from a subprocess, and is likely not a problem with pip.
    error: legacy-install-failure
    
    × Encountered error while trying to install package.
    ╰─> pycld2
    

    Does anyone know how to solve this? I'll try to install it in another computer to see if it works, meanwhile.

    opened by raulkviana 1
  • Fix the problem of text output from T5 model

    Fix the problem of text output from T5 model

    To deal with the text output from the T5 model, where the array of text output cannot be converted into a tensor. Therefore, directly processing the numpy array is better.

    opened by plasmashen 1
  • [Draft] Add adversarial defense benchmark

    [Draft] Add adversarial defense benchmark

    What does this PR do?

    Introduce an interface of adversarial defense to benchmark adversarial defense methods.

    Summary

    The test examples is available in https://github.com/yangheng95/TextAttack/blob/master/examples/reactive_defense/sst2_reactive_defense.py Please contact me and help test if you are interested in this PR. @jxmorris12

    This PR is related to https://github.com/QData/TextAttack/issues/687

    Additions

    • [1] https://github.com/yangheng95/TextAttack/blob/master/textattack/models/wrappers/pyabsa_model_wrapper.py
    • [2] https://github.com/yangheng95/TextAttack/tree/master/textattack/reactive_defense

    Changes

    • [1] Add **kwargs for many functions, see the source code.
    • [2] Modify the search_method()

    Deletions

    • Example: Remove unnecessary files under textattack.models...

    Checklist

    • [x] The title of your pull request should be a summary of its contribution.
    • [x] Please write detailed description of what parts have been newly added and what parts have been modified. Please also explain why certain changes were made.
    • [x] If your pull request addresses an issue, please mention the issue number in the pull request description to make sure they are linked (and people consulting the issue know you are working on it)
    • [x] To indicate a work in progress please mark it as a draft on Github.
    • [x] Make sure existing tests pass.
    • [x] Add relevant tests. No quality testing = no merge.
    • [x] All public methods must have informative docstrings that work nicely with sphinx. For new modules/files, please add/modify the appropriate .rst file in TextAttack/docs/apidoc.'
    opened by yangheng95 1
  • Wrong label mapping for released model textattack/bert-base-uncased-MNLI

    Wrong label mapping for released model textattack/bert-base-uncased-MNLI

    Describe the bug When I am using the released fine-tuned model checkpoint textattack/bert-base-uncased-MNLI to do evaluation on MNLI task with the Huggingface transformers texxt-classification example scripts, I can only get 7% accuracy, rather than the reported 84% accuracy. I investigate the error and find that is because there is a significant label mapping error. The output-GT label mapping should be: 0 (model) -> 2 (GT), 1 (model) -> 0 (GT), 2 (model) -> 1 (GT). With this mapping, I can successfully retain 84% accuracy.

    To Reproduce In Huggingface transformers texxt-classification example scripts, you can get the running snippets using run_glue.py in README.md. Simply replace the original bert-base-cased with textattack/bert-base-uncased-MNLI and remove --do_train. You can get the results.

    Expected behavior By running the above steps, after the evaluation is done, you should be able to see the evaluation accuracy is <10%.

    Screenshots or Traceback This is not a running bug, and due to some legal policies, I cannot show the running results in clusters. But it should be fairly easy to replicate.

    System Information (please complete the following information): I use A100, so I use torch==1.9.0+cu111. I use transformers version 4.21 and remove check_min_version from run_glue.py. But I think this operation is not related to the bug.

    Additional context Add any other context about the problem here.

    model-zoo 
    opened by yangalan123 1
  • Can you provide a simple attack API for single sentence attack test?

    Can you provide a simple attack API for single sentence attack test?

    Is your feature request related to a problem? Please describe. No

    Describe the solution you'd like Here is a simply idea for single text attack which is rewrited from original batch attack function

        def simple_attack(self, text, label):
            if torch.cuda.is_available():
                self.attack.cuda_()
    
            example, ground_truth_output = text, label
            try:
                example = textattack.shared.AttackedText(example)
                if self.dataset.label_names is not None:
                    example.attack_attrs["label_names"] = self.dataset.label_names
                try:
                    result = self.attack.attack(example, ground_truth_output)
                except Exception as e:
                    raise e
                if (isinstance(result, SkippedAttackResult) and self.attack_args.attack_n) or (
                    not isinstance(result, SuccessfulAttackResult)
                    and self.attack_args.num_successful_examples
                ):
                    return
                else:
                    return result
            except KeyboardInterrupt as e:
                raise e
    

    In this way, we need to initialize the attacker as usual, and the dataset loading method can be refactored, e.g., provide a default null dataset object instead of compulsory loading dataset from a file. e.g.,

            # dataset = HuggingFaceDataset("sst", split="test")
            # data = pandas.read_csv('examples.csv')
            dataset = [('', 0)]
            dataset = Dataset(dataset)
    
            self.attacker = Attacker(recipe, dataset)
    

    Describe alternatives you've considered No

    Additional context I also suggest to support mirror the tfhub which will help people in China(unable to access tfhub.dev). e.g., https://github.com/yangheng95/TextAttack/blob/018c561b27182164dc76fb5e43516f7bc64801b1/textattack/constraints/semantics/sentence_encoders/universal_sentence_encoder/universal_sentence_encoder.py#L21 and https://github.com/yangheng95/TextAttack/blob/018c561b27182164dc76fb5e43516f7bc64801b1/textattack/constraints/semantics/sentence_encoders/universal_sentence_encoder/multilingual_universal_sentence_encoder.py#L23 . However, this is up to you.

    Finally, I recommand to distinguish the name of attacked_text in the original_result and perturded_result, as I got trouble in confusing them in experiments, and lost some time.

    Thanks very much for this great work!

    documentation 
    opened by yangheng95 0
Releases(v0.3.8)
  • v0.3.8(Nov 2, 2022)

    #689: Add more type annotations and do some code cleanup in AttackedText notably removed some code that did Chinese word segmentation because it did not properly support words_from_text, which caused issues with various transformations.

    #691: Optimize comparison between two AttackedText objects (thanks @plasmashen!)

    #693: Fix bug with writing parameters twice in AttackedText (thanks @89x98!)

    #700: Lots of miscellaneous bug fixes and some helper function implementation

    #701: Fix bugs with loading TedTalk translation dataset, using T5, seq2sick/text-to-text goal functions

    Source code(tar.gz)
    Source code(zip)
  • v0.3.7(Aug 14, 2022)

    • Update dependency: transformers>=4.21.0
    • Update dependency: datasets==2.4.0
    • Update optional dependency: sentence_transformers==2.2.0
    • Update optional dependency: gensim==4.1.2
    • Update optional dependency: tensorflow==2.7.0 (Thanks @VijayKalmath !!!!)
    • Miscellaneous fixes for new packages to update things and remove warning messages
    • Fix logging attack args to W&B #647 (thanks @VijayKalmath)
    • Fix bug with word_swap_masked_lm #649 (thanks @Hanyu-Liu-123)
    • Fix small issues with textattack train #653 (thanks @VijayKalmath)
    • Fix issue with PWWS #654 (thanks @VijayKalmath)
    • Update recipe for FasterGeneticAlgorithm to match paper #656 (thanks @VijayKalmath)
    • Update adversarial dataset generation logic #657 (thanks @VijayKalmath)
    • Update dataset_args to correctly set dataset_split #659 (thanks @VijayKalmath)
    • Add logic for loading SQUAD via HuggingFaceDataset class #660 (thanks @VijayKalmath)
    • Fix ANSI color-printing #662
    • Make GreedyWordSwapWIR and related search methods more query-efficient under the presence of pre-transformation constraints #665 and #674 (thanks @VijayKalmath)
    • Save attack summary table as JSON (thanks @VijayKalmath -- great feature add!!)
    • Fix typo and update numpy #671 and #672 (thanks @JohnGiorgi -- and welcome!)
    • Finish CLARE attack #675 (thanks @Hanyu-Liu-123 and @VijayKalmath)
    • Add repr for better user experience with GoalFunctionResult #676 (thanks @VijayKalmath)
    • Better exception handling in WordSwapChangeNumber ((thanks @dangne -- and welcome!!)
    • Various other typo and bug fixes

    Thanks to everyone who contributed to TextAttack this summer, and a special shoutout once more to @VijayKalmath for all the hard work and attention to detail. Glad to see TextAttack so healthy 🙂

    Source code(tar.gz)
    Source code(zip)
  • v0.3.5(May 25, 2022)

    • #644:
      • Ability to specify device via TA_DEVICE env variable
      • New constraint, MaxNumWordsModified
      • Tracks previous AttackedText during attack to allow for reconstruction of chain of modifications
      • ChangeGreedyWordSwapWIR to allow passing of specific unk token
      • Formatting updates to new Black version
      • fix Universal Sentence Encoder from TF breakage
      • fix Flair to new API (thanks @VijayKalmath for the help!)
    • Bump version to 0.3.5
    • #623 Fix quotation bug, thanks @donggrant
    • #613 and others, fix dependencies
    • #609 Only initialize embeddings when needed :) thanks to @duesenfranz
    • #591 fix a bug with CLARE
    Source code(tar.gz)
    Source code(zip)
  • v0.3.4(Nov 10, 2021)

    What's Changed

    • [CODE] Keras parallel attack fix - Issue #499 by @sanchit97 in https://github.com/QData/TextAttack/pull/515
    • Bump tensorflow from 2.4.2 to 2.5.1 in /docs by @dependabot in https://github.com/QData/TextAttack/pull/517
    • Add a high level overview diagram to docs by @cogeid in https://github.com/QData/TextAttack/pull/519
    • readtheDoc fix by @qiyanjun in https://github.com/QData/TextAttack/pull/522
    • Add new attack recipe A2T by @jinyongyoo in https://github.com/QData/TextAttack/pull/523
    • Fix incorrect __eq__ method of AttackedText in textattack/shared/attacked_text.py by @wenh06 in https://github.com/QData/TextAttack/pull/509
    • Fix a bug when running textattack eval with --num-examples=-1 by @dangne in https://github.com/QData/TextAttack/pull/521
    • New metric module to improve flexibility and intuitiveness - moved from #475 by @sanchit97 in https://github.com/QData/TextAttack/pull/514
    • Update installation.md to add FAQ on installation by @qiyanjun in https://github.com/QData/TextAttack/pull/535
    • Fix dataset-split bug by @Hanyu-Liu-123 in https://github.com/QData/TextAttack/pull/533
    • Update by @Hanyu-Liu-123 in https://github.com/QData/TextAttack/pull/541
    • add custom dataset API use example in doc by @qiyanjun in https://github.com/QData/TextAttack/pull/543
    • Fix logger initiation bug by @Hanyu-Liu-123 in https://github.com/QData/TextAttack/pull/539
    • Updated Tutorial 0 to use the Rotten Tomatoes dataset instead of the … by @srujanjoshi in https://github.com/QData/TextAttack/pull/542
    • Back translation transformation by @cogeid in https://github.com/QData/TextAttack/pull/534
    • Fixed a bug in the allennlp tutorial by @donggrant in https://github.com/QData/TextAttack/pull/546
    • Logger bug fix by @ankitgv0 in https://github.com/QData/TextAttack/pull/551
    • add "textattack[tensorflow]" option in all tutorials by @qiyanjun in https://github.com/QData/TextAttack/pull/559
    • Fix CLARE Extra Character Bug by @Hanyu-Liu-123 in https://github.com/QData/TextAttack/pull/556
    • Fix metric-module Issue#532 by @sanchit97 in https://github.com/QData/TextAttack/pull/540
    • Add API docstrings for back translation by @cogeid in https://github.com/QData/TextAttack/pull/563
    • Fixed the "no attribute" error from #536 by @ankitgv0 in https://github.com/QData/TextAttack/pull/552
    • Enhance augment function by @Hanyu-Liu-123 in https://github.com/QData/TextAttack/pull/531
    • fix read-the-doc installation issue / clean up and add new docstrings for recently added classes/packages by @qiyanjun in https://github.com/QData/TextAttack/pull/569

    New Contributors

    • @wenh06 made their first contribution in https://github.com/QData/TextAttack/pull/509
    • @dangne made their first contribution in https://github.com/QData/TextAttack/pull/521
    • @srujanjoshi made their first contribution in https://github.com/QData/TextAttack/pull/542
    • @donggrant made their first contribution in https://github.com/QData/TextAttack/pull/546
    • @ankitgv0 made their first contribution in https://github.com/QData/TextAttack/pull/551

    Full Changelog: https://github.com/QData/TextAttack/compare/v0.3.3...v0.3.4

    Source code(tar.gz)
    Source code(zip)
  • v0.3.3(Aug 3, 2021)

    1. Merge pull request #508 from QData/example_bug_fix

    2. Merge pull request #505 from QData/s3-model-fix

    3. Merge pull request #503 from QData/multilingual-doc

    4. Merge pull request #502 from QData/Notebook-10-bug-fix

    5. Merge pull request #500 from QData/docstring-rework-missing

    6. Merge pull request #497 from QData/dependabot/pip/docs/tensorflow-2.4.2

    7. Merge pull request #495 from QData/readthedoc-fix

    Source code(tar.gz)
    Source code(zip)
  • v0.3.2(Jul 28, 2021)

    Multiple bug fixes:

    • Merge pull request #473 from cogeid/file-redirection-fix

    • Merge pull request #469 from xinzhel/allennlp_doc

    • Merge pull request #477 from cogeid/Fix-RandomSwap-and-RandomSynonymI…

    • Merge pull request #484 from QData/update-torch-version

    • Merge pull request #490 from QData/scipy-version-plus-two-doc-updates

    • Merge pull request #420 from QData/multilingual

    • Merge pull request #495 from QData/readthedoc-fix

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Jun 25, 2021)

    New Updated API

    We have added two new classes called Attacker and Trainer that can be used to perform adversarial attacks and adversarial training with full logging support and multi-GPU parallelism. This is intended to provide an alternative way of performing attacks and training for custom models and datasets.

    Attacker: Running Adversarial Attacks

    Below is an example use of Attacker to attack BERT model finetuned on IMDB dataset using TextFooler method. AttackArgs class is used to set the parameters of the attacks, including the number of examples to attack, CSV file to log the results, and the interval at which to save checkpoint.

    Screen Shot 2021-06-24 at 8 34 44 PM

    More details about Attacker and AttackArgs can be found here.

    Trainer: Running Adversarial Training

    Previously, TextAttack supported adversarial training in a limited manner. Users could only train models using the CLI command, and not every aspects of training was available for tuning.

    Trainer class introduces an easy way to train custom PyTorch/Transformers models on a custom dataset. Below is an example where we finetune BERT on IMDB dataset with an adversarial attack called DeepWordBug.

    Screen Shot 2021-06-25 at 9 28 57 PM

    Dataset

    Previously, datasets passed to TextAttack were simply expected to be an iterable of (input, target) tuples. While this offers flexibility, it prevents users from passing key information about the dataset that TextAttack can use to provide better experience (e.g. label names, label remapping, input column names used for printing).

    We instead explicitly define Dataset class that users can use or subclass for their own datasets.

    Bug Fixes:

    • #467: Don't check self.target_max_score when it is already known to be None.
    • #417: Fixed bug where in masked_lm transformations only subwords were candidates for top_words.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.15(Dec 27, 2020)

    CLARE Attack (#356, #392)

    We have added a new attack proposed by "Contextualized Perturbation for Textual Adversarial Attack" (Li et al., 2020). There's also a corresponding augmenter recipe using CLARE. Thanks to @Hanyu-Liu-123, @cookielee77.

    Custom Word Embedding (#333, #399)

    We have added support for custom word embedding via AbstractWordEmbedding, WordEmbedding, GensimWordEmbedding fromtextattack.shared. These three classes allow users to use their own custom word embeddings for transformations and constraints that require custom word embeddings. Thanks @tsinggggg and @alexander-zap for contributing!

    Bug Fixes and Changes

    • We fixed a bug that caused TextAttack to report fewer number of average queries than what it should be reporting (#350, thanks @ a1noack).
    • Update the dataset split used to evaluate robustness during adversarial training (#361, thanks @Opdoop).
    • Updated default parameters for TextBugger recipe (#373)
    • Fixed an issue with TextBugger by updating the default method used to segment text into words to work with homoglyphs. (#376, thanks @lethaiq!)
    • Updated ModelWrapper to not require get_grad method to be defined. (#381)
    • Fixed an issue with WordSwapMaskedLM that was causing words with lowest probability to be picked first. (#396)
    Source code(tar.gz)
    Source code(zip)
  • 0.2.14(Nov 18, 2020)

    Improvements

    Bug fixing Matching documentation in Readme.md and the files in /doc folder add checklist add multilingual USE add gradient-based word importance ranking update to a more complete API documentation add cola constraint add the lazy loader

    Source code(tar.gz)
    Source code(zip)
  • 0.2.12(Nov 13, 2020)

    Big Improvements

    • add checklist
    • add multilingual USE
    • add gradient-based word importance ranking
    • update to a more complete API documentation
    • add cola constraint
    • add the lazy loader
    Source code(tar.gz)
    Source code(zip)
  • 0.2.0(Jul 9, 2020)

    Big Improvements

    • Add tons of (over 70!) pre-trained models (#192, see Model Zoo page!)
    • Data augmentation integrated into training! (#195, thanks @jakegrigsby)
    • Allow for maximization goal functions (#151, thanks @uvafan )

    New Attacks

    • Add the Improved Genetic Algorithm (#183, thanks @sherlockyyc!)
    • Add BAE and BERT-Attack attack recipes (#160)
    • Add PWWS attack (#168, thanks @jakegrigsby)
    • Add typo-based attack from Pruthi et al. (#191, thanks @jakegrigsby )
    • Easy Data Augmentation augmentation recipe (#168, thanks @jakegrigsby)
    • Add input reduction attack from Feng et al. (#161, thanks @uvafan)

    Smaller Improvements

    • more accurate attack recipes for BAE and TextFooler (#199)
    • important fixes to model training code (#186, thanks so much @jind11!!)
    • abstract classes, better string representations when printing attacks to console (#202)
    • genetic algorithm improvements (#160, thanks @jinyongyoo )
    • fixes to parallel attacks (#164, thanks @jinyongyoo )
    • datasets to test out T5 on seq2seq attacks (#176)

    Bug Fixes

    • correctly print attack perturbed words in color, even when words are deleted & inserted (#200)
    • fix print_step bug with alzantot recipe (#195, thanks @heytitle for reporting!)
    • fix some annoying issues with dependency versioning
    Source code(tar.gz)
    Source code(zip)
  • 0.1.0(Jun 24, 2020)

    Version 0.1.0 is our biggest release yet! Here's a summary of the changes:

    Backwards compatibility note: python -m textattack <args> is renamed to python -m textattack attack <args>. Or, better yet, textattack attack <args>!

    Big improvements

    • add textattack command (#132)
      • add textattack augment, textattack eval, textattack attack, textattack list (#132)
      • add textattack train, textattack peek-dataset, and lots of infrastructure for training models (#139)
    • Move all datasets to nlp format; temporarily remove non-NLP datasets (AGNews, English->German translation) (#134)

    Smaller improvements

    • Better output formatted -- show labels ("Positive", "Entailment") and confidence score (91%) in output (#142)
    • add MaxLengthModification constraint that prevents modifications beyond tokenizer max_length (#143)
    • Add pytest tests and code formatting with black; run tests on Python 3.6, 3.7, 3.8 with Travis CI (#127, #136)
    • Update NLTK part-of-speech constraint and support part-of-speech tagging with FLAIR instead (#135)
    • add BERTScore constrained based on "BERTScore: Evaluating Text Generation with BERT" (Zhang et al, 2019) (#146)
    • make logging to file optional (#145)
    • Updates to Checkpoint class; track attack results in a worklist; attack resume fixes (#128, #141)
    • Silence Weights & Biases warning message when not being used (#130)
    • Optionally point all cache directories to a universal cache directory, TA_CACHE_DIR (#150)

    Bug fixes

    • Fix a bug that can be encountered when resuming attacks from checkpoints (#149)
    • Fix a bug in Greedy word-importance-ranking deletion (#152)
    • Documentation updates and fixes (#153)
    Source code(tar.gz)
    Source code(zip)
  • 0.0.3.0(Jun 11, 2020)

    big changes:

    • load transformers models from the command-line using the --model-from-huggingface option
    • load nlp datasets from the command-line using the --dataset-from-nlp option
    • command-line support for custom attacks, models, and datasets: --attack-from-file, --model-from-file, --dataset-from-file
    • implement attack recipe for TextBugger attack
    • add WordDeletion transformation

    small changes:

    • support white-box transformations via the command-line
    • allow Greedy-WIR to rank things in order of ascending importance
    • use fast tokenizers behind the scenes
    • fix some bugs with the attack Checkpoint class
    • some abbreviated syntax (textattack.shared.utils.get_logger() -> textattack.shared.logger, textattack.shared.utils.get_device() -> textattack.shared.utils.device)
    • substantially decrease overall TokenizedText memory usage
    Source code(tar.gz)
    Source code(zip)
  • 0.0.2(May 21, 2020)

    0.0.2: Better documentation, attack checkpoints, PreTransformationConstraints, and more

    • Major documentation restructure (check it out)
    • Some refactoring and variable renames to make it easier to jump right in and start working with TextAttack
    • Introduction of PreTransformationConstraints: constraints now can be applied before the transformation to prevent word modifications at certain indices. This abstraction allowed us to remove the notion of modified_indices from search methods, which paves the way for us to introduce attacks that insert or delete words and phrases, as opposed to simply swapping words.
    • Separation of Attack and SearchMethod: search methods are now a parameter to the attack instead of different subclasses of Attack. This syntax fits better with our framework and enforces a clearer sense of separation between the responsibilities of the attack and those of the search method.
    • Transformation and constraint compatibility: Constraints now ensure they're compatible with a specific transformation via a check_compatibility method
    • Goal function scores are now normalized between 0 and 1: UntargetedClassification and NonOverlappingOutput now return scores between 0 and 1.
    • Attack Checkpoints: Attacks can now save and resume their progress. This is really useful for running long, expensive attacks. python-m textattack supports new checkpoint-related arguments: --checkpoint-interval and --checkpoint-dir
    • Weights & Biases: Log attack results to Weights & Biases by adding the --enable-wandb flag
    Source code(tar.gz)
    Source code(zip)
Owner
QData
http://www.cs.virginia.edu/yanjun/
QData
Training RNNs as Fast as CNNs

News SRU++, a new SRU variant, is released. [tech report] [blog] The experimental code and SRU++ implementation are available on the dev branch which

Tao Lei 14 Dec 12, 2022
🚀 RocketQA, dense retrieval for information retrieval and question answering, including both Chinese and English state-of-the-art models.

In recent years, the dense retrievers based on pre-trained language models have achieved remarkable progress. To facilitate more developers using cutt

475 Jan 04, 2023
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

English | 简体中文 | 繁體中文 | 한국어 State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow 🤗 Transformers provides thousands of pretrained models

Hugging Face 77.1k Dec 31, 2022
ThinkTwice: A Two-Stage Method for Long-Text Machine Reading Comprehension

ThinkTwice ThinkTwice is a retriever-reader architecture for solving long-text machine reading comprehension. It is based on the paper: ThinkTwice: A

Walle 4 Aug 06, 2021
Estimation of the CEFR complexity score of a given word, sentence or text.

NLP-Swedish … allows to estimate CEFR (Common European Framework of References) complexity score of a given word, sentence or text. CEFR scores come f

3 Apr 30, 2022
Code of paper: A Recurrent Vision-and-Language BERT for Navigation

Recurrent VLN-BERT Code of the Recurrent-VLN-BERT paper: A Recurrent Vision-and-Language BERT for Navigation Yicong Hong, Qi Wu, Yuankai Qi, Cristian

YicongHong 109 Dec 21, 2022
A relatively simple python program to generate one of those reddit text to speech videos dominating youtube.

Reddit text to speech generator A basic reddit tts video generator Current functionality Generate videos for subs based on comments,(askreddit) so rea

Aadvik 17 Dec 19, 2022
WikiPron - a command-line tool and Python API for mining multilingual pronunciation data from Wiktionary

WikiPron WikiPron is a command-line tool and Python API for mining multilingual pronunciation data from Wiktionary, as well as a database of pronuncia

213 Jan 01, 2023
The ability of computer software to identify words and phrases in spoken language and convert them to human-readable text

speech-recognition-py Speech recognition is the ability of computer software to identify words and phrases in spoken language and convert them to huma

Deepangshi 1 Apr 03, 2022
Data preprocessing rosetta parser for python

datapreprocessing_rosetta_parser I've never done any NLP or text data processing before, so I wanted to use this hackathon as a learning opportunity,

ASReview hackathon for Follow the Money 2 Nov 28, 2021
A multi-voice TTS system trained with an emphasis on quality

TorToiSe Tortoise is a text-to-speech program built with the following priorities: Strong multi-voice capabilities. Highly realistic prosody and inton

James Betker 2.1k Jan 01, 2023
A Chinese to English Neural Model Translation Project

ZH-EN NMT Chinese to English Neural Machine Translation This project is inspired by Stanford's CS224N NMT Project Dataset used in this project: News C

Zhenbang Feng 29 Nov 26, 2022
Exploration of BERT-based models on twitter sentiment classifications

twitter-sentiment-analysis Explore the relationship between twitter sentiment of Tesla and its stock price/return. Explore the effect of different BER

Sammy Cui 2 Oct 02, 2022
T‘rex Park is a Youzan sponsored project. Offering Chinese NLP and image models pretrained from E-commerce datasets

T‘rex Park is a Youzan sponsored project. Offering Chinese NLP and image models pretrained from E-commerce datasets (product titles, images, comments, etc.).

55 Nov 22, 2022
Kestrel Threat Hunting Language

Kestrel Threat Hunting Language What is Kestrel? Why we need it? How to hunt with XDR support? What is the science behind it? You can find all the ans

Open Cybersecurity Alliance 201 Dec 16, 2022
Anuvada: Interpretable Models for NLP using PyTorch

Anuvada: Interpretable Models for NLP using PyTorch So, you want to know why your classifier arrived at a particular decision or why your flashy new d

EDGE 102 Oct 01, 2022
Comprehensive-E2E-TTS - PyTorch Implementation

A Non-Autoregressive End-to-End Text-to-Speech (text-to-wav), supporting a family of SOTA unsupervised duration modelings. This project grows with the research community, aiming to achieve the ultima

Keon Lee 114 Nov 13, 2022
A Non-Autoregressive Transformer based TTS, supporting a family of SOTA transformers with supervised and unsupervised duration modelings. This project grows with the research community, aiming to achieve the ultimate TTS.

A Non-Autoregressive Transformer based TTS, supporting a family of SOTA transformers with supervised and unsupervised duration modelings. This project grows with the research community, aiming to ach

Keon Lee 237 Jan 02, 2023
Repository to hold code for the cap-bot varient that is being presented at the SIIC Defence Hackathon 2021.

capbot-siic Repository to hold code for the cap-bot varient that is being presented at the SIIC Defence Hackathon 2021. Problem Inspiration A plethora

Aryan Kargwal 19 Feb 17, 2022
ProteinBERT is a universal protein language model pretrained on ~106M proteins from the UniRef90 dataset.

ProteinBERT is a universal protein language model pretrained on ~106M proteins from the UniRef90 dataset. Through its Python API, the pretrained model can be fine-tuned on any protein-related task in

241 Jan 04, 2023