An example project using OpenPrompt under pytorch-lightning for prompt-based SST2 sentiment analysis model

Last update: Oct 21, 2022

Related tags

Overview

pl_prompt_sst

An example project using OpenPrompt under the framework of pytorch-lightning for a training prompt-based text classification model on SST2 sentiment analysis dataset. Leveraging the pytorch-lightning features like logging, gradient accumulation and early stopping, etc. Can be used as a template for further development.

Run

Install requirement

pip install -r requirements.txt

Setup the prompt to use in sst2/prompt_config.json

{
    "template_text": "{\"placeholder\": \"text_a\"} In summary, the film was {\"mask\"}.",
    "label_words": [["bad"], ["good"]]
}

Adjust the arguments in run.sh or the code below for your need, and run it.

CUDA_VISIBLE_DEVICES=0 python -u main.py --input_dir ./sst2 \
                                         --prompt_config_dir ./sst2/prompt_config.json \
                                         --model_class bert \
                                         --model_name_or_path prajjwal1/bert-tiny \
                                         --lr 2e-4
                                         --bs 32 \
                                         --max_seq_length 64 \
                                         --patience 4 \
                                         --accumulation 2 \
                                         --seed 666

In my preliminary experiment with the settings above, the model achieve 0.822 F1 compared to 0.820 without prompt.

Note

Can only be executed after this fix on state_dict()

An example project using OpenPrompt under pytorch-lightning for prompt-based SST2 sentiment analysis model

Related tags

Overview

pl_prompt_sst

Run

Note

Owner

Zhiling Zhang

Pre-Training with Whole Word Masking for Chinese BERT

YACLC - Yet Another Chinese Learner Corpus

A simple implementation of N-gram language model.

ALIbaba's Collection of Encoder-decoders from MinD (Machine IntelligeNce of Damo) Lab

A full spaCy pipeline and models for scientific/biomedical documents.

Beautiful visualizations of how language differs among document types.

DeepPavlov Tutorials

Rethinking the Truly Unsupervised Image-to-Image Translation - Official PyTorch Implementation (ICCV 2021)

A PyTorch-based model pruning toolkit for pre-trained language models

RuCLIP-SB (Russian Contrastive Language–Image Pretraining SWIN-BERT) is a multimodal model for obtaining images and text similarities and rearranging captions and pictures. Unlike other versions of the model we use BERT for text encoder and SWIN transformer for image encoder.

In this project, we compared Spanish BERT and Multilingual BERT in the Sentiment Analysis task.

Must-read papers on improving efficiency for pre-trained language models.

A model library for exploring state-of-the-art deep learning topologies and techniques for optimizing Natural Language Processing neural networks

Transformers implementation for Fall 2021 Clinic

A fast and easy implementation of Transformer with PyTorch.

A BERT-based reverse-dictionary of Korean proverbs

Library for fast text representation and classification.

An extensive UI tool built using new data scraped from BBC News

Guide: Finetune GPT2-XL (1.5 Billion Parameters) and GPT-NEO (2.7 B) on a single 16 GB VRAM V100 Google Cloud instance with Huggingface Transformers using DeepSpeed

A practical and feature-rich paraphrasing framework to augment human intents in text form to build robust NLU models for conversational engines. Created by Prithiviraj Damodaran. Open to pull requests and other forms of collaboration.