RBERT for Relation Extraction task for KLUE
Project Description
- Relation Extraction task is one of the task of Korean Language Understanding Evaluation(KLUE) Benchmark.
- Relation extraction can be defined as multiclass classification task for relationship between subject entity and object entity.
- Classes are such as
no_relation,per:employee_of,org:founded_by... totaling 30 labels. - This repo contains custom fine-tuning method utilizing monologg's R-BERT Implementation.
- Custom punctuations with Pororo NER has been added to the dataset prior to the model's training.
- If you want to refer to the experimentation note such as punctuation method of the entity, please refer to the blog post
Arguments Usage
| Argument | type | Default | Explanation |
|---|---|---|---|
| batch_size | int | 40 | batch size for training and inferece |
| num_folds | int | 5 | number of fold for Stratified KFold |
| num_train_epochs | int | 5 | number of epochs for training |
| loss | str | focalloss | loss function |
| gamma | float | 1.0 | focalloss's gamma value |
| optimizer | str | adamp | optimizer for training |
| scheduler | str | get_cosine_schedule_with_warmup | learning rate scheduler |
| learning_rate | float | 0.00005 | initial learning rate |
| weight_decay | float | 0.01 | Loss function's weight decay, preventing overfit |
| warmup_step | int | 500 | |
| debug | bool | false | debug with CPU device for better error representation |
| dropout_rate | float | 0.1 | |
| save_steps | int | 100 | number of steps for saving the model |
| evaluation_steps | int | 100 | number of step until the evaluation |
| metric_for_best_model | str | eval/loss | the metric for determining which is the best model |
| load_best_model_at_end | bool | True |
References
- monologg's R-BERT Implementation in Pytorch
- Enriching Pre-trained Language Model with Entity Information for Relation Classification
- KLUE: Korean Language Understanding Evaluation
- KLUE RE (Relation Extraction) Dataset / CC BY-SA
- Pororo's NER Documentation
Authorship
Hardware
GPU : Tesla V100 32GB
409 Jan 06, 2023
27 Nov 20, 2022
40 Dec 20, 2022
1 Dec 17, 2021
3 May 29, 2022
119 Oct 08, 2022
1 Dec 17, 2021
6 Dec 20, 2022
4 Mar 20, 2022
11 Nov 17, 2022
1 Jan 06, 2022
60 Dec 25, 2022
156 Dec 21, 2022
32 Dec 29, 2022
1 Nov 18, 2021
94 Oct 26, 2022
77.2k Jan 03, 2023
94 Dec 08, 2022
1 Oct 31, 2021
13 Nov 24, 2022