On Wiki103 classification, I only get two modes of results:

1. solve the task, getting ~99+% train and dev accuracy
2. overfit to train, getting ~92% train, 58% dev accuracy Over 5-6 runs, I always get a result in one of these two modes. Are we doing anything weird in sampling that would make this too easy (mode 1) or too hard (mode 2)?

https://github.com/nyu-mll/jiant/issues/446

This is tested on BERT models and also GloVE embeddings, with all the major types of tasks.

Below is the table for BERT models.

| Task | Batch size with 1 GPU | Batch size with 2 GPUs | Time per 100 steps seconds (1 GPU) | Time per steps seconds (2 GPUs)| | ------------- | ------------- | ------------- | ------------- | ------------- | | ReCoRDl | 11 | 22 | 167.55 | 187.90 | | BoolQ | 14 | 28 | 122.2 | 146.6 |

These results were on a machine with 2 P40s.

Adding Sentence Order Prediction Task for ALBERT What this version of MLM supports: ALBERT embedder. Below are the runs for MLM + SOP + intermediate task (which is how we intend to use SOP) The results (on ALBERT) are: | Experiment description | Intermediate Task Performance | SOP performance| | ------------- |:-------------:| :-------------:| | SST + SOP | 0.85 | 0.95 | | QQP + SOP | 0.95 | 0.94 |

Hi, As always, thank you for this amazing contribution.

I am taking prof.bowman's class and attempting to run BioBERT https://github.com/dmis-lab/biobert-pytorch and SciBERT https://github.com/allenai/scibert on jiant. One of our objectives for the course project is to run BioBERT and SciBERT on common NLU tasks.

As far as I understand, it should be possible to add any Transformers encoder model to jiant, but both of those models will probably require a bit of code?

I am sketching out what I'd have to do in jiant vs. plain transformers. Will using jiant create more overheads (by trying to support those models) than just following the standard fine-tuning processes in transformers?

Any suggestions and pointers will be helpful. Thank you!

This should be considered a work-in-progress, not a complete PR. It passes tests, but might break some untested things, and there are a few things I'll flag for review.

Because I'm still getting to know jiant, I chose running the tutorial experiment using a pip-installed package as my goal. This changeset allows that (without requiring conda or manual dependency installation). It doesn't directly address setting up an environment or downloading data using the pip package. To try it yourself, you can:

# create a new python virtual environment (conda not required)
# set up environment variables per tutorial, including pointing to an existing data directory, e.g.
export JIANT_PROJECT_PREFIX=~/jiant/tutorial
export JIANT_DATA_DIR=~/jiant/data

# copy default.conf from repo

# copy tutorial config file content from https://github.com/nyu-mll/jiant/blob/master/tutorials/setup_tutorial.md to tutorial.conf

# install package
pip install --upgrade --index-url https://test.pypi.org/simple/ jiant==1.1.0a0

# run experiment
python -m jiant --config_file tutorial.conf

Some items worth reviewing:

• A maintainer should check the metadata in setup.py - I took a shot at it...
• Config was moved to inside the package, per setuptools recommendation, but this could break things. I only tried the tutorial and the test suite.
• Assuming a distributed package should include only what is required to run typical tasks, I only added package dependencies necessary to run the tutorial. Are there other tasks/tests that should be considered?
• This doesn't expose any shell or python scripts as CLI commands, but we might want some for downloading data or generating a local copy of default.conf, for example.
• I moved main.main() into jiant.__main__ for convenience of running python -m jiant, and to keep necessary code inside the jiant package. I kept main.py for backward compatibility, and there's some duplicated code, but not too much.
• tests and other root-level packages were not included in the package. Does this satisfy the need for a pip package without the other packages? If no, it might make sense to reorganize some directories.

Re: #571

Starting with a basic implementation of QA-SRL, seeking feedback.

• There is quite a lot of repeated/branching input sets in QA-SRL. Each sentence has multiple "verbs", each verb corresponds to multiple questions, each question has multiple answers from different annotators, and each answer contains potentially more than one span. Currently I'm doing a naive implementation of enumerating all of these. For a sense of scale, there are about 44k sentences in the training set, but about 600k+ total training examples when expanded.
• The QA-SRL / Large-Scale QA-SRL papers did not talk too much about scoring, and where they did I'm not sure if it works best for pretraining. I'm currently implementing the following:
• Training loss: I am currently following the BERT formulation for SQuAD of straightforwardly predicting the span start and ends via a softmax over the number of tokens in the input, with a cross-entropy loss. No correction for ensuring start < end.
• Evaluation: I am currently using the SQuAD-style macro-F1, which treats the predicted / gold tokens as individual observations, computing the F1 for each example, and then averaging across examples.

07/27 03:48:08 PM: Update 4001: task ccg, batch 1 (4001): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:48:18 PM: Update 4115: task ccg, batch 115 (4115): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:48:32 PM: Update 4197: task ccg, batch 197 (4197): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:48:42 PM: Update 4310: task ccg, batch 310 (4310): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:48:52 PM: Update 4428: task ccg, batch 428 (4428): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:49:05 PM: Update 4510: task ccg, batch 510 (4510): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:49:15 PM: Update 4624: task ccg, batch 624 (4624): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:49:25 PM: Update 4743: task ccg, batch 743 (4743): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:49:35 PM: Update 4785: task ccg, batch 785 (4785): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:49:46 PM: Update 4904: task ccg, batch 904 (4904): accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:49:54 PM: ***** Pass 5000 / Epoch 5 ***** 07/27 03:49:54 PM: ccg: trained on 1000 batches, 0.842 epochs 07/27 03:49:54 PM: Validating... 07/27 03:49:56 PM: Batch 19/157: accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:50:06 PM: Batch 147/157: accuracy: 0.0000, ccg_loss: 0.0000 || 07/27 03:50:06 PM: Best model found for ccg. 07/27 03:50:06 PM: Best model found for micro. 07/27 03:50:06 PM: Best model found for macro. 07/27 03:50:06 PM: Advancing scheduler. 07/27 03:50:06 PM: Best macro_avg: 0.000 07/27 03:50:06 PM: # bad epochs: 4 07/27 03:50:06 PM: Statistic: ccg_loss 07/27 03:50:06 PM: training: 0.000000 07/27 03:50:06 PM: validation: 0.000000 07/27 03:50:06 PM: Statistic: macro_avg 07/27 03:50:06 PM: validation: 0.000000 07/27 03:50:06 PM: Statistic: micro_avg 07/27 03:50:06 PM: validation: 0.000000 07/27 03:50:06 PM: Statistic: ccg_accuracy 07/27 03:50:06 PM: training: 0.000000 07/27 03:50:06 PM: validation: 0.000000 07/27 03:50:06 PM: global_lr: 0.000100 07/27 03:50:17 PM: Saved files to /misc/vlgscratch4/BowmanGroup/sbowman/exp//final/ccg-noelmo

Relevant to @dipanjand, @iftenney.

We need to still be able to write legal GLUE test set predictions, which means preserving the current behavior. We also need to make it easier to write dev set predictions and to write predictions files that include the input sentences.

One run has been on QQP Test for 2.5 hours without signs of progress. GPU usage is non-zero but low. This seems to have changed since #121.

@Jan21 @iftenney, any guesses? Did you verify that QQP test works?

Dev also appears to be quite slow, but I don't have numbers yet.

• Replacing old GPT implement with the one from huggingface pytorch transformers
• Add GPT2, Transformer-XL, XLM to pytorch transformer interface
• Refactor pytorch transformer interface a little bit to reduce duplicated / outdated code & comments

Config: defaults + train_for_eval = 0, val_interval = 9000, scaling = none, train_tasks = glue, eval_tasks = glue Restart with weighting: proportional * 2, power_0.75 * 3

Issue 1: cola score is low (around 5), in four out of five experiments Issue 2: glue score has large variation (1~2 points) between same experiments

Results and logs can be found in: GLUE Dev Results -> Experiment: MTL Mixing -> Row 273~277

Logs: /nfs/jsalt/exp/shuning-worker34/sampling_master/proportional_glue/log.log /nfs/jsalt/exp/shuning-worker31/sampling_master3/proportional_glue/log.log /nfs/jsalt/exp/shuning-worker38/sampling_master2/power_0.75_glue/log.log /nfs/jsalt/exp/shuning-worker32/sampling_master3/power_0.75_glue/log.log /nfs/jsalt/exp/shuning-worker32/sampling_master4/power_0.75_glue/log.log

Hi, I am doing experiments on different portions of the test dataset. Now it is possible to test the whole test set using your functions. I wanted to know if it is possible to teat on blocks of test set?

Describe the bug We are trying to reproduce the experiments for the paper Comparing Test Sets with Item Response Theory, and we were following the workflow mentioned in this README for the preprocessing of datasets, training models, generating responses, etc. We have successfully trained most of the models at our end, but now facing an issue while getting predictions from those models.

To Reproduce

1. jiant version: 2.1.4
2. Environment: A100 40GB

We ran the call_predict script. We did not find any sb_predict_results.sbatch file, and changed L21 to call the run_predict_results.sbatch file instead. For example, the exact command we ran for copa and for bert-base-cased model is:

python jiant/irt_scripts/call_predict.py bert-base-cased copa $(pwd)

This produced the following error:

pre args =  Namespace(ZZoverrides=['model_path'], ZZsrc=None)

Traceback (most recent call last):

  File "/dccstor/irlirteval/irt/jiant/jiant/proj/main/runscript.py", line 256, in <module>

    main()

  File "/dccstor/irlirteval/irt/jiant/jiant/proj/main/runscript.py", line 246, in main

    run_loop(RunConfiguration.default_run_cli(cl_args=cl_args))

  File "/dccstor/irlirteval/irt/jiant/jiant/utils/zconf/core.py", line 237, in default_run_cli

    return cls.run_cli_json_prepend(cl_args=cl_args, prog=prog, description=description)

  File "/dccstor/irlirteval/irt/jiant/jiant/utils/zconf/core.py", line 174, in run_cli_json_prepend

    result = cls.run_from_parser_json_prepend(parser=parser, cl_args=cl_args)

  File "/dccstor/irlirteval/irt/jiant/jiant/utils/zconf/core.py", line 217, in run_from_parser_json_prepend

    assert pre_args.ZZoverrides is None

AssertionError

An attempt to fix We tried to fix the zzoverrides error by modifying the run_predict_results.sbatch as follows:

python $JIANT_PATH/proj/main/runscript.py run   \
    --hf_pretrained_model_name_or_path ${model} \
    --model_path ${MODELS_DIR}/${SHORT_MODEL_NAME}/model/model.p \
    --model_config_path ${MODELS_DIR}/${SHORT_MODEL_NAME}/model/config.json \
    --jiant_task_container_config_path ${BASE_PATH}/experiments/run_config_dir/taskmaster/${model}/${task}_${config_no}/${task}.json  \
    --model_load_mode all --model_path ${BASE_PATH}/experiments/output_dir/taskmaster_${model}/${task}/config_${config_no}/${model_path}  \
    --output_dir  ${BASE_PATH}/experiments/predict_files/${model}/${task}_config_${config_no}_${model_path} --ZZoverrides model_path model_config_path hf_pretrained_model_name_or_path --write_val_preds \
    --ZZsrc ${MODELS_DIR}/${SHORT_MODEL_NAME}/config.json --do_val --do_save --force_overwrite

instead of this code block. This time it produced empty predictions.

Expected behavior Prediction files (.p) in experiments/predict_files/ folder, so that we could run the post-processing command python jiant/irt_scripts/postprocess_predictions.py $(pwd).

Describe the bug

When one uses run_simple() with different models of the same type roberta-base and roberta-large the run crashes because the code assumes they are the same model because weights are saved under hf_config.model_type (instead of args.hf_pretrained_model_name_or_path.). As such, the code tries to load incompatible weights and crashes.

To Reproduce

1. Install jiant
2. Run the simple example in README
3. Change the model in the sample from 'roberta-basetoroberta-large`

Expected behavior One should be able to run run_simple() with different models of the same type.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Solution: The hf_config.model_type should be used for caching tokenizer / tasks. The args.hf_pretrained_model_name_or_path for the weights.

Describe the bug The downloaded files for the CB task have swapped labels. This introduces a nasty silent bug because all the metrics calculated using these data seem correct but the fine-tuned model is actually predicting nonsense.

contradiction is mapped to entailment entailment to neutral neutral to contradiction

To Reproduce I used Jiant 2.2.0 available via pip3 install jiant.

import json
from collections import Counter
import jiant.scripts.download_data.runscript as downloader

# Download Jiant CB
downloader.download_data(["cb"], "tasks")
with open("tasks/data/cb/train.jsonl") as f:
    jiant_freqs = Counter([json.loads(line)["label"] for line in f.readlines()])
print(jiant_freqs.most_common())

>>> [('entailment', 119), ('neutral', 115), ('contradiction', 16)]

# Download official CB from SuperGLUE
!wget "https://dl.fbaipublicfiles.com/glue/superglue/data/v2/CB.zip"
!unzip CB.zip
with open("CB/train.jsonl") as f:
    official_freqs = Counter([json.loads(line)["label"] for line in f.readlines()])
print(official_freqs.most_common())

>>> [('contradiction', 119), ('entailment', 115), ('neutral', 16)]

This error is reported with BART, but not with BERT

Describe the bug A clear and concise description of what the bug is.

To Reproduce

1. Tell use which version of jiant you're using
2. Describe the environment where you're using jiant, e.g, "2 P40 GPUs"
3. Provide the experiment config artifact (e.g., defaults.conf)

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.