This release is mainly to address build errors resulting from functionality introduced in 1.7.0, i.e., n-gram blocking on the GPU. Full release notes below:

General Fixes/Improvements

• Protect loading of n-gram blocking on GPU (#4779)
• General lint fixes (#4771)
• Increase CI parallelism (#4702)
• Update DialCrowd to mephisto 2.0.1

Agent Improvements/Fixes

• [BB3] General fixes (#4786, #4789)
• [BB3] Memory usage heuristics (#4770)
• [BB3] README Updates (#4784)
• [DIRECTOR] Added shared embedding option to director model. (#4763)

New Releases

• BlenderBot 3 is now available in ParlAI! (#4709, #4710, #4711, #4712, #4713, #4715, #4716, #4725, #4746, #4747, #4753).
  • Includes the code from the paper, "Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback" (#4714)
  • Includes the code from the paper, "Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trolls"
• The DIRECTOR Project + Agent code is available in ParlAI (#4597, #4602, #4603, #4605, #4607, #4738, #4701)
• [DialCrowd] Incorporate the DialCrowd crowdsourcing toolkit into ParlAI (#4387)

New Features

• The Decoder-Only Transformer agent is now available in ParlAI! (#4329)
• Beam N-Gram blocking is now supported on the GPU (#4633, #4721)
• The model chat mephisto crowdsourced task now supports emojis! (#4666)

Agent Fixes + Improvements

• Updates to Reranker, Pacer (#4469, #4473, #4488)
• Updates to RagAgent, FidAgent, and Search (#4470, #4503, #4631, #4765)
• Updates to HuggingFace agents (#4508)
• Updates to T5Agent (#4505)
• Updates to BertClassifierAgent (#4553)
• Updates to SeeKeR (#4634)
• Updates to TorchAgent, TorchGeneratorAgent,TorchClassifierAgent` (#4654, #4650, #4700, #4727, #4754, #4720)
• Updates to models from the WhoAmI project (#4742)

Bug fixes + Performance Improvements

• Script fixes script (#4054, #4482, #4496, #4572)
• [TorchScript] fixes (#4489, #4499, #4554), add torchscriptable classifier + BPE tokenizer (#4566),
• Train model supports setting seed (#4662)
• ParlaiParser Fix (#4507)
• Various crowdsourcing fixes + enhancements (#4551, #4560, #4577, #4625)
• Fix downloads of TOD models (#4647)
• Test fixes (#4749)

Datasets & Teachers & Mutators

• Added datasets:
  • SPOLIN Dataset (#4540)
  • Friends Dataset (#4568, #4678, #4693)
• Deprecate Auxiliary BST Teachers (#4513)
• Updates to HuggingFace dataset integration (#4516)
• Update LCCC Download Link (#4542)
• MultiWoz V22 Updates (#4565, #4656, #4695, #4752, #4757, #4761, #4762, #4736, #4764)
• Wizard of Internet Updates (#4576)
• SaferDialogues updates (#4623)
• Various mutators bugs (#4730)
• BAD Teacher updates (#4732)
• GoogleSGD updates (#4698)

Logging & Metrics

• Improvements to Weights & Biases integration (#4484, #4548, #4708)
• Include tensorboard logging in eval_model (#4497)
• TimerMetric fixes (#4536)
• Expose precision + recall metrics (#4670)

Developers & Documentation

• Various documentation updates (#4487, #4509, #4609, #4613, #4612, #4624, #4632, #4687, #4688, #4690, #4691, #4686, #4733)
• Various dependency + CI updates (#4481, #4506, #4514, #4483, #4524, #4523, #4522, #4521, #4529, #4532, #4562, #4564, #4590, #4608, #4593, #4615, #4671, #4677, #4759)
• Various linting, syntactic updates (#4684, #4718, #4728, #4731)

New Releases

• SeeKeR project code and models release (#4447, #4458, #4449, #4453, #4448)
• User Simulators and Task Oriented Dialogue support (#4437, #4438, #4188, #4187, #4189, #4186, #4185, #4184, #4183, #4249, #4174, #4181, #4182, #4180, #4178, #4233)
• Per Turn evaluation project code release (#4304, #4323, #4333, #4362)
• SaFeRDialogues dataset and model release (#4229, #4300)
• Am I Me or You project code release (#4239, #4252, #4250)
• K2R Project release(#4251)

New Features

• Updating to Mephisto 1.0 (#4426)
• [TGA]
  • More flexible token metadata logging (#4169, #4427)
  • Record avg generation length (#4295)
  • change tga default to not sort (#4138)
• Add a small example Flask server (#4433)
• Add support for WorldLoggers in training (#4369)
• [T5] Support Distributed Training (#4434)
• [HuggingFace] Add support for any GPT-2 model hosted in Huggingface (#4360), and ranking (#4326)
• [Crowdsourcing]
  • [ACUTE-Eval] Add support for knowledgeable question (#4416), interestingness question (#4113)
  • [ACUTE-Eval] Record start and end times (#4208)
  • [Model chat] allow multiple final ratings (#4276)
  • [Model chat] allow spacing out of annotation buckets (#4275)
  • Allow Use of external database in crowdsourcing code (#4272)
  • Allow specification of blueprints from command line (#4254)
  • Unify turn annotation tasks (model chat and turn annotations static tasks) (#4162)
  • [ACUTE-Eval] Save worker name in Acute-Eval analysis script (#4126)
• [RAG/FiD/BB2]
  • [All] Incremental Decoding (#4088)
  • [BB2] Allow hybrid mode (skip search when needed) (#4221)
  • [FiD] Add specialized chunking to search engine retrievers (#4227)
  • [BB2] Support for gold docs training (#4163)
  • [FiD] Gold retrieved documents FiD Agent. (#4123)
  • Export memories to the observation output (#4040)
• [Style-Controlled Generation] OSS Second Classifier (#4380)
• [Re-ranker] Support for a classifier re-ranker agent (#4291)
• [TCA] return candidates (#4286)
• Curated response generators (#4197)
• [Chat Services] add option for host specification (#4335)

Bug fixes

• [TCA] Various updates and fixes (#4287, #4406, #4270)
• [BlenderBot2] Various updates and fixes (#4379, #4428, #4419, #4331, #4377, #4366, #4289, #4259, #4238, #4212, #4198, #4156)
• [TorchAgent] Fix an issue with warmup updates not working properly (introduced after 1.5.1) (#4384, #4242, #4196)
• [RAG/FiD] various updates and fixes (#4436, #4361, #4389, #4207, #4199, #4146)
• [GPT2] misc. Fixes (#4395)
• [Train Model] Fixes train_model worldlogging for multitask with mutators. (#4414)
• [Crowdsourcing] various updates and fixes (#4413, #4273, #4274, #4089)
• [Re-ranker] Bubble up all batch fields (#4296)
• Fix clobbering issues with WandB & Tensorboard requeueing (#4175, #4093)
• Fix interactive web (#4140)
• Several misc. fixes (#4459, #4403, #4386, #4439, #4425, #4021, #4218, #4164)

Datasets & Teachers & Mutators

• Relicense several parlai datasets as commercially friendly (#4269, #4213, #4126)
• LLLC, a large chinese dataset (#4325)
• Casino dataset (#4129)
• Upgrade internals of several teachers: WoW (#4284), Empathetic Dialogues (#4405), Natural Questions (#4205)
• WoW and WoI mutators (#4418, #4204, #4168, #4124, #4125, #4122, #4114)
• Speed up the json teachers (#4404)
• XPersona Dataset (#4314)
• ConversationTeacher parent class is now ParlAIDialogTeacher (#4256)
• [WizInt] Additional knowledge-related eval metrics (#4193); turn dicts to Messages (#4144)

Developers

• Various documentation improvements (#4423, #4352, #4375, #4074, #4172, #4423, #4100, #4385. #4349, #4298, #4297, #4217, #4173, #4133, #4120, #4028, #4104, #4102)
• Various test improvements and fixes (#4442, #4375, #4311, #4271, #4435, #4343, #4236, #4261, #4223, #4224, #4096, #4315, #4253, #4190)
• Various dependency bumps (#4464, #4463, #4420, #4351, #4348, #4346, #4337, #4317, #4302, #4299, #4301) Other QOL Improvements (#4095)

v1.5.1

Project Releases

Hi, my name is Martha - New reduced-bias dialogue models! (#3981)

Minor features

• Allow the style classifier to save output probabilities (#4026)
• Add gpu option to torchscript BART models (#3979)

Crowdsourcing

• Upgrade to Mephisto 0.4.0 (#3982, #4043)
• Save model chat data in the Mephisto database (#4005)
• Add Last Turn Annotation Only option to turn annotations static task (#3436)
• Crowdsourcing data compiler class for using Mephisto abstractions (#4029, #4034)
• Force model chats starting with "Hi!" to use BST-style context (#4004)

Datasets

• [LIGHT] Jericho World dataset (#3957)

Bugfixes

• [RAG] Fix ReGReT Cuda Issue (#4022)
• [BlenderBot2] Handle distributed (#4023)
• [mutators] Prevent name collisions in mutators (#4006)
• [crowdsourcing] Fix model chat frame height by (#4030)

Other

• Various error message improvements (#3987, #4018, #4007)
• Various typos and documentation clarifications (#4053, #4052, #4008, #4002, #4064)
• Various project-specific small additions and mutators (#3997, #3996, #3999, #4000, #3966, #4062)

Developer changes

• [Modular] Update Transformer Layer __init__s by @klshuster in #4061
• Migrate to the main branch by @stephenroller (#3998, #4010)
• Allow more subclassing of self-chat world (#3955)
• Moving Wizard of Internet task to crowdsourcing/projects directory. by @mojtaba-komeili in #3978
• Improve internal facebook compatibility (#3964, #3994, #4032, #3991)
• Various dependency bumps (#3995, #4072, #4017, #4077)

v1.5.0

Major Features

Model Cards have now been added to ParlAI. We support automatically generating cards for different models. See an example of BlenderBot2's model card. (#3857, #3865, #3899, #3884, #3915, #3965, #3860, #3863)

Minor Features

• Add support for an extra final evaluation in training scripts with custom opts (#3883)
• Add --checkpoint-activations to lower memory usage of training transformers (#3864)

Crowdsourcing

• Open source the Wizard of Internet crowdsourcing tooling (#3924)
• Open source the Personal Knowledge crowdsourcing tooling (#3945)
• More customization for the [Static Turn Annotations] task. (#3926)
• More flexibility in model-chat analysis code (#3844, #3958, #3935)

Bug Fixes

• [world logs] Fix a bug where dynamic batching didn't use episode boundaries in world logs (#3867)
• [self chat] Fix the order of openers so output is deterministic (#3923)
• [parser] Fix bug where parse_kwargs couldn't handle mutator args (#3900)
• [parser] Fix bug where nargs=+ wasn't working with parse_kwargs (#3930)
• [teachers] Fixed an issue where some preprocessing was missing from labels field (#3874)
• [blenderbot2] Fix a crash when not using internet search (#3950)
• [regret] Fix a regression in Regret (#3934)
• [rag] Fix a bug when receiving NaN scores

Teachers

• Auxillary data added in the Wizard of Internet Task (#3897)

Documentation changes

• Updates/improvements to some project documentations in Hallucination/RAG/etc (#3869, #3888, #3917, #3873)
• Remove a false statement in chat services tutorial
• Update a link to datasets in Contradiction (#3878)
• Repo-wide spelling corrections (#3894, #3960)

Developer changes

• Minor refactors (#3892)
• CI fixes and improvements (#3954, #3837, #3889)
• Small extensibilities to the torchscript functionality (#3851)
• Enable self_chat to seed messages from tasks in parlai_internal (#3852)
• Avoid exception in core/agents.py when arg is missing from dict (#3893)
• Allow customization of the AcceptabilityChecker (#3846)

Includes a fix to make sure that BlenderBot2 can be used with a pip install (#3848)

Also includes a teacher for training a Wikipedia Page Title Generator from the Wizard of Wikipedia Dataset (#3845)

v1.4.0 Changelog

Major Features/Paper Releases

BlenderBot 2.0 Models, Code, Datasets

This release of ParlAI includes all the code, models, and datasets used to build BlenderBot 2.0 - please see the project page for more information, including how to access and use the models themselves (#3790, #3793, #3794, #3795, #3796, #3797, #3798, #3801, #3802, #3805, #3803, #3815, #3817)

Internet-Augmented Dialogue Generation

Build and release a crowd-sourced dataset of humans searching the internet and talking in depth about a vast array of topics; search-engine-augmented dialogue models are trained on the dataset, and are shown to be more knowledgeable than their non-search-augmented counterparts. (#3792, #3800, #3814)

Multi-Session Chat

Build and release a dataset of multi-session chats for the purpose of studying long-term open-domain conversation. Models trained on the dataset prove to perform better at recalling information from the distant past in the chats. (#3791, #3799, #3814)

Safety Benchmark Tests

With the release of Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling, ParlAI now supports running safety unit tests for conversational models. Check out the project page to see how to run the safety benchmarks. (#3767, #3768, #3769, #3784)

Convenience Functions for Agents

With a recent update to ParlAI's agent API, you can now bypass the act/observe loop for interacting with agents. (#3775)

In [1]: from parlai.core.agents import create_agent

In [2]: from parlai.core.params import ParlaiParser

In [3]: opt = ParlaiParser(True, True).parse_args(['--model_file', 'zoo:blender/blender_90M/model'])

In [4]: agent = create_agent(opt)

In [5]: agent.respond("Hi! How's it going?")
Out[5]: "it ' s going well . i ' m getting ready to go to work . how about you ?"

Minor Features

[Metrics] AUC Metrics added to parlai eval_model (#3751)

Bug Fixes

[Crowdsourcing] Crowdsourcing Fixes concerning Left Pane Text and Model Chat (#3773, #3780, #3789, #3831) [RAG] Fix some issues with RAG/FiD agents (#3787, #3818, #3834, #3833, #3836) Fix JSON serialization in interactive_web.py script (#3821) Fix CI test issues (#3825, #3823) Misc. README typos (#3807, #3839, #3841)

Developer Changes

Stripping before processing Conversations to allow leniency (#3772) [ACUTE-Eval] Dev improvements to running ACUTE-Eval (#3781, #3782, #3783) Allow self-chat in nested folders (#3785) Allow opt presets from parlai_internal folders (#3819)

v1.3.0 Changelog

Major Features

Fully Sharded Data Parallel

Implementation of DeepSpeed/FairScale's Zero2 sharding. Improves training speed and reduces memory usage over vanilla DistributedDataParallel. Switch to the new mode with --ddp-backend zero2 to see free improvements in your training! (#3740)

Swappable Transformer Components

We've added support for overriding internal components within Transformers. It is now easy to swap only an attention module, or customize your layers, without having to fully override all classes. (#3567, #3703, #3708, #3715, #3638)

ChunkTeacher no longer requires num_examples/num_episodes to be correct

For as long as we've had ChunkTeacher, the value of num_examples/num_episodes must be exactly correct, or your training would hang. Furthermore, that calculation would need to be done outside of ParlAI. We've relaxed this restriction: these methods can now return arbitrary values, and you will correctly iterate through all of your data. However, using the wrong value of num_examples can cause the "epoch" counter (used in parlai train) to be wrong relative to your dataset. (#3681, #3745)

Eliminate dummy batches and init_cuda_buffer

You are no longer required to implement dummy batches in your Generator agents, when using custom batch formats. Additionally, you will no longer see a dummy batch as the first batch when debugging. Instead, the first batch your agent sees will be reserved as the future dummy batch. (#3732, #3744)

Paper Releases

Reducing Hallucination

Exploratory architectures that add retrieval mechanisms to dialogue models, reducing hallucination while maintaining conversational ability. (#3611, #3657, #3693, #3688, #3668)

Hash Layers & Ladder Transformers

More Parameters or More Compute? Answer: Both! Two new methods that explore this question: Hash Layers for more parameters, and Staircase Attention for more power per parameter. (#3697, #3699, #3700, #3746, #3747)

Minor Features

• [TGA] Substantial speedups during generation on GPUs (#3730, #3729, #3669)
• [Datasets] Add GLUE teachers, and support for HuggingFace datasets (#3570, #3624)
• [Datasets] [Safety] Release the Non Adversarial Data (#3684)
• [TA] Support temp history via special field in observation (#3617)
• [TGA] Allow setting prefix tokens (#3760)
• [TCA] Classifier on generator for TGA (#3716)
• [ChunkTeacher] Remove exception for specifying non-streaming data (#3653)
• [Transformer] Better initiaization for segment embeddings (#3680)
• [Message] Add a new json_safe_payload method for serialization (#3643, #3726, #3686)
• [JIT] Support special tokens in torchscript module. (#3644)
• [JIT] Fix a parsing error with parlai torchscript in Python 3.8 (#3641)
• [ACUTE] Support randomize_conversations (#3636, #3642)

Bugfixes

• [train] Fix bugs with loading validation impatience. (#3713)
• [train] Fix LR scheduler cooldown (#3719)
• [train] Dynamic Batching doesn't choke with really small datasets (#3721)
• [Logging] Fix a bug with world logging and multitasking (#3718)
• [Mutators] Ensure mutations do not persist across epochs (#3649)
• [BART] Do not add start/end tokens multiple times (#3714)
• [TCA] weighted_f1 no longer assumes binary classification (#3728)
• [Safety] Fix a Static Task bug and Safety README (#3612)
• [logging] Fix an issue where --loglevel debug was ignored (#3658)
• [Tensorboard] Fix an exception in some versions of Tensorboard (#3637)
• [vacuum] Add support for PathManager in vacuum (#3635)
• [Crowdsourcing] Slightly improve the analysis script to make it more robust (#3683, #3629)
• Various locations where the change to is_padding caused issues (#3704, #3634, #3674)
• Various typos/lint (#3621, #3622, #3646)

Developer changes

• Helper functions for building deterministic data splits (#3676)
• Teacher URL updates (#3749, #3627, #3678)
• CI bugfixes & version bumps (#3754, #3724, #3672, #3652, #3710, #3628, #3452, #3720)
• Documentation updates (#3748, #3690, #3742, #3671)
• Mutators and Scripts support for parlai_internal (#3623, #3625)
• [Crowdsourcing] Small refactor in Model-Chat

This Saturday marks the 4 year anniversary since the initial release of ParlAI. I'd like to offer my sincere gratitude to our users, our contributors, and all of the core development team. ParlAI wouldn't be what it is without all of you. [email protected]

Major new features

Background Preprocessing

Improve your training speeds by 1.25x-5.0x by switching from --num-workers 0 to --num-workers N. See our Speeding up training docs for details. (#3527, #3586, #3575, #3533, #3389)

(Beta) Support for torch.jit Deploy faster models by exporting models with TorchScript. Currently limited to BART models only. (#3459)

Support for T5 We now have agents for Google's T5 models (#3519)

Opt Presets Opt presets. Easily use prepackaged opt files as shorthand for long command line arguments (#3564)

Log/validate/stop based on number of steps Get up to a 10% speedup of distributed training by switching from -vtim or -veps to -vstep (#3379, #3555)

Backwards-incompatible changes

• DictionaryAgent.text2vec now requires input to be a string (#3472)
• A number of older projects have been archived: DrQA (#3559), Controllable Dialogue (#3557), and Self-Feeding Chatbot (#3557).

Minor improvements

• Performance speedup in generation using Transformer Generators (#3550)
• Improvements to the Transformer API, making Transformer models more easily extensible. More to come soon. (#3486, #3545, #3466, #3501)
• Various performance improvements when loading ParlAI or performing some activities (#3544, #3482)
• Metrics:
  • New truncation metrics show you how much context/label you're losing (#3458, #3508)
  • Additional metrics in the Wizard of Wikipedia teacher (#3566, #3592, #3503)
  • New token_em metric, an equivalent to accuracy with --skip-generation true (#3497)
• Self-chat can now use seed messages (#3580)
• New "normalized" ConvAI2 teachers for the non-standard variants (#3509)
• Update FusedAdam support to use FairScale (#3522)
• Add --wandb-entity flag to the logger (#3562)
• Tensorboard now provides nicer names of metrics (#3534)

Bugfixes

• [core] Fix a bug when resuming with the cosine LR scheduler after preemption (#3599)
• [core] Improve robustness to serialization of Observations (#3591)
• [core] ParlaiDialogTeacher now parses the rewards field as a number (#3517)
• [core] Fix recently introduced ChunkTeacher bugs (#3549, #3542, #3543)
• [core] Minor FP16 fixes when converting old checkpoints (#3514)
• [core] Fix annoying ambiguity issues with commandline parsing in Python 3.8 (#3598)
• [core] Fix a rare situation in case a dictionary contained tokens with leading whitespace (#3613)
• [mutators] Fix a bug with the flatten mutator providing the wrong history (#3578, #3584)
• [metrics] Fix a bug with computation of fairseq metrics (#3518)
• [task] Fix a bug with Wizard of Wikipedia teacher causing some data to be omitted (#3585)
• [task] Fix a crash in Wizard of Wikipedia end2end agent when given zero documents (#3602)
• [task] Update a dead dataset to a new link (#3520)
• [task] Fix an issue with CCPE (#3487)
• [tga] Fix a case where TGA used as a ranking model could crash (#3541)
• [agent] Fix a crash in BertDictionaryAgent (#3560)
• [other] Various rare issues (#3505) and quality improvements (#3496)

Crowdsourcing improvements

• Add new option to avoid randomizing pairs in ACUTE-Eval (#3528)
• ACUTE-Eval provides additional warnings when options are not set to recommended values (#3536, #3524)

New Datasets

• MetaLWoZ (#3583)
• CMU_DoG (#3593, #3615)

Doc improvements

• New tutorial: How to write a PR contributing (moving to a fork) tutorial (#3490)
• New reference manual of standard ParlAI metrics (#3498)
• Fix a bug keeping some tasks from being listed in our docs (#3495)
• Various minor doc changes (#3513, #3561, #3596)

Developer improvements

• Update to pytorch 1.8. No changes necessary. (#3607)
• Various reliability improvements and speed ups to our CI and tests (#3603, #3597, #3604, #3606, #3605, #3587, #3588, #3590, #3539, #3535, #3526, #3521)
• Various dependency bumps (#3553, #3537, #3515)
• Switched logging.warn to logging.warning (#3569)
• TorchAgent.history prettier debug printing (#3510)
• Small other improvements (#3506)

Bug fix release

Fix a bug introduced in v1.1.0 caused by the removal of APEX, which increased the numerical instability of training with fp16. See #3532 for details.

Breaking changes

• Removed dependence on APEX. --fp16-impl safe is the new standard (#3469)
• Change DialoGPT default to medium as that's the best results and most common setup (#3401)

Major new features

• Mutators are a new way to perform transformations on arbitrary datasets (#3448)
• Support for WandB (#3417)
• display_model now displays teacher-metrics, and world loggers store them (#3388, #3419)

Minor new features

• Make convenience script for dumping out chat to conversations format (#3354)
• Allow a variable number of transformer heads in distillation, and improved loss normalization (#3424, #3447, #3430)
• Better error messages when incorrectly providing a model zoo file (#3445)
• Add --no-backup option to vacuum. (#3431)
• Release BST human annotations (#3405)

Crowdsourcing changes

• Abstract class for compiling annotation results (#3422, #3423, #3464)
• Improvements to output format for some tasks (#3400, #3425)
• Configuration option for setting soft blocks (#3435)

Bug fixes

• [security] RCE fixed in chat services (#3402, #3429)
• Speedups when using conversation openers in self-chat (#3461)
• Speedups in metrics accumulation (#3474)
• Fix an error bounds error when manually applying vec2txt with --dict-tokenizer bytelevelbpe (#3437)
• Add pathmanager support to BERT agent (#3439)
• Fixes to Natural Questions (#3440, #3438)
• Remove a warning in --fp16-impl mem_efficient (#3443)
• Fix an issue with windows compatibility (#3428)
• Update data urls for Taskmaster-1 (-t taskmaster) (#3475)
• Fix a bug with terminal chat where the socket would close early (#3457)

Developer features

• New abstract method for ranking candidates in TorchGeneratorAgent (#3455)
• Improved type annotations and code cleanup (#3413, #3462, #3420)
• Various documentation immprovements (#3398, #3399, #3349, #3384, #3385, #3427, #3446, #3441)
• Test fixes (#3442, #3467)

This marks a major release for ParlAI worthy of celebration. Thank you to the team, our numerous external contributors, and all of our users.

Major changes

MTurk is dead, long live Crowdsourcing

As the primary impetus for the release of v1.0.0, we have officially removed parlai.mturk from our repo, fully migrating to the newer, Mephisto-based parlai.crowdsourcing. If you have any questions about upgrading, please make sure to check the new tutorial and feel free to ask questions in our new Discussions forum

New discussions forum

We've enabled GitHub Discussions as a new place for individuals to ask more long-lived questions. We still encourage using issues for bug reports, but many "How do I...?" questions are better for Discussions now.

Distilled models

We have released several distilled BlenderBot models (#3313, #3314), as well as code for distilling your own models. See project page.

DECODE model releases

New project on addressing contradictions in dialogue models (#3324, #3325, #3327, #3334, #3309). See project page for more information.

Backwards incompatible changes

• parlai.mturk has been removed in favor of parlai.crowdsourcing (#3362, #3371)
• The API for add_cmdline_args has changed. Please see the PR for information on upgrading. This enables things like defaults that depend on other values, or arguments that depend on other arguments. (#3328)
• Improvements to world logger arguments (#3380)

New features in core

• New distinct metrics (including intra-distinct and inter-distinct). (#3347)
• 40% Speedups in --model-parallel true generations and training (#3326)
• Updated README, autodoc descriptions, PR templates (#3359, #3342, #3345, #3322))
• [developer] Many more task regression tests (#3240)

New features in crowdsourcing

• General improvements to ACUTE-Eval implementation, docs, and introduction of analysis framework (#3374, #3333, #3344, #3305, #3323)
• Conversion of turn annotations to Mephisto (#3262, #3356)
• Mephisto QA data collection (#3318, #3330)
• Mephisto image-chat task (#3320)
• Documentation improvements (#3319, #3373)
• Fixing flaky tests (#3363, #3372, #3321, #3355)

Smaller bugfixes

• Allow differing arguments for dynamic batching in eval/train (#3382)
• fix: deprecated argument when calling transformers>4.0.0 (#3393)
• Small bugs with crowdsourcing (#3346, #3386, #3387)
• Fix a misparse of -o (#3347)
• Fix bert models in distributed. (#3351)
• Suppress PermissionErrors in windows deletions. (#3340)
• Fix for Interactive Web with TF-IDF models (#3381)
• Fix a small bug in Wizard of Wikipedia task (#3100)
• Fix minor idx not-existing issue in world_logging (#3317)

Backwards compatibility breakages:

• TorchClassifierAgent changed use_reply default from 'label' to 'none' (#3128)
• [BERT] Deprecate Adam with Decay (#3235)

Major features

• [safety] Open source Sensitive Topics classifier and data (#3253, #3259, #3260, #3258, #3230, #3227, #3252)
• Support for model distillation (#3274)
• [docs] Link to request Multi-Modal BlenderBot weights (#3281)

Minor features

• DialogTeacher provides verbose warning if opt['datafile'] is not set (#3140)
• [bpe] Support BPE dropout (#3232)
• Support special tokens in non-HF BPE dictionaries. (#3213)
• Unify verbose & display_verbose, enable verbose during interactive (#3205, #3248)
• BART Supports incremental decoding (#3231)
• Use more modern metrics reporting with data_stats. (#2854)
• Ease-of-use improvements to wrapper teacher (#3247)
• Add classifier for style-controlled generation (#2829)
• Add support for frozen encoders (only train head) in TorchClassifierAgent (#3215)

Mephisto Migrations & Upgrades

• [CrowdSourcing] Turn annotation edits (#3234)
• Fast ACUTE in Mephisto (#3297)
• Mephisto API upgrades (#3275, #3285)
• Remove two old MTurk tasks (#3225)
• Port static turn annotations to Hydra (#3149)
• Add command for viewing all Hydra parameters (#3210)
• Analysis script for static turn annotations (#3263)
• Make the log rate an adjustable parameter (#3189)

Bug Fixes:

• Fix argparse issues in python 3.8 (#3284)
• Torch 1.8 Compatibility (#3228)
• Fix dialogpt dual usage of END_IDX (#3256), broken by adding support by adding batchsize > 1
• Fix minor bugs with GPT2 (#3261, #3207, #3208)
• Fix an issue where ChunkTeacher would sometimes repeat examples (#3244, #3301)
• make dir at start of dict.save (#3250)
• Fix a bug in distributed sync when metrics dicts become large (#3291)
• [BART] Some smaller bug fixes (#3272, #3236)
• Data parallel is now false in Offensive Language Classifier (#3212)
• Fix a bug with LR scheduler loading from disk in TCA (#3229)

Developer changes:

• Task regression testing (#3223)
• Stabilized distributed gpt2 test (#3219)
• CI for Mephisto tasks (#3206, #3238, #3246, #3254)
• CircleCI refactor (#3245)
• Documentation improvements (#3226, #3251)
• Deprecated some dead code (#3257)
• Various CI fixes (#3310, #3264)

v0.9.4 release

Known issues

• Short options like -m and -t do fail in Python 3.8. Use --model and --task

New project/paper releases

• Safety recipes project (#3190, #3199)
• Gender bias project (#3171, #3146, #3131))
• Multimodal Blender Bot (#3152, #3151, #3144)
• LIGHT Quests project (#3153, #3156)

Major MTurk changes

• [Removal] Deleting 2/3 of MTurk (#3161)
• [Upgrade] Porting ACUTE-Eval task to Hydra (#3103)
• [Task] Adding option for file with annotation indices per conversation and other features to turn annotations static (#3154, #3142)
• [Docs] AcceptabilityChecker doc (#3124)

New Features

• [ParlAI] Adding tensorboard_logdir argument to TensorboardLogger. (#3184)
• [ParlAI] [TGA] Return text for all beams (#3123, #3179)
• [ParlAI] Using the parlai color (#3137)
• [ParlAI] Add a --version flag to the parlai command #3163 (#3164)
• [Task] Improvements to Taskmaster-2 (#3135, #3134, #3129)
• [Task] Flag to enable polarity categories in ImageChat (#3132)
• [Model] Allow Polyencoder to resize token embs (#3194)

Bug Fixes

• [Model] Fix GPT-2 generation when using batchsize >1
• [ParlAI] Show subcommand help error messages on parse error in supercommand. (#3196)
• [ParlAI] Fixes to --init-model options and --init-opt. (#3162, #3112)
• [ParlAI] Update requests version (#3155)
• [HuggingFace/GPT2] Check directory depending on size param for gpt2 files (#3158)
• [Docs] Correct swapped README links (#3166)
• [Task] ConvAI2: Bug fix in convai2:normalized NormalizedTeacher (#3168)
• [Task] COCO: Make sure folder is always created when unzipping COCO (#3143)

Developer notes & Tests

• [Tests] New/fixed tests (#3167, #3117, #3114, #3082)
• [Feature] Add ability for Metrics to add metrics from another Metrics object (#3145)
• [Docs] Comments in AbstractWrapperTeacher.act() (#3160)
• [Docs] Documentation improvements to poly-encoder (#3159)
• [Quality] distributed_eval now mimics distributed_train (#3157)
• [Quality] Add "fb" option to autoformat script for parlai_fb (#3165)

Known issues

• Short options like -m and -t do fail in Python 3.8. Use --model and --task

Breaking Changes

• A number of old MTurk tasks have been archived and removed from the code (#3085)

New Features

• [image] Detectron feature extraction (#3083)
• [data] Natural questions (#3070)
• [data] TaskMaster-2 (#2678)
• [data] New versions of multiwoz (#3072)
• [distributed] Allow non-tcp based distributed setup (#3095)
• [core] Move torch.load/torch.save to PathManager. (#3094, #3077)
• [mturk] New task on static turn annotations (#3053)
• [mturk] New features in human+model annotation (#3006)
• [core] TorchClassifierAgent now prints its number of prameters (#3086)

Doc Changes:

• New Worlds tutorial (#3049)
• Tutorial on using -t jsonfile (#3061)
• Better help message for --init-model (#3090)
• Additions to FAQ (#3073)
• Updated model zoo descriptions for BlenderBot (#3096)

Bug Fixes

• Distributed evaluation now writes to world logs earlier (#3122)
• An argument was updated from store_true to bool (#3113)
• Self-chat now fails loudly with unexpected batchsize (#3081)
• Update drqa default tokenizer away from removed (#3069)
• Using wizard of wikipedia in interactive mode downloads data (#3079)

Developer notes:

• New pre-commit git-secrets (#3106)
• Code coverage improvements (#3110, #3091)
• More reliable tests. (#3108, #3097, #3055)
• Mephisto task dependencies have been updates due to security bugs (#3111, #3101, #3104)
• MTurk config folders are exempt from init.py requirements (#3105)

v0.9.2 release notes

Breaking changes:

• Fix no-op mask fill in some of the the transformer/ranker models (#3068). We believe this to be fine, but it may change the performance of some ranking models. If you are training a transformer/ranker or polyencoder model and you notice your model no longer gets the same performance, please file an issue on GitHub.

Bugfixes:

• Fix a bug where using tensorboard in training caused a crash (#3067)
• Fix a bug where BART failed to load from checkpoint properly (#3060)
• Fix an issue where some arguments could not be overridden in BART (#3065)

Other:

• fromfile now emits a warning about using valid/test (#3063)

Version 0.9.1 Changelog:

ParlAI now supports pytorch 1.6.

Breaking changes:

• Change the arguments for JsonFile teacher, and fix some bugs (#3054)
• ParlAI now expects HuggingFace tokenizers >=0.8.0 (#2967)

New features/Improvements:

• Release of LIGHT-WILD project (#2992)
• HRED has now been added as a standard model (#2989)
• DialoGPT has been added as a standard model (#3007)
• Allow colored logging in jupyter notebooks. (#3024)
• [Squad2] Add option to change label for impossible answers (#3020)
• ChunkTeacher has support to prevent auto enqueuing (#2993)
• Add support for PathManager. (#3011)
• DistilGPT2 has been added as an option of GPT2 model (#2997)
• tfidf_retriever now reports doc ids (#3008)

Lots of new/updated documentation

• Custom metrics tutorial (#2975)
• Chat Services tutorial clarification (#3048)
• Release of Mephisto ACUTE-Evals (#3002)
• Writing custom scripts and integration with parlai command (#3046)
• Writing tests for ParlAI (#3044)
• Contributor guide (#3047)
• Mark deprecation of ParlAI MTurk for Mephisto (#2988)
• Updates to docs on how to build docs (#3041)
• Update Link and Commands Display format (#2952)
• Small modifications (#3036, #2994, #3031, #2916)

Bugfixes:

• Upgrade to pytorch 1.6 (#3022)
• Linear/Cosine LR schedulers don't crash in distributed (#3000)
• Fix unexpected crashes in convo_render (#3052)
• Context block segfault fix (#3019)
• Fix crash in loading from checkpoint in Cosine/Linear LR schedulers (#3025)
• Upgrade schedulers to be Pytorch 1.6 friendly (#3025)
• Fix bug where "restart conversation" didn't work in chat_services browser (#3050)
• Faster amazon QA teacher (#2851)
• Fix crashes to Reward Unlikelihood agent (#3034)
• Fix a misalignment in BART agent between generation and training (#3021)
• Prevent mixing model parallel and multiprocessing. (#2964)
• [ACUTE][Mephisto] Fix extend that should append (#3029)
• Adding option to load pretrained model from directory for gpt2 (#3027)
• Fix an obscure crash with dict_file and GPT2 (#3038)

Developer:

• Improve coverage of some base agents (#3026)
• CircleCI now produces a build of website as an artifact (#3035)
• Rework sphinx docs and upgrade docs to MarkDown (#2972, #2999, #3003)
• Minor refactor of transformer decoder forward (#3009)

v0.9.0 ChangeLog

Breaking Changes

• Removed support for HogWild and ExecutableWorld (#2919)
• Archived a number of older projects (Mastering the dungeon, TalkTheWalk, Memnn_feed) (#2919)

New Features

• [Dynamic Batching] Add world logging capability (#2941)
• [Torch Generator Agent] Option to block full context (#2928)
• [Safety] can supply custom offensive language file (#2968)
• [Safety] Add new model file argument to safety classifiers (#2955)
• Script for comparing two different opts (#2932)
• [Image S2S] Standardize Image Seq2Seq Image Processing (#2911)

Bug fixes

• [Chunk teacher] Bug with distributed evaluation (#2935)
• Add encoding to "with open"s in teachers.py (#2946)
• Fix an issue where parlai suppercommand wasn't handling set_params appropriately (#2956)
• Fix a bug where Git was trying to be called regard (#2961)
• Project specific: Personality Captions interactive mode fix (#2963)
• Project specific: fix openers loading opt (#2965)
• Fix underscore opt not being handled by parlai supercommand (#2938)
• [Style] Add style-controlled generation agent (#2913)
• Add return in report of wrapper agents (#2907)

Doc updates

• Fixed a mistake in documentation for creating tasks based on ParlAIDialogTeacher. (#2973)
• Updated link for LICENSE file (#2970)
• Added pip installation guide to Readme.md (#2959)
• Mention colab tutorial in our quickstart tutorial (#2936)

Other

• Tweak colors of logs. (#2940)
• Add printing of opts into scripts. (#2933)
• [Dodeca] Split Dodeca tar file (#2934)
• [BART/TGA] BART and TGA API Updates (#2840)
• Show all stdout on GPU tests (#2929)

Tying off a "pieces of 8" release, with an arbitrary version number to switch to semantic versioning.

Notable additions:

• Fix a bug in distributed launching (#2850)
• Add support for reversed history in TorchAgent (--history-reversed true), which reverses text_vec. (#2786, #2888 )
• A new parlai super command, with a script registry. This replaces all the usage of "examples/train_model.py" etc. (#2859, #2845, #2860, #2887, #2889)
• Some improvements to model parallelization (#2839)
• Add some timing metrics, including tps (tokens per second), etc. (#2878)
• Self-chat can set up two models talking to each other with different options (#2803)
• An mturk task with per-turn annotations (#2877, #2886)
• An abstract class for writing "wrapper" teachers, which contain some transformation of data (#2842)
• Support writing to a log in an interactive script (#2881)

Bug fixes:

• Fix a type error in valid_step (#2890)
• Improve the output when listing differences between an init model and specified options (#2880)
• Upgrade TGA to be compatible with pytorch 1.6 (#2902)
• Fix a bug in seq2seq that broke with pytorch 1.6 (#2910)

Breaking changes:

• Switch --load-from-checkpoint to True by default in train_model
• Upgrade dependency to pillow 7.2.0 for a security reason (#2896)
• Switch to using a suffix on the model file, instead of a NamedTemporaryFile, for atomic save (#2898)
• Removed some old deprecated pytorch data teacher text (#2883)

Notable additions since 2020-07-13:

• Support for adding special to tokenizers (#2828)
• Distributed Evaluation, to help scale generations/evaluations across many GPUs (#2775)
• Google SGD dataset (#2738).

Other notable features since 2020-06-16:

• Multi-turn chunk teacher (#2805)
• Fromfile jsonl teacher; works directly from files dumped by self_chat, ACUTE logs, etc. (#2717)
• BART model now part of parlai (-m bart) (#2781)
• Atomic saves to be more robust to preemption (#2760)

Recent bugfixes:

• #2789: crash when printing an empty report
• #2788: multi-tasking failed to sample appropriately with --datatype train:stream.

Release of jupyter notebook features for ODSC tutorial. You can already try the interactive colab notebook.

For pip