A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

Last update: Dec 26, 2022

Overview

MMF is a modular framework for vision and language multimodal research from Facebook AI Research. MMF contains reference implementations of state-of-the-art vision and language models and has powered multiple research projects at Facebook AI Research. See full list of project inside or built on MMF here.

MMF is powered by PyTorch, allows distributed training and is un-opinionated, scalable and fast. Use MMF to bootstrap for your next vision and language multimodal research project by following the installation instructions. Take a look at list of MMF features here.

MMF also acts as starter codebase for challenges around vision and language datasets (The Hateful Memes, TextVQA, TextCaps and VQA challenges). MMF was formerly known as Pythia. The next video shows an overview of how datasets and models work inside MMF. Checkout MMF's video overview.

Installation

Follow installation instructions in the documentation.

Documentation

Learn more about MMF here.

Citation

If you use MMF in your work or use any models published in MMF, please cite:

@misc{singh2020mmf,
  author =       {Singh, Amanpreet and Goswami, Vedanuj and Natarajan, Vivek and Jiang, Yu and Chen, Xinlei and Shah, Meet and
                 Rohrbach, Marcus and Batra, Dhruv and Parikh, Devi},
  title =        {MMF: A multimodal framework for vision and language research},
  howpublished = {\url{https://github.com/facebookresearch/mmf}},
  year =         {2020}
}

License

MMF is licensed under BSD license available in LICENSE file

Comments

[feat] Add ViLT model
Stack from ghstack:

#1142

#1106

#1103

-> #1102

#1141

#1101

Add ViLT model and unit tests. ViLT model from megaVLT, using ViT from huggingface.

Differential Revision: D31117427
CLA Signed Merged
opened by Ryan-Qiyu-Jiang 49
[feat] Add ViLT image and text embeddings
Stack from ghstack:

#1142

#1106

#1103

#1102

#1141

-> #1101

Add ViT image and Bert text embedding encoders for ViLT model.

Differential Revision: D31117428
CLA Signed Merged
opened by Ryan-Qiyu-Jiang 46
[feat][PL][8/N] save config with checkpoint
Stack from ghstack:

#1035 [feat][PL][11/N] Inference with test

#1033 [feat][PL][10/N] lightning distributed

#1030 [refactor][PL][9/N] change the name of the _load_checkpoint to be more descriptive to contain test

#1024 [feat][PL][8/N] save config with checkpoint

#1023 [feat][PL][7/N] trainer checkpoint from mmf to lightning update

#1022 [feat][PL][6/N] pretrained state mapping for PL

Differential Revision: D29670421
CLA Signed Merged
opened by ytsheng 39
[feat][PL][6/N] pretrained state mapping for PL
Stack from ghstack:

#1035 [feat][PL][11/N] Inference with test

#1033 [feat][PL][10/N] lightning distributed

#1030 [refactor][PL][9/N] change the name of the _load_checkpoint to be more descriptive to contain test

#1024 [feat][PL][8/N] save config with checkpoint

#1023 [feat][PL][7/N] trainer checkpoint from mmf to lightning update

#1022 [feat][PL][6/N] pretrained state mapping for PL

Differential Revision: D29670417
CLA Signed
opened by ytsheng 37
[refactor][PL][9/N] change the name of the _load_checkpoint to be more descriptive to contain test
Stack from ghstack:

#1035 [feat][PL][11/N] Inference with test

#1033 [feat][PL][10/N] lightning distributed

#1030 [refactor][PL][9/N] change the name of the _load_checkpoint to be more descriptive to contain test

#1024 [feat][PL][8/N] save config with checkpoint

#1023 [feat][PL][7/N] trainer checkpoint from mmf to lightning update

#1022 [feat][PL][6/N] pretrained state mapping for PL

Differential Revision: D29888294
CLA Signed
opened by ytsheng 36
[feat] Add ViLT encoder
Stack from ghstack:

#1142

#1106

#1103

#1102

#1141

#1101

-> #1100

Add encoder for loading pretrained ViTModel weights from huggingface

Differential Revision: D31117425
CLA Signed
opened by Ryan-Qiyu-Jiang 33
[feat][PL][10/N] lightning distributed
Stack from ghstack:

#1035 [feat][PL][11/N] Inference with test

#1033 [feat][PL][10/N] lightning distributed

#1030 [refactor][PL][9/N] change the name of the _load_checkpoint to be more descriptive to contain test

#1024 [feat][PL][8/N] save config with checkpoint

#1023 [feat][PL][7/N] trainer checkpoint from mmf to lightning update

#1022 [feat][PL][6/N] pretrained state mapping for PL

Differential Revision: D29981634
CLA Signed
opened by ytsheng 33
[feat][PL][7/N] trainer checkpoint from mmf to lightning update
Stack from ghstack:

#1035 [feat][PL][11/N] Inference with test

#1033 [feat][PL][10/N] lightning distributed

#1030 [refactor][PL][9/N] change the name of the _load_checkpoint to be more descriptive to contain test

#1024 [feat][PL][8/N] save config with checkpoint

#1023 [feat][PL][7/N] trainer checkpoint from mmf to lightning update

#1022 [feat][PL][6/N] pretrained state mapping for PL

Differential Revision: D29670422
CLA Signed
opened by ytsheng 33
[feat] Add ViT modules
Stack from ghstack:

#1142

#1106

#1103

#1102

#1141

#1101

#1100

-> #1099

Add modified huggingface ViT modules from megaVLT supporting attention masks

Differential Revision: D31117429
CLA Signed
opened by Ryan-Qiyu-Jiang 30
[feat] Add UNITER model wrapper
Stack from ghstack:

#1144

#1128

-> #1127

#1133

Add UNITER model to mmf registry with support for pretraining through yaml head configs.

Differential Revision: D31768457
CLA Signed
opened by Ryan-Qiyu-Jiang 28
Error during training Textvqa

❓ Questions and Help

Hi,

While I am trying the training code with m4c model, I am getting the following error,

2021-03-11T03:34:15 | mmf.utils.general: Total Parameters: 90850184. Trained Parameters: 90850184 2021-03-11T03:34:15 | mmf.trainers.core.training_loop: Starting training... Traceback (most recent call last): File "C:\Users\kvman\anaconda3\envs\mmf\Scripts\mmf_run-script.py", line 33, in sys.exit(load_entry_point('mmf', 'console_scripts', 'mmf_run')()) File "d:\project\new folder\mmf\mmf_cli\run.py", line 133, in run main(configuration, predict=predict) File "d:\project\new folder\mmf\mmf_cli\run.py", line 56, in main trainer.train() File "d:\project\new folder\mmf\mmf\trainers\mmf_trainer.py", line 132, in train self.training_loop() File "d:\project\new folder\mmf\mmf\trainers\core\training_loop.py", line 31, in training_loop self.run_training_epoch() File "d:\project\new folder\mmf\mmf\trainers\core\training_loop.py", line 74, in run_training_epoch for idx, batch in enumerate(self.train_loader): File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data\dataloader.py", line 363, in next data = self._next_data() File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data\dataloader.py", line 989, in _next_data return self._process_data(data) File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data\dataloader.py", line 1014, in _process_data data.reraise() File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch_utils.py", line 395, in reraise raise self.exc_type(msg) KeyError: Caught KeyError in DataLoader worker process 0. Original Traceback (most recent call last): File "d:\project\new folder\mmf\mmf\datasets\databases\readers\feature_readers.py", line 231, in load image_id = int(split.split("")[-1]) ValueError: invalid literal for int() with base 10: 'train\7f14a505b6edcbc5'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data_utils\worker.py", line 185, in _worker_loop data = fetcher.fetch(index) File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data\dataset.py", line 207, in getitem return self.datasets[dataset_idx][sample_idx] File "d:\project\new folder\mmf\mmf\datasets\builders\textvqa\dataset.py", line 100, in getitem features = self.features_db[idx] File "d:\project\new folder\mmf\mmf\datasets\databases\features_database.py", line 91, in getitem return self.get(image_info) File "d:\project\new folder\mmf\mmf\datasets\databases\features_database.py", line 99, in get return self.from_path(feature_path) File "d:\project\new folder\mmf\mmf\datasets\databases\features_database.py", line 107, in from_path features, infos = self._get_image_features_and_info(path) File "d:\project\new folder\mmf\mmf\datasets\databases\features_database.py", line 80, in _get_image_features_and_info image_feats, infos = self._read_features_and_info(feat_file) File "d:\project\new folder\mmf\mmf\datasets\databases\features_database.py", line 65, in _read_features_and_info feature, info = feature_reader.read(feat_file) File "d:\project\new folder\mmf\mmf\datasets\databases\readers\feature_readers.py", line 95, in read return self.feat_reader.read(image_feat_path) File "d:\project\new folder\mmf\mmf\datasets\databases\readers\feature_readers.py", line 158, in read image_info = self._load(image_feat_path) File "d:\project\new folder\mmf\mmf\datasets\databases\readers\feature_readers.py", line 238, in _load img_id_idx = self.image_id_indices[image_id] KeyError: b'train\7f14a505b6edcbc5'

When I tried with model = "Lorra", I am getting the below error,

2021-03-11T03:27:37 | mmf.utils.general: Total Parameters: 192497485. Trained Parameters: 192497485 2021-03-11T03:27:37 | mmf.trainers.core.training_loop: Starting training... Traceback (most recent call last): File "C:\Users\kvman\anaconda3\envs\mmf\Scripts\mmf_run-script.py", line 33, in sys.exit(load_entry_point('mmf', 'console_scripts', 'mmf_run')()) File "d:\project\new folder\mmf\mmf_cli\run.py", line 133, in run main(configuration, predict=predict) File "d:\project\new folder\mmf\mmf_cli\run.py", line 56, in main trainer.train() File "d:\project\new folder\mmf\mmf\trainers\mmf_trainer.py", line 132, in train self.training_loop() File "d:\project\new folder\mmf\mmf\trainers\core\training_loop.py", line 31, in training_loop self.run_training_epoch() File "d:\project\new folder\mmf\mmf\trainers\core\training_loop.py", line 74, in run_training_epoch for idx, batch in enumerate(self.train_loader): File "d:\project\new folder\mmf\mmf\datasets\multi_dataset_loader.py", line 213, in iter return iter(self.loaders[0]) File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data\dataloader.py", line 291, in iter return _MultiProcessingDataLoaderIter(self) File "C:\Users\kvman\anaconda3\envs\mmf\lib\site-packages\torch\utils\data\dataloader.py", line 737, in init w.start() File "C:\Users\kvman\anaconda3\envs\mmf\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Users\kvman\anaconda3\envs\mmf\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Users\kvman\anaconda3\envs\mmf\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Users\kvman\anaconda3\envs\mmf\lib\multiprocessing\popen_spawn_win32.py", line 89, in init reduction.dump(process_obj, to_child) File "C:\Users\kvman\anaconda3\envs\mmf\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) BrokenPipeError: [Errno 32] Broken pipe

Kindly help me to resolve this issue.
needs more info triaged

opened by Mano2610 28
Bump terser from 5.10.0 to 5.16.1 in /website
Bumps terser from 5.10.0 to 5.16.1.

Changelog

Sourced from terser's changelog.

v5.16.1

Properly handle references in destructurings (const { [reference]: val } = ...)

Allow parsing of .#privatefield in nested classes

Do not evaluate operations that return large strings if that would make the output code larger

Make collapse_vars handle block scope correctly

Internal improvements: Typos (#1311), more tests, small-scale refactoring

v5.16.0

Disallow private fields in object bodies (#1011)

Parse #privatefield in object (#1279)

Compress #privatefield in object

v5.15.1

Fixed missing parentheses around optional chains

Avoid bare let or const as the bodies of if statements (#1253)

Small internal fixes (#1271)

Avoid inlining a class twice and creating two equivalent but !== classes.

v5.15.0

Basic support for ES2022 class static initializer blocks.

Add AudioWorkletNode constructor options to domprops list (#1230)

Make identity function inliner not inline id(...expandedArgs)

v5.14.2

Security fix for RegExps that should not be evaluated (regexp DDOS)

Source maps improvements (#1211)

Performance improvements in long property access evaluation (#1213)

v5.14.1

keep_numbers option added to TypeScript defs (#1208)

Fixed parsing of nested template strings (#1204)

v5.14.0

Switched to @jridgewell/source-map for sourcemap generation (#1190, #1181)

Fixed source maps with non-terminated segments (#1106)

Enabled typescript types to be imported from the package (#1194)

Extra DOM props have been added (#1191)

Delete the AST while generating code, as a means to save RAM

v5.13.1

Removed self-assignments (varname=varname) (closes #1081)

Separated inlining code (for inlining things into references, or removing IIFEs)

Allow multiple identifiers with the same name in var destructuring (eg var { a, a } = x) (#1176)

v5.13.0

... (truncated)

Commits

b90215a 5.16.1

319fe31 update changelog

d941a11 mark computed properties in for..of initializers as used. Closes #1238

635400e move AST_Scope#drop_unused into its own file

a192fac some cleanups

dcc3030 add shadowed variable test

4d37f39 traverse exported code while hoisting object properties. Closes #1307

1e99e5d update test

68461d4 properly detect nested classes. Closes #1313

b91e0ce docs: fix simple typo, concatentation -> concatenation (#1311)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies javascript
opened by dependabot[bot] 0
Bump minimist from 1.2.5 to 1.2.7 in /website
Bumps minimist from 1.2.5 to 1.2.7.

Changelog

Sourced from minimist's changelog.

v1.2.7 - 2022-10-10

Commits

[meta] add auto-changelog 0ebf4eb

[actions] add reusable workflows e115b63

[eslint] add eslint; rules to enable later are warnings f58745b

[Dev Deps] switch from covert to nyc ab03356

[readme] rename and add badges 236f4a0

[meta] create FUNDING.yml; add funding in package.json 783a49b

[meta] use npmignore to autogenerate an npmignore file f81ece6

Only apps should have lockfiles 56cad44

[Dev Deps] update covert, tape; remove unnecessary tap 49c5f9f

[Tests] add aud in posttest 228ae93

[meta] add safe-publish-latest 01fc23f

[meta] update repo URLs 6b164c7

v1.2.6 - 2022-03-21

Commits

test from prototype pollution PR bc8ecee

isConstructorOrProto adapted from PR c2b9819

security notice for additional prototype pollution issue ef88b93

Commits

c590d75 v1.2.7

0ebf4eb [meta] add auto-changelog

e115b63 [actions] add reusable workflows

01fc23f [meta] add safe-publish-latest

f58745b [eslint] add eslint; rules to enable later are warnings

228ae93 [Tests] add aud in posttest

236f4a0 [readme] rename and add badges

ab03356 [Dev Deps] switch from covert to nyc

49c5f9f [Dev Deps] update covert, tape; remove unnecessary tap

783a49b [meta] create FUNDING.yml; add funding in package.json

Additional commits viewable in compare view

Maintainer changes

This version was pushed to npm by ljharb, a new releaser for minimist since your current version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies javascript
opened by dependabot[bot] 0
Bump prismjs from 1.26.0 to 1.29.0 in /website
Bumps prismjs from 1.26.0 to 1.29.0.

Release notes

Sourced from prismjs's releases.

v1.29.0

Release 1.29.0

v1.28.0

Release 1.28.0

v1.27.0

Release 1.27.0

Changelog

Sourced from prismjs's changelog.

1.29.0 (2022-08-23)

New components

BBj (#3511) 1134bdfc

BQN (#3515) 859f99a0

Cilk/C & Cilk/C++ (#3522) c8462a29

Gradle (#3443) 32119823

METAFONT (#3465) 2815f699

WGSL (#3455) 4c87d418

Updated components

AsciiDoc

Some regexes are too greedy (#3481) c4cbeeaa

Bash

Added "sh" alias (#3509) 6b824d47

Added support for parameters and the java and sysctl commands. (#3505) b9512b22

Added cargo command (#3488) 3e937137

BBj

Improve regexes (#3512) 0cad9ae5

CSS

Fixed @-rules not accounting for strings (#3438) 0d4b6cb6

CSS Extras

Added support for RebeccaPurple color (#3448) 646b2e0a

Hoon

Fixed escaped strings (#3473) 64642716

Java

Added support for constants (#3507) 342a0039

Markup

Fixed quotes in HTML attribute values (#3442) ca8eaeee

NSIS

Added missing commands (#3504) b0c2a9b4

Scala

Updated keywords to support Scala 3 (#3506) a090d063

SCSS

Fix casing in title of the scss lang (#3501) 2aed9ce7

Updated plugins

Line Highlight

Account for offset when clamping ranges (#3518) 098e3000

Ignore ranges outside of actual lines (#3475) 9a4e725b

Normalize Whitespace

Add configuration via attributes (#3467) 91dea0c8

Other

Added security policy (#3070) 05ee042a

Added list of maintainers (#3410) 866b302e

... (truncated)

Commits

59e5a34 1.29.0

cd080f2 Updated npmignore to include new MD files (#3534)

751664b Added PR stop notice (#3532)

248f6ab Added changelog for v1.29.0 (#3533)

098e300 Line Highlight: Account for offset when clamping ranges (#3518)

6b824d4 Bash: Added "sh" alias (#3509)

15272f7 Website: Added third-party tutorial for Pug template (#3459)

c8462a2 Cilk: Add support for Cilk (with C/C++) (#3522)

859f99a Added bqn language support (#3515)

0cad9ae BBj: Improve regexes (#3512)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies javascript
opened by dependabot[bot] 0
Bump eventsource from 1.1.0 to 1.1.2 in /website
Bumps eventsource from 1.1.0 to 1.1.2.

Changelog

Sourced from eventsource's changelog.

1.1.2

Inline origin resolution, drops original dependency (#281 Espen Hovlandsdal)

1.1.1

Do not include authorization and cookie headers on redirect to different origin (#273 Espen Hovlandsdal)

Commits

0a8b85b 1.1.2

f99ae66 docs: update history for 1.1.2

06c9721 chore: rebuild polyfill

9494642 fix: inline origin resolution, drop original dependency (#281)

aa7a408 1.1.1

56d489e chore: rebuild polyfill

4a951e5 docs: update history for 1.1.1

f9f6416 fix: strip sensitive headers on redirect to different origin

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies javascript
opened by dependabot[bot] 0
Bump follow-redirects from 1.14.7 to 1.15.2 in /website
Bumps follow-redirects from 1.14.7 to 1.15.2.

Commits

9655237 Release version 1.15.2 of the npm package.

6e2b86d Default to localhost if no host given.

449e895 Throw invalid URL error on relative URLs.

e30137c Use type functions.

76ea31f ternary operator syntax fix

84c00b0 HTTP header lines are separated by CRLF.

d28bcbf Create SECURITY.md (#202)

62a551c Release version 1.15.1 of the npm package.

7fe0779 Use for ... of.

948c30c Fix redirecting to relative URL when using proxy

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies javascript
opened by dependabot[bot] 0
What are the inputs to be provided in pretrained VilBERT model?

Hello, I loaded pretrained model by the following provided code. from mmf.common.registry import registry

model_cls = registry.get_model_class("vilbert") model = model_cls.from_pretrained("vilbert.finetuned.hateful_memes.direct")

Now, I want to use this model on my dataset but I can't figure out what are the inputs to be provided in the model. These all are the inputs to be given, input_0, input_1, targets, input_mask, segment_ids, return_output_dict, image_info_0 , image_features_0

opened by ApoorvVerma5 0

Releases(v0.3.1)

v0.3.1(Aug 26, 2019)

Source code(tar.gz)
Source code(zip)
v0.3(Jun 26, 2019)
Features

Multi-tasking support: Multitasking over various datasets available in Pythia

Distributed training support

Better Customization Support: Use your custom losses, metrics, optimizers and lot of more

Standardized Trainer API: A standard trainer API to fit most of your use-cases, if not inherit and build your own trainer

Processors: Use processors to build out your datasets easily and without pain

SampleList and Sample: Use SampleList and Sample to have more granular control over what you pass and a single unified API for accessing attributes whether inside a dataset, a single sample or a batch

Feature Extraction: A new simple script to extract out features and related information from VQA MaskRCNN Benchmark

Registry: No need to manually load datasets and models anymore, registry takes care of loading your models, datasets and other classes at the fly. Think of registry as a singleton containing all that you need.

Tensorboard Logging: Tensorboard logging is now provided by default.

Configuration: Better hierarchal configuration system for better separation of concerns.

Checkpointing: Better control checkpointing and resuming

Logging: Better logging is provided now with eta, individual val losses and metrics. Just pass them back from your model and everything logs automatically

EvalAI Evaluation: Now, directly output JSON files that can be uploaded to EvalAI

Early Stopping

New Embeddings Support: FastText, GloVe, BERT etc.

Datasets

Following new datasets were added:

TextVQA

VizWiz

COCO Captioning

Preliminary GQA support

Models

LoRRA

Revamped Pythia

BUTD

Preliminary BAN Support

Note: There are a lot of breaking changes in the API from v0.1. Refer to the documentation to learn more on how to work with Pythia v0.3.
Source code(tar.gz)
Source code(zip)

Owner

Facebook Research

GitHub Repository https://mmf.sh/

neural network based speaker embedder

Content What is deepaudio-speaker? Installation Get Started Model Architecture How to contribute to deepaudio-speaker? Acknowledge What is deepaudio-s

20 Dec 29, 2022

Pytorch code for ICRA'21 paper: "Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation"

Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation This repository is the pytorch implementation of our paper: Hierarchical Cr

44 Jan 06, 2023

The source code of "Language Models are Few-shot Multilingual Learners" (MRL @ EMNLP 2021)

Language Models are Few-shot Multilingual Learners Paper This is the source code of the paper [Arxiv] [ACL Anthology]: This code has been written usin

45 Nov 21, 2022

NLP: SLU tagging

3 Jan 14, 2022

Train 🤗transformers with DeepSpeed: ZeRO-2, ZeRO-3

Fork from https://github.com/huggingface/transformers/tree/86d5fb0b360e68de46d40265e7c707fe68c8015b/examples/pytorch/language-modeling at 2021.05.17.

12 Oct 26, 2022

Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"

T5: Text-To-Text Transfer Transformer The t5 library serves primarily as code for reproducing the experiments in Exploring the Limits of Transfer Lear

4.6k Jan 01, 2023

CoSENT 比Sentence-BERT更有效的句向量方案

201 Dec 12, 2022

Open-Source Toolkit for End-to-End Speech Recognition leveraging PyTorch-Lightning and Hydra.

🤗 Contributing to OpenSpeech 🤗 OpenSpeech provides reference implementations of various ASR modeling papers and three languages recipe to perform ta

513 Jan 03, 2023

Unlimited Call - Text Bombing Tool

FastBomber Unlimited Call - Text Bombing Tool Installation On Termux

6 Nov 10, 2022

Code for hyperboloid embeddings for knowledge graph entities

Implementation for the papers: Self-Supervised Hyperboloid Representations from Logical Queries over Knowledge Graphs, Nurendra Choudhary, Nikhil Rao,

30 Dec 10, 2022

fastNLP: A Modularized and Extensible NLP Framework. Currently still in incubation.

fastNLP fastNLP是一款轻量级的自然语言处理（NLP）工具包，目标是快速实现NLP任务以及构建复杂模型。 fastNLP具有如下的特性：统一的Tabular式数据容器，简化数据预处理过程；内置多种数据集的Loader和Pipe，省去预处理代码; 各种方便的NLP工具，例如Embedd

2.8k Jan 01, 2023

Chinese NewsTitle Generation Project by GPT2.带有超级详细注释的中文GPT2新闻标题生成项目。

GPT2-NewsTitle 带有超详细注释的GPT2新闻标题生成项目 UpDate 01.02.2021 从网上收集数据，将清华新闻数据、搜狗新闻数据等新闻数据集，以及开源的一些摘要数据进行整理清洗，构建一个较完善的中文摘要数据集。数据集清洗时，仅进行了简单地规则清洗。

785 Dec 29, 2022

VMD Audio/Text control with natural language

This repository is a proof of principle for performing Molecular Dynamics analysis, in this case with the program VMD, via natural language commands.

13 Jun 09, 2022

A demo for end-to-end English and Chinese text spotting using ABCNet.

ABCNet_Chinese A demo for end-to-end English and Chinese text spotting using ABCNet. This is an old model that was trained a long ago, which serves as

45 Oct 04, 2022

Python package to easily retrain OpenAI's GPT-2 text-generating model on new texts

gpt-2-simple A simple Python package that wraps existing model fine-tuning and generation scripts for OpenAI's GPT-2 text generation model (specifical

3.1k Jan 07, 2023

German Text-To-Speech Engine using Tacotron and Griffin-Lim

jotts JoTTS is a German text-to-speech engine using tacotron and griffin-lim. The synthesizer model has been trained on my voice using Tacotron1. Due

6 Aug 28, 2022

NSFW A chatbot based on GPT2-chitchat

DangBot -- 好怪哦，再来一句卡群怪话bot，powered by GPT2 for Chinese chitchat Training Example: python train.py --lr 5e-2 --epochs 30 --max_len 300 --batch_size 8

11 Jul 21, 2022

Associated Repository for "Translation between Molecules and Natural Language"

MolT5: Translation between Molecules and Natural Language Associated repository for "Translation between Molecules and Natural Language". Table of Con

67 Dec 15, 2022

Coreference resolution for English, French, German and Polish, optimised for limited training data and easily extensible for further languages

Coreferee Author: Richard Paul Hudson, Explosion AI 1. Introduction 1.1 The basic idea 1.2 Getting started 1.2.1 English 1.2.2 French 1.2.3 German 1.2

70 Dec 12, 2022

📔️ Generate a text-based journal from a template file.

JGen 📔️ Generate a text-based journal from a template file. Contents Getting Started Example Overview Usage Details Reserved Keywords Gotchas Getting

21 Sep 25, 2022

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

Related tags

Overview

Installation

Documentation

Citation

License

Comments

❓ Questions and Help

v5.16.1

v5.16.0

v5.15.1

v5.15.0

v5.14.2

v5.14.1

v5.14.0

v5.13.1

v5.13.0

v1.2.7 - 2022-10-10

Commits

v1.2.6 - 2022-03-21

Commits

v1.29.0

v1.28.0

v1.27.0

1.29.0 (2022-08-23)

New components

Updated components

Updated plugins

Other