AliceMind
AliceMind: ALIbaba's Collection of Encoder-decoders from MinD (Machine IntelligeNce of Damo) Lab
This repository provides pre-trained encoder-decoder models and its related optimization techniques developed by Alibaba's MinD (Machine IntelligeNce of Damo) Lab.
The family of AliceMind:
- Language understanding model: StructBERT (
ICLR 2020) - Generative language model: PALM (
EMNLP 2020) - Cross-lingual language model: VECO (
ACL 2021) - Cross-modal language model: StructVBERT (
CVPR 2020 VQA Challenge Runner-up) - Structural language model: StructuralLM (
ACL 2021) - Chinese language understanding model with multi-granularity inputs: LatticeBERT (
NAACL 2021) - Pre-training table model: SDCUP (
Under Review)
News
- March, 2021: AliceMind released!
- May, 2021: VECO and StructuralLM were accepted by ACL 2021.
- September, 2021: The first Chinese pre-training table model SDCUP released!
Models
-
StructBERT (March 15, 2021): pre-trained models for natural language understanding (NLU). We extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. "StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding" (
ICLR 2020) -
PALM (March 15, 2021): pre-trained models for natural language generation (NLG). We propose a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. It achieves new SOTA results in several downstream tasks. "PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation" (
EMNLP 2020) -
VECO v0 (March 15, 2021): pre-trained models for cross-lingual (x) natural language understanding (x-NLU) and generation (x-NLG). VECO (v0) achieves the new SOTA results on various cross-lingual understanding tasks of the XTREME benchmark, covering text classification, sequence labeling, question answering, and sentence retrieval. For cross-lingual generation tasks, it also outperforms all existing cross-lingual models and state-of-the-art Transformer variants on WMT14 English-to-German and English-to-French translation datasets, with gains of up to 1~2 BLEU. “VECO: Variable Encoder-decoder Pre-training for Cross-lingual Understanding and Generation" (
ACL 2021) -
StructVBERT (March 15, 2021): pre-trained models for vision-language understanding. We propose a new single-stream visual-linguistic pre-training scheme by leveraging multi-stage progressive pre-training and multi-task learning. StructVBERT obtained the 2020 VQA Challenge Runner-up award, and SOTA result on VQA 2020 public Test-standard benchmark (June 2020). "Talk Slides" (
CVPR 2020 VQA Challenge Runner-up). -
StructuralLM (March 15, 2021): pre-trained models for document-image understanding. We propose a new pre-training approach, StructuralLM, to jointly leverage cell and layout information from scanned documents. The pre-trained StructuralLM achieves new state-of-the-art results in different types of downstream tasks. "StructuralLM: Structural Pre-training for Form Understanding" (
ACL 2021) -
LatticeBERT (March 15, 2021): we propose a novel pre-training paradigm for Chinese — Lattice-BERT which explicitly incorporates word representations with those of characters, thus can model a sentence in a multi-granularity manner. "Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models" (
NAACL 2021) -
SDCUP (September 6, 2021): pre-trained models for table understanding. We design a schema dependency pre-training objective to impose the desired inductive bias into the learned representations for table pre-training. We further propose a schema-aware curriculum learning approach to alleviate the impact of noise and learn effectively from the pre-training data in an easy-to-hard manner. The experiment results on SQUALL and Spider demonstrate the effectiveness of our pre-training objective and curriculum in comparison to a variety of baselines. "SDCUP: Schema Dependency Enhanced Curriculum Pre-Training for Table Semantic Parsing" (
Under Review)
Contact Information
AliceMind Official Website: https://nlp.aliyun.com/portal#/alice
AliceMind Open Platform: https://alicemind.aliyuncs.com
Please submit a GitHub issue if you have want help or have issues using ALICE.
For more information, you can join the AliceMind Users Group on DingTalk to contact us. The number of the DingTalk group is 35738533.
For other business communications, please contact [email protected]
License
AliceMind is released under the Apache 2.0 license.
Copyright 1999-2020 Alibaba Group Holding Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at the following link.
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
这是图文检索里的代码，我想问一下self.distill是代表什么意思，做蒸馏吗？还有，self._momentum_update()、self._dequeue_and_enqueue(image_feat_m, text_feat_m, idx)这好像动量对比学习里的代码，我也不明白这是什么意思。如果有可能的话，请您尽可能地告诉我这段代码的含义。十分感谢！
20 Nov 03, 2022
286 Jan 02, 2023
1.5k Dec 28, 2022
152 Sep 02, 2022
255 Dec 27, 2022
27 Oct 30, 2022
113 Dec 15, 2022
2.1k Jan 01, 2023
22 Dec 14, 2022
1 Jan 18, 2022
1 May 30, 2022
9 Jul 21, 2022
2 Jan 16, 2022
0 Oct 21, 2021
224 Nov 29, 2022
9 Oct 11, 2022
102 Dec 07, 2022
2 Nov 12, 2021
2.5k Jan 03, 2023
1 Dec 22, 2021