AutoML - Neural Architecture Search
This is a collection of our AutoML-NAS work
iRPE (
NEW): Rethinking and Improving Relative Position Encoding for Vision Transformer
AutoFormer (
NEW): AutoFormer: Searching Transformers for Visual Recognition
Cream (
@NeurIPS'20): Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
We also implemented our NAS algorithms on Microsoft NNI (Neural Network Intelligence).
News
-
โ๏ธ Hiring research interns for neural architecture search, tiny transformer design, model compression projects: [email protected] -
๐ฅ Oct, 2021: AutoFormerV2 has been accepted by NeurIPS'21, will be released soon. -
๐ฅ Aug, 2021: Code for AutoFormer is now released. -
๐ฅ July, 2021: iRPE code (with CUDA Acceleration) is now released. Paper is here. -
๐ฅ July, 2021: iRPE has been accepted by ICCV'21. -
๐ฅ July, 2021: AutoFormer has been accepted by ICCV'21. -
๐ฅ July, 2021: AutoFormer is now available on arXiv. -
๐ฅ Oct, 2020: Code for Cream is now released. -
๐ฅ Oct, 2020: Cream was accepted to NeurIPS'20
Works
AutoFormer
AutoFormer is new one-shot architecture search framework dedicated to vision transformer search. It entangles the weights of different vision transformer blocks in the same layers during supernet training. Benefiting from the strategy, the trained supernet allows thousands of subnets to be very well-trained. Specifically, the performance of these subnets with weights inherited from the supernet is comparable to those retrained from scratch.
iRPE
Image RPE (iRPE for short) methods are new relative position encoding methods dedicated to 2D images, considering directional relative distance modeling as well as the interactions between queries and relative position embeddings in self-attention mechanism. The proposed iRPE methods are simple and lightweight, being easily plugged into transformer blocks. Experiments demonstrate that solely due to the proposed encoding methods, DeiT and DETR obtain up to 1.5% (top-1 Acc) and 1.3% (mAP) stable improvements over their original versions on ImageNet and COCO respectively, without tuning any extra hyperparamters such as learning rate and weight decay. Our ablation and analysis also yield interesting findings, some of which run counter to previous understanding.
Cream
[Paper] [Models-Google Drive][Models-Baidu Disk (password: wqw6)] [Slides] [BibTex]
In this work, we present a simple yet effective architecture distillation method. The central idea is that subnetworks can learn collaboratively and teach each other throughout the training process, aiming to boost the convergence of individual models. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Distilling knowledge from the prioritized paths is able to boost the training of subnetworks. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop.
Bibtex
@InProceedings{iRPE,
author = {Wu, Kan and Peng, Houwen and Chen, Minghao and Fu, Jianlong and Chao, Hongyang},
title = {Rethinking and Improving Relative Position Encoding for Vision Transformer},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2021},
pages = {10033-10041}
}
@article{AutoFormer,
title={AutoFormer: Searching Transformers for Visual Recognition},
author={Chen, Minghao and Peng, Houwen and Fu, Jianlong and Ling, Haibin},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2021}
}
@article{Cream,
title={Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search},
author={Peng, Houwen and Du, Hao and Yu, Hongyuan and Li, Qi and Liao, Jing and Fu, Jianlong},
journal={Advances in Neural Information Processing Systems},
volume={33},
year={2020}
}
License
License under an MIT license.
125 Nov 26, 2022
51 Nov 10, 2022
5 Jul 07, 2022
2 Jan 05, 2023
109 Nov 16, 2022
12 Mar 29, 2022
115 Nov 27, 2022
74 Jan 03, 2023
468 Jan 03, 2023
90 Dec 07, 2022
2 Mar 12, 2022
23 Nov 29, 2022
64 Nov 11, 2022
2 Nov 14, 2022
1.5k Dec 29, 2022
68 Dec 27, 2022
163 Dec 26, 2022
51 Nov 29, 2022
594 Jan 06, 2023
6 Feb 28, 2022
