BossNAS
This repository contains PyTorch evaluation code, retraining code and pretrained models of our paper: BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search. [pdf link]
Illustration of the Siamese supernets training with ensemble bootstrapping.
Illustration of the fabric-like Hybrid CNN-transformer Search Space with flexible down-sampling positions.
Our Results and Trained Models
-
Here is a summary of our searched models:
Model MAdds Steptime Top-1 (%) Top-5 (%) Url BossNet-T0 w/o SE 3.4B 101ms 80.5 95.0 checkpoint BossNet-T0 3.4B 115ms 80.8 95.2 checkpoint BossNet-T0^ 5.7B 147ms 81.6 95.6 same as above BossNet-T1 7.9B 156ms 81.9 95.6 checkpoint BossNet-T1^ 10.5B 165ms 82.2 95.7 same as above -
Here is a summary of architecture rating accuracy of our method:
Search space Dataset Kendall tau Spearman rho Pearson R MBConv ImageNet 0.65 0.78 0.85 NATS-Bench Ss Cifar10 0.53 0.73 0.72 NATS-Bench Ss Cifar100 0.59 0.76 0.79
Usage
1. Requirements
- Install PyTorch 1.7.0+ and torchvision 0.8.1+, for example:
conda install -c pytorch pytorch torchvision
- Install pytorch-image-models 0.3.2, for example:
pip install timm==0.3.2
-
Download the ImageNet dataset from http://image-net.org/, and move validation images to labeled subfolders
- To do this, you can use the following script: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.shvalprep.sh
2. Retrain or Evaluate our BossNet-T models
-
First, move to retraining code directory to perform Retraining or Evaluation.
cd HyTra_retrainingOur retraining code of BossNet-T is based on DeiT repository.
-
You can evaluate our BossNet-T models with the following command:
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T0-80_8.pth
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T1-81_9.pth
Please download our checkpoint files from the result table. Please change the
--nproc_per_nodeoption to suit your GPU numbers, and change the--data-path,--resumeand--input-sizeaccordingly. -
You can retrain our BossNet-T models with the following command:
Please change the
--nproc_per_nodeand--data-pathaccordingly. Note that the learning rate will be automatically scaled according to the GPU numbers and batchsize. We recommend training with 128 batchsize and 8 GPUs.python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
Architecture of our BossNet-T0
3. Evaluate architecture rating accuracy of BossNAS
-
You can get the ranking correlations of BossNAS on MBConv search space with the following commands:
cd MBConv_ranking python get_model_score_mbconv.py
- You can get the ranking correlations of BossNAS on NATS-Bench Ss with the following commands:
cd NATS_SS_ranking python get_model_score_nats.py
Citation
@article{li2021bossnas,
author = {Li, Changlin and
Tang, Tao and
Wang, Guangrun and
Peng, Jiefeng and
Wang, Bing and
Liang, Xiaodan and
Chang, Xiaojun},
title = {BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search},
journal = {arXiv:2103.12424},
year = 2021,
}
TODO
Searching code will be released later.
6 Aug 22, 2022
78 Dec 18, 2022
25.7k Jan 09, 2023
172 Dec 22, 2022
12 Apr 30, 2022
9 Jan 02, 2023
118 Dec 25, 2022
1 Feb 09, 2022
0 Jul 26, 2021
694 Dec 23, 2022
50 Jul 12, 2022
10 Jun 11, 2022
68 Jan 04, 2023
262 Dec 31, 2022
0 Nov 19, 2021
17 Dec 30, 2022
643 Dec 30, 2022
193 Dec 26, 2022
1.4k Dec 29, 2022
10 Dec 08, 2022