BossNAS
This repository contains PyTorch evaluation code, retraining code and pretrained models of our paper: BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search. [pdf link]
Illustration of the Siamese supernets training with ensemble bootstrapping.
Illustration of the fabric-like Hybrid CNN-transformer Search Space with flexible down-sampling positions.
Our Results and Trained Models
-
Here is a summary of our searched models:
Model MAdds Steptime Top-1 (%) Top-5 (%) Url BossNet-T0 w/o SE 3.4B 101ms 80.5 95.0 checkpoint BossNet-T0 3.4B 115ms 80.8 95.2 checkpoint BossNet-T0^ 5.7B 147ms 81.6 95.6 same as above BossNet-T1 7.9B 156ms 81.9 95.6 checkpoint BossNet-T1^ 10.5B 165ms 82.2 95.7 same as above -
Here is a summary of architecture rating accuracy of our method:
Search space Dataset Kendall tau Spearman rho Pearson R MBConv ImageNet 0.65 0.78 0.85 NATS-Bench Ss Cifar10 0.53 0.73 0.72 NATS-Bench Ss Cifar100 0.59 0.76 0.79
Usage
1. Requirements
- Install PyTorch 1.7.0+ and torchvision 0.8.1+, for example:
conda install -c pytorch pytorch torchvision
- Install pytorch-image-models 0.3.2, for example:
pip install timm==0.3.2
-
Download the ImageNet dataset from http://image-net.org/, and move validation images to labeled subfolders
- To do this, you can use the following script: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.shvalprep.sh
2. Retrain or Evaluate our BossNet-T models
-
First, move to retraining code directory to perform Retraining or Evaluation.
cd HyTra_retrainingOur retraining code of BossNet-T is based on DeiT repository.
-
You can evaluate our BossNet-T models with the following command:
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T0-80_8.pth
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T1-81_9.pth
Please download our checkpoint files from the result table. Please change the
--nproc_per_nodeoption to suit your GPU numbers, and change the--data-path,--resumeand--input-sizeaccordingly. -
You can retrain our BossNet-T models with the following command:
Please change the
--nproc_per_nodeand--data-pathaccordingly. Note that the learning rate will be automatically scaled according to the GPU numbers and batchsize. We recommend training with 128 batchsize and 8 GPUs.python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
Architecture of our BossNet-T0
3. Evaluate architecture rating accuracy of BossNAS
-
You can get the ranking correlations of BossNAS on MBConv search space with the following commands:
cd MBConv_ranking python get_model_score_mbconv.py
- You can get the ranking correlations of BossNAS on NATS-Bench Ss with the following commands:
cd NATS_SS_ranking python get_model_score_nats.py
Citation
@article{li2021bossnas,
author = {Li, Changlin and
Tang, Tao and
Wang, Guangrun and
Peng, Jiefeng and
Wang, Bing and
Liang, Xiaodan and
Chang, Xiaojun},
title = {BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search},
journal = {arXiv:2103.12424},
year = 2021,
}
TODO
Searching code will be released later.
32 Dec 18, 2022
5 Oct 18, 2022
19 Dec 12, 2022
2.7k Jan 04, 2023
38 Aug 17, 2022
154 Dec 15, 2022
1.5k Jan 07, 2023
44 Nov 18, 2022
71 Dec 04, 2022
12 Nov 25, 2022
33 Dec 06, 2022
11 Jul 13, 2022
100 Dec 06, 2022
42 Dec 08, 2022
3 Jan 07, 2022
88 Dec 19, 2022
35 Aug 23, 2022
143 Sep 13, 2022
27 Aug 08, 2022
116 Jan 04, 2023