Training for 480 epoch on single GPU will take quite a lot of time. So I wanted to check if multi GPU training is possible in the current code. I tried --dist_mode with mpi and auto are not supported as global_utils.AutoGPU() is not defined. But looks like --dist_mode=horovod is supported. Can it be used to do multi GPU training? Is it possible to share sample command to do multi-gpu training?

Btw it took around 2.83 hrs to complete 1 epoch on my GTX 1080-TI, does this sound reasonable or too high? Typically I have seen on my machine (4 x GTX1080) around 15 mins per epoch for EfficientNet-ES using Ross wightman's timm repo.

From Figure 2. in the paper, it can be seen that deeper and wider network has higher Zen-Score. And Zen-Score positively correlates with model accuracy. So Deeper and wider network has higher accuracy, which is a well-known principle. Then whats' the meaning of Zen-Score?

Your proposed Zen-NAS is a very efficient way to search for neural network structures. I read your article and GitHub code carefully, and did my own search on your code, but one thing I found is that the network structure searched almost always repeats more times the deeper the network is the network block, and the first few layers of the network block are repeated once, for example, I used your code to search the structure of MNas (the search space has been changed according to MNas), MNas0.35 optimal structure: 

SuperConvK3BNRELU(3,16,2,1)SuperResMnasV1K3(16,8,1,16,1)SuperResMnasV3K3(8,8,2,8,3)SuperResMnasV3K5(8,16,2,8,3) SuperResMnasV6K5(16,32,2,16,3)SuperResMnasV6K3(32,32,1,32,2)SuperResMnasV6K5(32,64,2,32,4)SuperResMnasV6K3(64,112,1,64,1) SuperConvK1BNRELU(112,1280,1,1) but I searched with your architecture and the structure is as follows SuperConvK3BNRELU(3,8,2,1)SuperResMnasV1K3(8,8,1,8,1)SuperResMnasV3K5(8,16,2,8,1)SuperResMnasV3K5(16,24,2,8,1)SuperResMnasV3K5(24,64,2,40,1)SuperResMnasV3K5(64,24,1,48,1)SuperResMnasV3K5(24,64,2,176,4)SuperResMnasV3K5(64,48,1,256,5)SuperConvK1BNRELU(48,2048,1, 1) so the search out of the structure compared to the original structure is not very good, the structure is still the problem mentioned above, the search out of the network shallow block duplication for 1, only the deeper network has block duplication, also tested your code in the Flops400M,600M,900M, found the same problem, this is why?

hi, @MingLin-home , In paper and official code, retrain the searched model by the feature loss cost too much resource, the 1.2ms latency model is very difficult to train completely end to end. so whether the series model‘s log can be released?

Hello,

As I remember from the paper, your method works on Vanilla CNN. However, in algorithm 1, you just mentioned that residual connections are deleted.

It confused me a little bit, and I do not know your method can be applied on any CNN without residual connection or it only can be applied on Vanilla CNN.

Can I use your code on benchmarks such as NAS-Bench-201?

Dear researcher, Hello
I ran into some problems while running the code that I couldn't solve when I wanted to test CIFAR10
（python val_cifar.py --dataset cifar10 --gpu 0 --arch zennet_cifar10_model_size05M_res32）

（RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.）

The version I use is Torch1.7.1 Torchvision0.8.2 cudatoolkit10.2.89. May I ask if this is a problem with my operating environment? Or do I need to make changes in the code? Thanks a million, dear researcher.

First of all, thanks a lot for releasing the code for such a nice work. I have few doubts,

1. The seach_space_block_type_list_list in both files, search_space_IDW_fixfc.py as well as search_space_XXBL.py are same. Is it intended?

2. The script, Zen_NAS_ImageNet_flops400M.sh and Zen_NAS_ImageNet_latency0.2ms use same search space. But the paper mentions Zen_NAS_ImageNet_flops400M.sh using MB block and Zen_NAS_ImageNet_latency0.2ms uses botn block like Resnet50.

Thanks again.

Hi, everyone. I'm currently trying to reproduce your previous work "Neural Architecture Design for GPU-Efficient Networks", here is the repo link. After determining the model structure, such as "GENet_large", "GENet_small", etc. Can we refer to the training script(train_image_classification.py) in this repo to train the model and get the effect consistent with the paper description?

My block look like this.

BottleneckCSP(
  (cv1): Conv(
    (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (act): Mish()
  )
  (cv2): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (cv3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (cv4): Conv(
    (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (act): Mish()
  )
  (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (act): Mish()
  (m): Sequential(
    (0): Bottleneck(
      (cv1): Conv(
        (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act): Mish()
      )
      (cv2): Conv(
        (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act): Mish()
      )
    )
    (1): Bottleneck(
      (cv1): Conv(
        (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act): Mish()
      )
      (cv2): Conv(
        (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act): Mish()
      )
    )
  )
)

forward(self, x)

def forward(self, x):
        d = self.m(self.cv1(x))
        y1 = self.cv3(d)
        y2 = self.cv2(x)
        return self.cv4(self.act(self.bn(torch.cat((y1, y2), dim=1))))

Hello MingLin,

I found your NAS approach to be very interesting.

Did you ever try to combine the ZEN-Score with the NTK-score like the TE-NAS paper did? The TE-NAS paper suggests that combining the NTK-Score with a expressivity measure would improve the overall performance.

Thanks, SXK

Thank you for your great work.

I have tried zenscore and get great kendall tau in plain network.

I wonder if zenscore can apply to MobileNet-like Search Space?

Looking forward to your reply.

i follow the paper "Neural Architecture Design for GPU-Efficient Networks", and see "We also use ResNet-152 as teacher network to get soft-labels. The soft-label loss and the true-label loss are weighted 1:1." at page 8 how to use soft-labels，teach genet. like knowledge distillation? have any code in this repo?