The ImageNet-CoG Benchmark

Code repository for the ImageNet-CoG Benchmark introduced in the paper "Concept Generalization in Visual Representation Learning" (ICCV 2021). It contains code for reproducing all the experiments reported in the paper, as well as instructions on how to evaluate any custom model on the ImageNet-CoG Benchmark.

@InProceedings{sariyildiz2021conceptgeneralization,
    title={Concept Generalization in Visual Representation Learning},
    author={Sariyildiz, Mert Bulent and Kalantidis, Yannis and Larlus, Diane and Alahari, Karteek},
    booktitle={International Conference on Computer Vision},
    year={2021}
}

Contents of the Readme file:

• Prerequisites: Benchmark data and code
  • Installation
  • Data
• Evaluating a model on the ImageNet-CoG benchmark
  • CoG step-1: Model preparation
  • CoG step-2: Feature extraction
  • CoG step-3: Learning linear classifiers and testing

Prerequisites: Benchmark data and code

Installation

We developed the benchmark using:

• GCC 7.5.0
• Python 3.7.9
• PyTorch 1.7.1
• Torchvision 0.8.2
• CUDA 10.2.89
• Optuna 2.3.0
• YACS 0.1.8
• termcolor 1.1.0

We recommend creating a separate conda environment for the benchmark:

conda create -n cog python=3.7.9
conda activate cog
conda install -c pytorch pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.2
conda install -c conda-forge optuna=2.3
conda install termcolor
pip install yacs

Note: To reproduce the results for SimCLR-v2 and BYOL, you will also need TensorFlow, as well as the repos that contain code for converting the SimCLR-v2 and BYOL pre-trained models into PyTorch. More info can be found in the corresponding scripts under prepare_models/.

Data

To evaluate a model on the ImageNet-CoG benchmark you will need the following data:

• The full ImageNet dataset (IN-21K)
• The ILSVRC-2012 dataset (IN-1K)
• The CoG level files (for L₁, L₂, L₃, L₄, L₅)

The full ImageNet dataset (IN-21K)

The levels of ImageNet-CoG consist of a selection of 5K synsets in the full ImageNet. To download the full ImageNet, you need to create an account on the ImageNet website. Then, under the "Download" section, you will find the "Winter 2021" release (a tar file of size 1.1TB). Once you download it, extract it to folder <imagenet_images_root> (you will need this path later when extracting features). After extracting you should see the following folder structure, i.e., a separate folder of images per synset:

<imagenet_images_root>
...
|--- n07696728
|   |--- *.JPEG
|--- n11944954
|   |--- *.JPEG
...

The ILSVRC-2012 dataset (IN-1K)

To evaluate models on the pretraining dataset, you will need the ubiquitous ILSVRC-2012 subset of ImageNet, which we also refer to as IN-1K. It is also available on the ImageNet website. You will download two tar files for training (138GB) and validation (6.3GB) images, and extract them under <in1k_images_root> (again, you will need this path later). We expect the following folder structure for IN-1K:

<in1k_images_root>
...
|--- train
|   |--- n11939491
|       |--- *.JPEG
|   |--- n07836838
|       |--- *.JPEG
|--- val
|   |--- n11939491
|       |--- *.JPEG
|   |--- n07836838
|       |--- *.JPEG
...

The CoG level files

These are files that contain the concepts and data splits for ImageNet-CoG, and can be directly downloaded from the links below:

• cog_levels_mapping_file.pkl: List of ImageNet concept names for each ImageNet-CoG level (~100KB).
• cog_concepts_split_file.pkl: List of image filenames in the train and test splits for all 5000 ImageNet concepts in the CoG levels (~678MB).

(Note: If clicking on the file names does not open a pop-up window for download, try 1) entering the file URLs directly on the address bar of your browser, or 2) using wget by giving the file URLs as arguments.)

Evaluating a model on the ImageNet-CoG benchmark

After installing the required packages and downloading the ImageNet data, you can follow the steps below:

1. CoG step-1: Prepare the model you want to evaluate.
2. CoG step-2: Extract image features for IN-1K and the CoG levels using the frozen backbone of the model.
3. CoG step-3: Train linear classifiers on the pre-extracted features and measure accuracy.

CoG step-1: Model preparation

The ImageNet-CoG benchmark is designed to evaluate models that are pre-trained on IN-1K (the ILSVRC-2012 dataset).

Models evaluated in the paper

To reproduce the results for any of the models evaluated in our paper you can follow the instructions for downloading the their checkpoints. You will end up with all the checkpoints under the folder <models_root_dir>. Note that every model has a unique name that should be passed with the --model argument in all the scripts below. You can find the model names we used for all the models evaluated in our paper in this table.

Custom models

Follow these three steps to prepare your model for evaluation on the CoG benchmark.

1. Give a name to your model: Every model has a unique name that is passed with the --model=<model_name> argument in all the scripts below. The model name consists of two parts <model_name>="<model_title>_<architecture_name>". You need to give such a name to your model, e.g., <myModel_myArchitecture>.

2. Write a model loader function: To be able to extract features from your custom model, the load_pretrained_backbone() function must be able to load your pretrained model correctly. To do so, you will need to add a custom model loader function in model_loaders.py, that takes as input an arguments dict init_args and returns the backbone module and its forward function, e.g., backbone, forward = load_my_model(init_args)

3. Register your model: Add a new element in MODEL_LOADER_DICT in the model_loader_dict.py script specifying the name and loader function for your model, e.g.:

[...]
"myModel_myArchitecture": load_my_model,
[...]

CoG step-2: Feature extraction

In this step, you will extract image features for the 6 datasets (IN-1K and our CoG levels L₁, L₂, L₃, L₄, L₅). We provide for you a bash script to automatize feature extraction ./bash-scripts/extract_features.sh for a given model. As you notice, there are variables in this script that you need to set properly; Moreover, this script calls extract_features.py with the provided arguments for each of 6 datasets. In our GPU cluster, each feature extraction experiment takes ~60min using a V100 GPU and 8 CPU cores.

Please see below for the documentation of the arguments and several examples for extract_features.py.

python extract_features.py \
    --model="sup_resnet50" \
    --dataset="in1k" \
    --split="train" \
    --models_root_dir=<models_root_dir> \
    --in1k_images_root=<in1k_images_root>

python extract_features.py \
    --model="swav_resnet50" \
    --dataset="cog_l5" \
    --split="test" \
    --models_root_dir=<models_root_dir> \
    --imagenet_images_root=<imagenet_images_root> \
    --cog_levels_mapping_file=cog_levels_mapping_file.pkl \
    --cog_concepts_split_file=cog_concepts_split_file.pkl

CoG step-3: Learning linear classifiers and testing

After extracting features for the 6 datasets (for their both training and test sets), we train linear classifiers on them. We divide the classification experiments into two settings:

1. Learning with all available images: Training classifiers with all available training data for the concepts. (This setting is to reproduce the scores we report in Section 4.2.1 - Generalization to unseen concepts of our paper.) We train 5 classifiers with different seeds on each CoG level and IN-1K separately, then report their average score. So, for this setting, you will need to train 30 (6 datasets, 5 seeds) logistic regression classifiers using all available training data for the concepts. We provide for you a bash script ./bash-scripts/logreg.sh to automatize running 5 classifiers on a specific dataset (or on a specific training and test set features). There are variables in this script that you need to set properly. Then, this script calls logreg.py with the provided arguments for 5 seeds. Finally, it prints the average score of these 5 experiments. In our GPU cluster, each experiment takes ~75min using a V100 GPU and 8 CPU cores.

2. Few-shot: Training classifiers with N in {1, 2, 4, 8, 16, 32, 64 or 128} training samples per concept. (This setting is to reproduce the scores we report in Section 4.2.2 - How fast can models adapt to unseen concepts? of our paper.) We again train 5 classifiers with different seeds on each dataset separately, then report their average score. But this time, we also change the number of training samples per concept. So, for this setting, you will need to train 240 (8 x 6 x 5) logistic regression classifiers. We provide for you a bash script ./bash-scripts/fewshot.sh to automatize running 5 classifiers on a specific dataset for N in {1, 2, 4, 8, 16, 32, 64 or 128}. There are variables in this script that you need to set properly. Then this script calls logreg.py with the provided arguments for 5 seeds and each N value. Finally, it prints the average score for each N.

Note on hyper-parameter tuning. To minimize performance differences due to sub-optimal hyper-parameters, we use the Optuna hyperparameter optimization framework to tune the learning rate and weight decay hyper-parameters when training a classifier. We sample 30 learning rate and weight decay pairs and perform hyper-parameter tuning by partitioning the training set into two temporary subsets used for hyper-parameter training and validation. Once the optimal hyper-parameters are found, then we train the final classifier with these hyper-parameters on the complete training set and report the accuracy on the test set. We combine hyper-parameter tuning, classifier training and testing in a single script; one just needs to run the logreg.py script.

Please see below for the documentation of the arguments and several examples for logreg.py.

python logreg.py \
    --model="swav_resnet50" \
    --dataset="cog_l5" \
    --models_root_dir=<models_root_dir> \
    EVAL.SEED 55  # --> training the classifier with seed 55

python print_results.py \
    --logreg_root_dir="<models_root_dir>/swav/resnet50/cog_l5/eval_logreg"

python logreg.py \
    --model="swav_resnet50" \
    --dataset="cog_l5" \
    --models_root_dir=<models_root_dir> \
    CLF.N_SHOT 8 # --> training the classifier with 8 random training samples per concept