This repository is the official implementation of the NeurIPS 2021 paper Robust Contrastive Learning Using Negative Samples with Diminished Semantics.
tl;dr Contrastive learning utilizes positive pairs which preserve semantic information while perturbing superficial features in the training images. Similarly, we propose to generate negative samples to make the model more robust, where only the superfluous instead of the semantic features are preserved.
- Updated scripts and added new checkpoints based on the hyperparameters identified by IFM on the ImageNet-100 dataset.
Install PyTorch and check preprocess/ for ImageNet-100 and ImageNet-Texture preprocessing scripts.
The following code is used to pre-train MoCo-v2 + patch / texture-based NS. The major code is developed with minimal modifications from the official implementation.
python moco-non-sem-neg.py -a resnet50 --lr 0.8 --batch-size 512 --moco-m 0.99 --dist-url 'tcp://localhost:10001' \
--multiprocessing-distributed --world-size 1 --rank 0 \
--mlp --moco-t 0.2 --aug-plus --cos --moco-k 16384 \
--robust patch --num-nonsem 1 --alpha 2 --epochs 200 --patch-ratio 16 72 \
--ckpt_path ./ckpts/mocov2_mocok16384_bs512_lr0.8_nonsem_16_72_noaug_nn1_alpha2_epoch200 \
/path/to/imagenet-100/
python moco-non-sem-neg.py -a resnet50 --lr 0.8 --batch-size 512 --moco-m 0.99 --dist-url 'tcp://localhost:10001' \
--multiprocessing-distributed --world-size 1 --rank 0 \
--mlp --moco-t 0.2 --aug-plus --cos --moco-k 16384 \
--robust texture --num-nonsem 1 --alpha 2 --epochs 200 \
--ckpt_path ./ckpts/mocov2_mocok16384_bs512_lr0.8_texture_nn1_alpha2_epoch200 \
/path/to/imagenet-100-texture/
- Change
/path/to/imagenet-100/with the ImageNet-100 dataset directory. - Change
--alphaand-moco-kto reproduce results with different configurations.
Run following code is used to reproduce MoCo-v2 + patch-based NS model reported in Table 1.
python main_lincls.py -a resnet50 --lr 10.0 --batch-size 128 --epochs 60 \
--pretrained ./ckpts/mocov2_mocok16384_bs128_lr0.03_nonsem_16_72_noaug_nn1_alpha2_epoch200/checkpoint_0199.pth.tar \
--dist-url 'tcp://localhost:10001' --multiprocessing-distributed --world-size 1 --rank 0 \
--ckpt_path ./ckpts/mocov2_mocok16384_bs128_lr0.03_nonsem_16_72_noaug_nn1_alpha2_epoch200 \
/path/to/imagenet-100/
You can download pretrained models here:
| k | α | ImageNet100 | Corruption | Sketch | Stylized | Rendition | Checkpoints | |
|---|---|---|---|---|---|---|---|---|
| MoCo | 16384 | - | 77.88±0.28 | 43.08±0.27 | 28.24±0.58 | 16.20±0.55 | 32.92±0.12 | R1, R2, R3 |
| +Texture | 16384 | 2 | 77.76±0.17 | 43.58±0.33 | 29.11±0.39 | 16.59±0.17 | 33.36±0.15 | R1, R2, R3 |
| +Patch | 16384 | 2 | 79.35±0.12 | 45.13±0.35 | 31.76±0.88 | 17.37±0.19 | 34.78±0.15 | R1, R2, R3 |
| +Patch | 16384 | 3 | 75.58±0.52 | 44.45±0.15 | 34.03±0.58 | 18.60±0.26 | 36.89±0.11 | R1, R2, R3 |
| MoCo | 8192 | - | 77.73±0.38 | 43.22±0.39 | 28.45±0.36 | 16.83±0.12 | 33.19±0.44 | R1, R2, R3 |
| +Patch | 8192 | 2 | 79.54±0.32 | 45.48±0.20 | 33.36±0.45 | 17.81±0.32 | 36.31±0.37 | R1, R2, R3 |
| MoCo* | 65536 | - | 80.00±0.14 | 45.15±0.42 | 30.38±0.30 | 16.68±0.39 | 30.38±0.30 | R1, R2, R3 |
| +Patch | 65536 | 2 | 81.18±0.09 | 46.74±0.32 | 32.46±0.55 | 17.63±0.14 | 36.66±0.18 | R1, R2, R3 |
| +Patch | 16394 | 2 | 81.49±0.11 | 47.48±0.20 | 34.20±0.40 | 17.95±0.41 | 38.45±0.19 | R1, R2, R3 |
* denotes training with the IFM hyperparameters.
@article{ge2021robust,
title={Robust Contrastive Learning Using Negative Samples with Diminished Semantics},
author={Ge, Songwei and Mishra, Shlok and Li, Chun-Liang and Wang, Haohan and Jacobs, David},
journal={Advances in Neural Information Processing Systems},
volume={34},
year={2021}
}