Pytorch implementation of the unsupervised object discovery method LOST.

Last update: Dec 25, 2022

Related tags

Overview

LOST

Pytorch implementation of the unsupervised object discovery method LOST. More details can be found in the paper:

Localizing Objects with Self-Supervised Transformers and no Labels [arXiv]
by Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet and Jean Ponce

If you use the LOST code or framework in your research, please consider citing:

@article{LOST,
   title = {Localizing Objects with Self-Supervised Transformers and no Labels},
   author = {Oriane Sim\'eoni and Gilles Puy and Huy V. Vo and Simon Roburin and Spyros Gidaris and Andrei Bursuc and Patrick P\'erez and Renaud Marlet and Jean Ponce},
   journal = {arXiv preprint arXiv:2109.14279},
   month = {09},
   year = {2021}
}

Installation

Dependencies

This code was implemented with python 3.7, PyTorch 1.7.1 and CUDA 10.2. Please install PyTorch. In order to install the additionnal dependencies, please launch the following command:

pip install -r requirements.txt

Install DINO

This method is based on DINO paper. The framework can be installed using the following commands:

> __init__.py; cd ../; ">

git clone https://github.com/facebookresearch/dino.git
cd dino; 
touch __init__.py
echo -e "import sys\nfrom os.path import dirname, join\nsys.path.insert(0, join(dirname(__file__), '.'))" >> __init__.py; cd ../;

The code was made using the commit ba9edd1 of DINO repo (please rebase if breakage).

Apply LOST to one image

Following are scripts to apply LOST to an image defined via the image_path parameter and visualize the predictions (pred), the maps of the Figure 2 in the paper (fms) and the visulization of the seed expansion (seed_expansion). Box predictions are also stored in the output directory given by parameter output_dir.

python main_lost.py --image_path examples/VOC07_000236.jpg --visualize pred
python main_lost.py --image_path examples/VOC07_000236.jpg --visualize fms
python main_lost.py --image_path examples/VOC07_000236.jpg --visualize seed_expansion

Launching on datasets

Following are the different steps to reproduce the results of LOST presented in the paper.

PASCAL-VOC

Please download the PASCAL VOC07 and PASCAL VOC12 datasets (link) and put the data in the folder datasets. There should be the two subfolders: datasets/VOC2007 and datasets/VOC2012. In order to apply lost and compute corloc results (VOC07 61.9, VOC12 64.0), please launch:

python main_lost.py --dataset VOC07 --set trainval
python main_lost.py --dataset VOC12 --set trainval

COCO

Please download the COCO dataset and put the data in datasets/COCO. Results are provided given the 2014 annotations following previous works. The following command line allows you to get results on the subset of 20k images of the COCO dataset (corloc 50.7), following previous litterature. To be noted that the 20k images are a subset of the train set.

python main_lost.py --dataset COCO20k --set train

Different models

We have tested the method on different setups of the VIT model, corloc results are presented in the following table (more can be found in the paper).

arch	pre-training	dataset
		VOC07	VOC12	COCO20k
ViT-S/16	DINO	61.9	64.0	50.7
ViT-S/8	DINO	55.5	57.0	49.5
ViT-B/16	DINO	60.1	63.3	50.0
ResNet50	DINO	36.8	42.7	26.5
ResNet50	Imagenet	33.5	39.1	25.5

Previous results on the dataset VOC07 can be obtained by launching:

python main_lost.py --dataset VOC07 --set trainval #VIT-S/16
python main_lost.py --dataset VOC07 --set trainval --patch_size 8 #VIT-S/8
python main_lost.py --dataset VOC07 --set trainval --arch vit_base #VIT-B/16
python main_lost.py --dataset VOC07 --set trainval --arch resnet50 #Resnet50/DINO
python main_lost.py --dataset VOC07 --set trainval --arch resnet50_imagenet #Resnet50/imagenet

Comments

Is LOST designed to perform well with DINO features specifically?

I've replaced LOST's backbone (basically the dino weights) with the ones in CLIP, and it did not work well. But when switching back to dino weights, both ViT and ResNet50 backbone could generate good feature maps. Why would this happen?
question

opened by zengyuy 3
Error in evaluation with Detectron2

Hi @osimeoni,

Thank you for making the code available!

When evaluating Detectron2 on VOC12 with the obtained pseudolables. I obtain the following error: AttributeError: "int object has no attribute 'value'. It seems that the coco_style_file is not registered by 'register_coco_instances' (see image underneath). Any idea how this can be fixed? Thanks.

opened by MarcVisions 2
Class-aware detection

Do you plan on releasing code for class-aware detection (i.e., to produce the results in Table 3 of https://arxiv.org/pdf/2109.14279.pdf)? I don't believe I see any of the necessary code for assigning object categories to boxes, but please correct me if I'm wrong.

opened by gholste 2
Multi-object discovery

HI, I have a confusion about the interesting work. How to perform multi-target discovery in the figure 1 (middle) of your paper？ Any advice is greatly appreciated.
question

opened by rgbd-zml 1
Lost not performing well using DINO with fine-tuning

I’ve trained DINO’s model with my own Dataset, doing a finetuning on the ViT’s pre trained models of DINO. After a feel experiments I noticed that, every time that a epoch of the DINO’s finetune ran, the loss of the training reduce, however the IoU (the validation metric that we are using) of the bounding boxes generated by the LOST algorithm gets worse. Can anyone explain me why this is happening and how can I fix it?

opened by ericyoshida 1
corLoc evaluation

Hi @osimeoni . I am suspicious about the corLoc evaluation part in the code. The corLoc for each image is true whenever one of the ground truth objects is hit! https://github.com/valeoai/LOST/blob/2b678aca89c18aa79c56ec3f6d4a0b979a91608d/main_lost.py#L311 What about other objects? Is it right?

opened by Mirsadeghi 1

Pytorch implementation of the unsupervised object discovery method LOST.

Related tags

Overview

LOST

Installation

Dependencies

Install DINO

Apply LOST to one image

Launching on datasets

PASCAL-VOC

COCO

Different models

Comments

Is LOST designed to perform well with DINO features specifically?

Error in evaluation with Detectron2

Class-aware detection

Multi-object discovery

Lost not performing well using DINO with fine-tuning

corLoc evaluation

Releases(models)

models(Feb 28, 2022)

Owner

Valeo.ai

IDM: An Intermediate Domain Module for Domain Adaptive Person Re-ID,

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

Evaluation toolkit of the informative tracking benchmark comprising 9 scenarios, 180 diverse videos, and new challenges.

Reading Group @mila-iqia on Computational Optimal Transport for Machine Learning Applications

Pytorch library for fast transformer implementations

DCT-Mask: Discrete Cosine Transform Mask Representation for Instance Segmentation

The Ludii general game system, developed as part of the ERC-funded Digital Ludeme Project.

Official implementation for the paper "SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization".

YolactEdge: Real-time Instance Segmentation on the Edge

Generating Anime Images by Implementing Deep Convolutional Generative Adversarial Networks paper

Resco: A simple python package that report the effect of deep residual learning

The code of Zero-shot learning for low-light image enhancement based on dual iteration

Rethinking the U-Net architecture for multimodal biomedical image segmentation

A code generator from ONNX to PyTorch code

HINet: Half Instance Normalization Network for Image Restoration

Learned image compression

A platform to display the carbon neutralization information for researchers, decision-makers, and other participants in the community.

AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation

PyTorch IPFS Dataset

SAGE: Sensitivity-guided Adaptive Learning Rate for Transformers