Visual Interestingness
-
Refer to the project description for more details.
-
This code based on the following paper.
Chen Wang, Yuheng Qiu, Wenshan Wang, Yafei Hu, Seungchan Kim and Sebastian Scherer, Unsupervised Online Learning for Robotic Interestingness with Visual Memory, IEEE Transactions on Robotics (T-RO), 2021.
-
It is an extended version of the conference paper:
Chen Wang, Wenshan Wang, Yuheng Qiu, Yafei Hu, and Sebastian Scherer, Visual Memorability for Robotic Interestingness via Unsupervised Online Learning, European Conference on Computer Vision (ECCV), 2020.
-
If you want the original version, go to the ECCV Branch instead.
-
We also provide ROS wrapper for this project, you may go to interestingness_ros.
Install Dependencies
This version is tested in PyTorch 1.7
pip3 install -r requirements.txt
Long-term Learning
-
You may skip this step, if you download the pre-trained vgg16.pt into folder "saves".
-
Download coco dataset into folder [data-root]:
bash download_coco.sh [data-root] # replace [data-root] by your desired locationThe dataset will be look like:
data-root ├──coco ├── annotations │ ├── annotations_trainval2017 │ └── image_info_test2017 └── images ├── test2017 ├── train2017 └── val2017 -
Run
python3 longterm.py --data-root [data-root] --model-save saves/vgg16.pt # This requires a long time for training on single GPU. # Create a folder "saves" manually and a model named "ae.pt" will be saved.
Short-term Learning
-
Dowload the SubT front camera data (SubTF) and put into folder "data-root", so that it looks like:
data-root ├──SubTF ├── 0817-ugv0-tunnel0 ├── 0817-ugv1-tunnel0 ├── 0818-ugv0-tunnel1 ├── 0818-ugv1-tunnel1 ├── 0820-ugv0-tunnel1 ├── 0821-ugv0-tunnel0 ├── 0821-ugv1-tunnel0 ├── ground-truth └── train -
Run
python3 shortterm.py --data-root [data-root] --model-save saves/vgg16.pt --dataset SubTF --memory-size 100 --save-flag n100usage # This will read the previous model "ae.pt". # A new model "ae.pt.SubTF.n1000.mse" will be generated. -
You may skip this step, if you download the pre-trained vgg16.pt.SubTF.n100usage.mse into folder "saves".
On-line Learning
-
Run
python3 online.py --data-root [data-root] --model-save saves/vgg16.pt.SubTF.n100usage.mse --dataset SubTF --test-data 0 --save-flag n100usage # --test-data The sequence ID in the dataset SubTF, [0-6] is avaiable # This will read the trained model "vgg16.pt.SubTF.n100usage.mse" from short-term learning. -
Alternatively, you may test all sequences by running
bash test.sh -
This will generate results files in folder "results".
-
You may skip this step, if you download our generated results.
Evaluation
-
We follow the SubT tutorial for evaluation, simply run
python performance.py --data-root [data-root] --save-flag n100usage --category normal --delta 1 2 3 # mean accuracy: [0.64455275 0.8368784 0.92165116 0.95906876] python performance.py --data-root [data-root] --save-flag n100usage --category difficult --delta 1 2 4 # mean accuracy: [0.42088688 0.57836163 0.67878168 0.75491805] -
This will generate performance figures and create data curves for two categories in folder "performance".
Citation
@inproceedings{wang2020visual,
title={Visual memorability for robotic interestingness via unsupervised online learning},
author={Wang, Chen and Wang, Wenshan and Qiu, Yuheng and Hu, Yafei and Scherer, Sebastian},
booktitle={European Conference on Computer Vision (ECCV)},
year={2020},
organization={Springer}
}
@article{wang2021unsupervised,
title={Unsupervised Online Learning for Robotic Interestingness with Visual Memory},
author={Wang, Chen and Qiu, Yuheng and Wang, Wenshan and Hu, Yafei anad Kim, Seungchan and Scherer, Sebastian},
journal={IEEE Transactions on Robotics (T-RO)},
year={2021},
publisher={IEEE}
}
- Download conferencec version paper.
- Download journal version paper.
You may watch the following video to catch the idea of this work.
44 Dec 14, 2022
207 Jan 5, 2023
21 Nov 20, 2022
54 Dec 15, 2022
98 Dec 7, 2022
32 Dec 13, 2022
242 Dec 20, 2022
![[ECCV 2020] Gradient-Induced Co-Saliency Detection](https://github.com/zzhanghub/gicd/raw/master/img/GICD_LOGO.png){kind=logo}
35 Nov 25, 2022
85 Dec 22, 2022
12 Dec 18, 2022
395 Dec 30, 2022
2 Sep 01, 2022
1.3k Dec 26, 2022
47 Jan 03, 2023
3 Jun 03, 2021
297 Dec 17, 2022
14 Oct 31, 2022
29 Nov 13, 2022
310 Jan 03, 2023
115 Jan 02, 2023
7 Aug 16, 2022
8 Oct 03, 2022
2 Sep 29, 2022
50 Oct 08, 2021
177 Dec 12, 2022
16 Nov 14, 2022
838 Jan 01, 2023
494 Dec 30, 2022
6 Dec 13, 2022