Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Last update: Sep 08, 2022

Overview

Visual Interestingness

Refer to the project description for more details.
This code based on the following paper.

Chen Wang, Yuheng Qiu, Wenshan Wang, Yafei Hu, Seungchan Kim and Sebastian Scherer, Unsupervised Online Learning for Robotic Interestingness with Visual Memory, IEEE Transactions on Robotics (T-RO), 2021.
It is an extended version of the conference paper:

Chen Wang, Wenshan Wang, Yuheng Qiu, Yafei Hu, and Sebastian Scherer, Visual Memorability for Robotic Interestingness via Unsupervised Online Learning, European Conference on Computer Vision (ECCV), 2020.
If you want the original version, go to the ECCV Branch instead.
We also provide ROS wrapper for this project, you may go to interestingness_ros.

Install Dependencies

This version is tested in PyTorch 1.7

  pip3 install -r requirements.txt

Long-term Learning

You may skip this step, if you download the pre-trained vgg16.pt into folder "saves".

Download coco dataset into folder [data-root]:

bash download_coco.sh [data-root] # replace [data-root] by your desired location

The dataset will be look like:

data-root
├──coco
   ├── annotations
   │   ├── annotations_trainval2017
   │   └── image_info_test2017
   └── images
       ├── test2017
       ├── train2017
       └── val2017

Run

python3 longterm.py --data-root [data-root] --model-save saves/vgg16.pt

# This requires a long time for training on single GPU.
# Create a folder "saves" manually and a model named "ae.pt" will be saved.

Short-term Learning

Dowload the SubT front camera data (SubTF) and put into folder "data-root", so that it looks like:

data-root
├──SubTF
   ├── 0817-ugv0-tunnel0
   ├── 0817-ugv1-tunnel0
   ├── 0818-ugv0-tunnel1
   ├── 0818-ugv1-tunnel1
   ├── 0820-ugv0-tunnel1
   ├── 0821-ugv0-tunnel0
   ├── 0821-ugv1-tunnel0
   ├── ground-truth
   └── train

Run

python3 shortterm.py --data-root [data-root] --model-save saves/vgg16.pt --dataset SubTF --memory-size 100 --save-flag n100usage

# This will read the previous model "ae.pt".
# A new model "ae.pt.SubTF.n1000.mse" will be generated.

You may skip this step, if you download the pre-trained vgg16.pt.SubTF.n100usage.mse into folder "saves".

On-line Learning

Run

  python3 online.py --data-root [data-root] --model-save saves/vgg16.pt.SubTF.n100usage.mse --dataset SubTF --test-data 0 --save-flag n100usage

  # --test-data The sequence ID in the dataset SubTF, [0-6] is avaiable
  # This will read the trained model "vgg16.pt.SubTF.n100usage.mse" from short-term learning.

Alternatively, you may test all sequences by running
```
  bash test.sh
```
This will generate results files in folder "results".
You may skip this step, if you download our generated results.

Evaluation

We follow the SubT tutorial for evaluation, simply run

python performance.py --data-root [data-root] --save-flag n100usage --category normal --delta 1 2 3
# mean accuracy: [0.64455275 0.8368784  0.92165116 0.95906876]

python performance.py --data-root [data-root] --save-flag n100usage --category difficult --delta 1 2 4
# mean accuracy: [0.42088688 0.57836163 0.67878168 0.75491805]

This will generate performance figures and create data curves for two categories in folder "performance".

Citation

      @inproceedings{wang2020visual,
        title={Visual memorability for robotic interestingness via unsupervised online learning},
        author={Wang, Chen and Wang, Wenshan and Qiu, Yuheng and Hu, Yafei and Scherer, Sebastian},
        booktitle={European Conference on Computer Vision (ECCV)},
        year={2020},
        organization={Springer}
      }
      
      @article{wang2021unsupervised,
        title={Unsupervised Online Learning for Robotic Interestingness with Visual Memory},
        author={Wang, Chen and  Qiu, Yuheng and Wang, Wenshan and Hu, Yafei anad Kim, Seungchan and Scherer, Sebastian},
        journal={IEEE Transactions on Robotics (T-RO)},
        year={2021},
        publisher={IEEE}
      }

Download conferencec version paper.
Download journal version paper.

You may watch the following video to catch the idea of this work.

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh

44 Dec 14, 2022

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

207 Jan 5, 2023

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Causality In Traffic Accident (Under Construction) Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020) Overview Data Prepa

21 Nov 20, 2022

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

54 Dec 15, 2022

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

98 Dec 7, 2022

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

32 Dec 13, 2022

SNE-RoadSeg in PyTorch, ECCV 2020

SNE-RoadSeg Introduction This is the official PyTorch implementation of SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentati

242 Dec 20, 2022

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Gradient-Induced Co-Saliency Detection Zhao Zhang*, Wenda Jin*, Jun Xu, Ming-Ming Cheng ⭐ Project Home » The official repo of the ECCV 2020 paper Grad

35 Nov 25, 2022

Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

85 Dec 22, 2022

Comments

Variable

https://github.com/wang-chen/interestingness/blob/6994d50bd47d14b617f34f5c36c1beaba03acfdc/test_interest.py#L94

I think using Variable() will just return a tensor object in the new pytorch version.

opened by haleqiu 2

Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Related tags

Overview

Visual Interestingness

Install Dependencies

Long-term Learning

Short-term Learning

On-line Learning

Evaluation

Citation

You might also like...

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

SNE-RoadSeg in PyTorch, ECCV 2020

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Code for Towards Streaming Perception (ECCV 2020) :car:

Comments

Variable

Releases(v2.0)

v2.0(Apr 12, 2021)

v1.0(Jun 19, 2020)

Owner

Chen Wang

ImageNet-CoG is a benchmark for concept generalization. It provides a full evaluation framework for pre-trained visual representations which measure how well they generalize to unseen concepts.

Ludwig is a toolbox that allows to train and evaluate deep learning models without the need to write code.

Based on the paper "Geometry-aware Instance-reweighted Adversarial Training" ICLR 2021 oral

Pretrained Cost Model for Distributed Constraint Optimization Problems

Bootstrapped Unsupervised Sentence Representation Learning (ACL 2021)

PyTorch implementation of EigenGAN

A Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities

Qt-GUI implementation of the YOLOv5 algorithm (ver.6 and ver.5)

BlueFog Tutorials

[CVPR'20] TTSR: Learning Texture Transformer Network for Image Super-Resolution

RM Operation can equivalently convert ResNet to VGG, which is better for pruning; and can help RepVGG perform better when the depth is large.

A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis

CT-Net: Channel Tensorization Network for Video Classification

A curated list of awesome papers for Semantic Retrieval (TOIS Accepted: Semantic Models for the First-stage Retrieval: A Comprehensive Review).

🧮 Matrix Factorization for Collaborative Filtering is just Solving an Adjoint Latent Dirichlet Allocation Model after All

The codes I made while I practiced various TensorFlow examples

This is an official repository of CLGo: Learning to Predict 3D Lane Shape and Camera Pose from a Single Image via Geometry Constraints

This repository is for our EMNLP 2021 paper "Automated Generation of Accurate & Fluent Medical X-ray Reports"

Keras-1D-ACGAN-Data-Augmentation

Code for CVPR2021 paper "Robust Reflection Removal with Reflection-free Flash-only Cues"