Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Last update: Sep 08, 2022

Overview

Visual Interestingness

Refer to the project description for more details.
This code based on the following paper.

Chen Wang, Yuheng Qiu, Wenshan Wang, Yafei Hu, Seungchan Kim and Sebastian Scherer, Unsupervised Online Learning for Robotic Interestingness with Visual Memory, IEEE Transactions on Robotics (T-RO), 2021.
It is an extended version of the conference paper:

Chen Wang, Wenshan Wang, Yuheng Qiu, Yafei Hu, and Sebastian Scherer, Visual Memorability for Robotic Interestingness via Unsupervised Online Learning, European Conference on Computer Vision (ECCV), 2020.
If you want the original version, go to the ECCV Branch instead.
We also provide ROS wrapper for this project, you may go to interestingness_ros.

Install Dependencies

This version is tested in PyTorch 1.7

  pip3 install -r requirements.txt

Long-term Learning

You may skip this step, if you download the pre-trained vgg16.pt into folder "saves".

Download coco dataset into folder [data-root]:

bash download_coco.sh [data-root] # replace [data-root] by your desired location

The dataset will be look like:

data-root
├──coco
   ├── annotations
   │   ├── annotations_trainval2017
   │   └── image_info_test2017
   └── images
       ├── test2017
       ├── train2017
       └── val2017

Run

python3 longterm.py --data-root [data-root] --model-save saves/vgg16.pt

# This requires a long time for training on single GPU.
# Create a folder "saves" manually and a model named "ae.pt" will be saved.

Short-term Learning

Dowload the SubT front camera data (SubTF) and put into folder "data-root", so that it looks like:

data-root
├──SubTF
   ├── 0817-ugv0-tunnel0
   ├── 0817-ugv1-tunnel0
   ├── 0818-ugv0-tunnel1
   ├── 0818-ugv1-tunnel1
   ├── 0820-ugv0-tunnel1
   ├── 0821-ugv0-tunnel0
   ├── 0821-ugv1-tunnel0
   ├── ground-truth
   └── train

Run

python3 shortterm.py --data-root [data-root] --model-save saves/vgg16.pt --dataset SubTF --memory-size 100 --save-flag n100usage

# This will read the previous model "ae.pt".
# A new model "ae.pt.SubTF.n1000.mse" will be generated.

You may skip this step, if you download the pre-trained vgg16.pt.SubTF.n100usage.mse into folder "saves".

On-line Learning

Run

  python3 online.py --data-root [data-root] --model-save saves/vgg16.pt.SubTF.n100usage.mse --dataset SubTF --test-data 0 --save-flag n100usage

  # --test-data The sequence ID in the dataset SubTF, [0-6] is avaiable
  # This will read the trained model "vgg16.pt.SubTF.n100usage.mse" from short-term learning.

Alternatively, you may test all sequences by running
```
  bash test.sh
```
This will generate results files in folder "results".
You may skip this step, if you download our generated results.

Evaluation

We follow the SubT tutorial for evaluation, simply run

python performance.py --data-root [data-root] --save-flag n100usage --category normal --delta 1 2 3
# mean accuracy: [0.64455275 0.8368784  0.92165116 0.95906876]

python performance.py --data-root [data-root] --save-flag n100usage --category difficult --delta 1 2 4
# mean accuracy: [0.42088688 0.57836163 0.67878168 0.75491805]

This will generate performance figures and create data curves for two categories in folder "performance".

Citation

      @inproceedings{wang2020visual,
        title={Visual memorability for robotic interestingness via unsupervised online learning},
        author={Wang, Chen and Wang, Wenshan and Qiu, Yuheng and Hu, Yafei and Scherer, Sebastian},
        booktitle={European Conference on Computer Vision (ECCV)},
        year={2020},
        organization={Springer}
      }
      
      @article{wang2021unsupervised,
        title={Unsupervised Online Learning for Robotic Interestingness with Visual Memory},
        author={Wang, Chen and  Qiu, Yuheng and Wang, Wenshan and Hu, Yafei anad Kim, Seungchan and Scherer, Sebastian},
        journal={IEEE Transactions on Robotics (T-RO)},
        year={2021},
        publisher={IEEE}
      }

Download conferencec version paper.
Download journal version paper.

You may watch the following video to catch the idea of this work.

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh

44 Dec 14, 2022

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

207 Jan 5, 2023

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Causality In Traffic Accident (Under Construction) Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020) Overview Data Prepa

21 Nov 20, 2022

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

54 Dec 15, 2022

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

98 Dec 7, 2022

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

32 Dec 13, 2022

SNE-RoadSeg in PyTorch, ECCV 2020

SNE-RoadSeg Introduction This is the official PyTorch implementation of SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentati

242 Dec 20, 2022

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Gradient-Induced Co-Saliency Detection Zhao Zhang*, Wenda Jin*, Jun Xu, Ming-Ming Cheng ⭐ Project Home » The official repo of the ECCV 2020 paper Grad

35 Nov 25, 2022

Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

85 Dec 22, 2022

Comments

Variable

https://github.com/wang-chen/interestingness/blob/6994d50bd47d14b617f34f5c36c1beaba03acfdc/test_interest.py#L94

I think using Variable() will just return a tensor object in the new pytorch version.

opened by haleqiu 2

Visual Memorability for Robotic Interestingness via Unsupervised Online Learning (ECCV 2020 Oral and TRO)

Related tags

Overview

Visual Interestingness

Install Dependencies

Long-term Learning

Short-term Learning

On-line Learning

Evaluation

Citation

You might also like...

Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

SNE-RoadSeg in PyTorch, ECCV 2020

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Code for Towards Streaming Perception (ECCV 2020) :car:

Comments

Variable

Releases(v2.0)

v2.0(Apr 12, 2021)

v1.0(Jun 19, 2020)

Owner

Chen Wang

Official PyTorch implementation of "Improving Face Recognition with Large AgeGaps by Learning to Distinguish Children" (BMVC 2021)

PRTR: Pose Recognition with Cascade Transformers

Repo for Photon-Starved Scene Inference using Single Photon Cameras, ICCV 2021

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

a pytorch implementation of auto-punctuation learned character by character

An official implementation of "Background-Aware Pooling and Noise-Aware Loss for Weakly-Supervised Semantic Segmentation" (CVPR 2021) in PyTorch.

Keras implementation of PersonLab for Multi-Person Pose Estimation and Instance Segmentation.

Implementation of the bachelor's thesis "Real-time stock predictions with deep learning and news scraping".

Group Fisher Pruning for Practical Network Compression(ICML2021)

Official Pytorch implementation of 'RoI Tanh-polar Transformer Network for Face Parsing in the Wild.'

This repository is for our paper Exploiting Scene Graphs for Human-Object Interaction Detection accepted by ICCV 2021.

Sample code from the Neural Networks from Scratch book.

Auto-Encoding Score Distribution Regression for Action Quality Assessment

The undersampled DWI image using Slice-Interleaved Diffusion Encoding (SIDE) method can be reconstructed by the UNet network.

A simple, unofficial implementation of MAE using pytorch-lightning

Computer Vision Script to recognize first person motion, developed as final project for the course "Machine Learning and Deep Learning"

The code for MM2021 paper "Multi-Level Counterfactual Contrast for Visual Commonsense Reasoning"

Flower - A Friendly Federated Learning Framework

📖 Deep Attentional Guided Image Filtering

source code for 'Finding Valid Adjustments under Non-ignorability with Minimal DAG Knowledge' by A. Shah, K. Shanmugam, K. Ahuja