Visual Interestingness

• Refer to the project description for more details.

• This code based on the following paper.

  Chen Wang, Yuheng Qiu, Wenshan Wang, Yafei Hu, Seungchan Kim and Sebastian Scherer, Unsupervised Online Learning for Robotic Interestingness with Visual Memory, IEEE Transactions on Robotics (T-RO), 2021.

• It is an extended version of the conference paper:

  Chen Wang, Wenshan Wang, Yuheng Qiu, Yafei Hu, and Sebastian Scherer, Visual Memorability for Robotic Interestingness via Unsupervised Online Learning, European Conference on Computer Vision (ECCV), 2020.

• If you want the original version, go to the ECCV Branch instead.

• We also provide ROS wrapper for this project, you may go to interestingness_ros.

Install Dependencies

This version is tested in PyTorch 1.7

  pip3 install -r requirements.txt

Long-term Learning

• You may skip this step, if you download the pre-trained vgg16.pt into folder "saves".

• Download coco dataset into folder [data-root]:

  bash download_coco.sh [data-root] # replace [data-root] by your desired location
  

  The dataset will be look like:

  data-root
  ├──coco
     ├── annotations
     │   ├── annotations_trainval2017
     │   └── image_info_test2017
     └── images
         ├── test2017
         ├── train2017
         └── val2017
  

• Run

  python3 longterm.py --data-root [data-root] --model-save saves/vgg16.pt
  
  # This requires a long time for training on single GPU.
  # Create a folder "saves" manually and a model named "ae.pt" will be saved.
  

Short-term Learning

• Dowload the SubT front camera data (SubTF) and put into folder "data-root", so that it looks like:

  data-root
  ├──SubTF
     ├── 0817-ugv0-tunnel0
     ├── 0817-ugv1-tunnel0
     ├── 0818-ugv0-tunnel1
     ├── 0818-ugv1-tunnel1
     ├── 0820-ugv0-tunnel1
     ├── 0821-ugv0-tunnel0
     ├── 0821-ugv1-tunnel0
     ├── ground-truth
     └── train
  

• Run

  python3 shortterm.py --data-root [data-root] --model-save saves/vgg16.pt --dataset SubTF --memory-size 100 --save-flag n100usage
  
  # This will read the previous model "ae.pt".
  # A new model "ae.pt.SubTF.n1000.mse" will be generated.
  

• You may skip this step, if you download the pre-trained vgg16.pt.SubTF.n100usage.mse into folder "saves".

On-line Learning

• Run

    python3 online.py --data-root [data-root] --model-save saves/vgg16.pt.SubTF.n100usage.mse --dataset SubTF --test-data 0 --save-flag n100usage
  
    # --test-data The sequence ID in the dataset SubTF, [0-6] is avaiable
    # This will read the trained model "vgg16.pt.SubTF.n100usage.mse" from short-term learning.
  

• Alternatively, you may test all sequences by running

    bash test.sh
  

• This will generate results files in folder "results".

• You may skip this step, if you download our generated results.

Evaluation

• We follow the SubT tutorial for evaluation, simply run

  python performance.py --data-root [data-root] --save-flag n100usage --category normal --delta 1 2 3
  # mean accuracy: [0.64455275 0.8368784  0.92165116 0.95906876]
  
  python performance.py --data-root [data-root] --save-flag n100usage --category difficult --delta 1 2 4
  # mean accuracy: [0.42088688 0.57836163 0.67878168 0.75491805]
  

• This will generate performance figures and create data curves for two categories in folder "performance".

Citation

      @inproceedings{wang2020visual,
        title={Visual memorability for robotic interestingness via unsupervised online learning},
        author={Wang, Chen and Wang, Wenshan and Qiu, Yuheng and Hu, Yafei and Scherer, Sebastian},
        booktitle={European Conference on Computer Vision (ECCV)},
        year={2020},
        organization={Springer}
      }
      
      @article{wang2021unsupervised,
        title={Unsupervised Online Learning for Robotic Interestingness with Visual Memory},
        author={Wang, Chen and  Qiu, Yuheng and Wang, Wenshan and Hu, Yafei anad Kim, Seungchan and Scherer, Sebastian},
        journal={IEEE Transactions on Robotics (T-RO)},
        year={2021},
        publisher={IEEE}
      }

• Download conferencec version paper.
• Download journal version paper.

You may watch the following video to catch the idea of this work.