Computer Vision Script to recognize first person motion, developed as final project for the course "Machine Learning and Deep Learning"

Last update: Jul 16, 2022

Related tags

Deep Learning Imprinting-the-Motion

Overview

Overview of The Code

BaseColab/MLDL_FPAR.pdf: it contains the full explanation of our work
Base Colab: it contains the base colab used to perform all the training required for the project
Script: it contains the scripts used in Colab for calling the correct module
module.py: python files

Imprinting The Motion

Abstract

First person action recognition (FPAR) task is one of the most challenging in action recognition field. Most of the existing works address this issue with two-stream architectures, where the visual appearance and the motion information of the object of interest, are exploited. In this paper, we use as starting point the Ego-RNN architecture with the addition of the Motion Segmentation (MS) auxiliary task. We propose the injection of a new branch in the architecture, in order to employ the motion information more effectively. This leads to have better predictions.

Our Architecture

The Action Recognition Block extracts important spatial and temporal information from the video with the exploitation of the ResNet-34 (mustard), Spatial Attention Layer (yellow) and ConvLSTM (orange). Moreover, it takes advantage from the auxiliary task of the Motion Prediction Block, by embedding its knowledge inside the first layers of the backbone. This is performed with a feedback branch (blue) that takes as input the features of the motion segmentation (MS) task (green). The Motion Prediction Block takes, as input, the appearance features from the layer 4 of the ResNet and tries to identifies which parts of the image are going to move

Results

Our architecture is able to further exploit the motion information provided by the motion segmentation by merging them with the appearance features in the first layers of the backbone. The result is a model that better focuses on the relevant elements for action recognition and this lead to the correct prediction (shake tea cup instead of stir spoon cup)

Computer Vision Script to recognize first person motion, developed as final project for the course "Machine Learning and Deep Learning"

Related tags

Overview

Overview of The Code

Imprinting The Motion

Abstract

Our Architecture

Results

Owner

Simone Papicchio

Controlling the MicriSpotAI robot from scratch

A Deep Learning Framework for Neural Derivative Hedging

Doosan robotic arm, simulation, control, visualization in Gazebo and ROS2 for Reinforcement Learning.

Code for the CVPR 2021 paper: Understanding Failures of Deep Networks via Robust Feature Extraction

Implementation of Stochastic Image-to-Video Synthesis using cINNs.

Yoloxkeypointsegment - An anchor-free version of YOLO, with a simpler design but better performance

Keyword-BERT: Keyword-Attentive Deep Semantic Matching

A python package to perform same transformation to coco-annotation as performed on the image.

Boost learning for GNNs from the graph structure under challenging heterophily settings. (NeurIPS'20)

Scribble-Supervised LiDAR Semantic Segmentation, CVPR 2022 (ORAL)

Scheduling BilinearRewards

Curating a dataset for bioimage transfer learning

A baseline code for VSPW

Funnels: Exact maximum likelihood with dimensionality reduction.

这是一个yolox-pytorch的源码，可以用于训练自己的模型。

Code for DeepCurrents: Learning Implicit Representations of Shapes with Boundaries

Python periodic table module

pyhsmm - library for approximate unsupervised inference in Bayesian Hidden Markov Models (HMMs) and explicit-duration Hidden semi-Markov Models (HSMMs), focusing on the Bayesian Nonparametric extensions, the HDP-HMM and HDP-HSMM, mostly with weak-limit approximations.

Lbl2Vec learns jointly embedded label, document and word vectors to retrieve documents with predefined topics from an unlabeled document corpus.

MoViNets PyTorch implementation: Mobile Video Networks for Efficient Video Recognition;