Computer Vision Script to recognize first person motion, developed as final project for the course "Machine Learning and Deep Learning"

Last update: Jul 16, 2022

Related tags

Deep Learning Imprinting-the-Motion

Overview

Overview of The Code

BaseColab/MLDL_FPAR.pdf: it contains the full explanation of our work
Base Colab: it contains the base colab used to perform all the training required for the project
Script: it contains the scripts used in Colab for calling the correct module
module.py: python files

Imprinting The Motion

Abstract

First person action recognition (FPAR) task is one of the most challenging in action recognition field. Most of the existing works address this issue with two-stream architectures, where the visual appearance and the motion information of the object of interest, are exploited. In this paper, we use as starting point the Ego-RNN architecture with the addition of the Motion Segmentation (MS) auxiliary task. We propose the injection of a new branch in the architecture, in order to employ the motion information more effectively. This leads to have better predictions.

Our Architecture

The Action Recognition Block extracts important spatial and temporal information from the video with the exploitation of the ResNet-34 (mustard), Spatial Attention Layer (yellow) and ConvLSTM (orange). Moreover, it takes advantage from the auxiliary task of the Motion Prediction Block, by embedding its knowledge inside the first layers of the backbone. This is performed with a feedback branch (blue) that takes as input the features of the motion segmentation (MS) task (green). The Motion Prediction Block takes, as input, the appearance features from the layer 4 of the ResNet and tries to identifies which parts of the image are going to move

Results

Our architecture is able to further exploit the motion information provided by the motion segmentation by merging them with the appearance features in the first layers of the backbone. The result is a model that better focuses on the relevant elements for action recognition and this lead to the correct prediction (shake tea cup instead of stir spoon cup)

Computer Vision Script to recognize first person motion, developed as final project for the course "Machine Learning and Deep Learning"

Related tags

Overview

Overview of The Code

Imprinting The Motion

Abstract

Our Architecture

Results

Owner

Simone Papicchio

Multitask Learning Strengthens Adversarial Robustness

Neural style transfer as a class in PyTorch

ThunderGBM: Fast GBDTs and Random Forests on GPUs

RL and distillation in CARLA using a factorized world model

An alarm clock coded in Python 3 with Tkinter

TransNet V2: Shot Boundary Detection Neural Network

Resources for the Ki testnet challenge

Towards uncontrained hand-object reconstruction from RGB videos

Sequence to Sequence Models with PyTorch

Python Implementation of the CoronaWarnApp (CWA) Event Registration

Code and data for the paper "Hearing What You Cannot See"

Code used to generate the results appearing in "Train longer, generalize better: closing the generalization gap in large batch training of neural networks"

Source code for deep symbolic optimization.

Banglore House Prediction Using Flask Server (Python)

Must-read Papers on Physics-Informed Neural Networks.

Immortal tracker

PyTorch implementation and pretrained models for XCiT models. See XCiT: Cross-Covariance Image Transformer

GAN-based Matrix Factorization for Recommender Systems

DeepMind Alchemy task environment: a meta-reinforcement learning benchmark

A Streamlit component to render ECharts.