Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Last update: Sep 16, 2022

Related tags

Deep Learning AngularGrad

Overview

AngularGrad Optimizer

This repository contains the oficial implementation for AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks in PyTorch.

AngularGrad reduces the zig-zag effect in the optimization trajectory. Fluctuations are significantly smoothed, tracing a more direct path towards the minimum of the cost function.

If you have questions or suggestions, please feel free to open an issue. Please cite as:

@article{roy2021angulargrad,
  title={AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks},
  author={S.K. Roy, M.E. Paoletti, J.M. Haut, S.R. Dubey, P. Kar, A. Plaza and B.B. Chaudhuri},
  journal={arXiv preprint arXiv:2105.10190},
  year={2021}
}

In PyTorch:

optimizer = cosangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
<or>
optimizer = tanangulargrad(params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0)
Example:
optimizer = cosangulargrad(net.parameters())

Experiments

Experiments in the paper:

Analitycal

cd analitycal/
python main.py

CIFAR-10/100

cd cifar/
python main.py --dataset <cifar10/cifar100> --model <r18/r34/.../vgg16/d121> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py --dataset cifar10 --model r50 --alg cosangulargrad --lr 1e-3

Mini-ImageNet:

cd mini-imagenet/
wget URL dataset
python main.py DATADIR --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./split_mini/ --alg cosangulargrad --model r50 --lr 1e-3

Fine-Grained:

cd fine-grained/
wget URL datasets
python main.py DATADIR --dataset <dsetname> --alg <adam/sgd/.../cosangulargrad/tanangulargrad> --lr <float>
Example:
python main.py ./data/Car196/ --dataset cars --alg adam --lr 1e-3

Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

Related tags

Overview

AngularGrad Optimizer

Experiments

Owner

mario

Imaginaire - NVIDIA's Deep Imagination Team's PyTorch Library

Spatial Intention Maps for Multi-Agent Mobile Manipulation (ICRA 2021)

Learning Skeletal Articulations with Neural Blend Shapes

Official implementation of the paper "Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering"

A PyTorch Implementation of the Luna: Linear Unified Nested Attention

A minimal implementation of face-detection models using flask, gunicorn, nginx, docker, and docker-compose

PyTorch implementation of GLOM

This is the repository for CVPR2021 Dynamic Metric Learning: Towards a Scalable Metric Space to Accommodate Multiple Semantic Scales

Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

From Perceptron model to Deep Neural Network from scratch in Python.

An implementation of IMLE-Net: An Interpretable Multi-level Multi-channel Model for ECG Classification

Generative Art Using Neural Visual Grammars and Dual Encoders

Here is the diagnostic tool for BMVC 2021 paper Diagnosing Errors in Video Relation Detectors.

The Illinois repository for Climatehack (https://climatehack.ai/). We won 1st place!

Convert weight file.pth to weight file.blob

NLMpy - A Python package to create neutral landscape models

Moer Grounded Image Captioning by Distilling Image-Text Matching Model

Ensembling Off-the-shelf Models for GAN Training

Allows including an action inside another action (by preprocessing the Yaml file). This is how composite actions should have worked.

Accelerated Multi-Modal MR Imaging with Transformers