ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Last update: Dec 08, 2022

Related tags

Overview

[ 👷 🏗 👷 🏗 Coming soon! Official release with improved docs. Stay tuned. 👷 🏗 👷 🏗 ]

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

[]

ViViT is a collection of numerical tricks to efficiently access curvature from the generalized Gauss-Newton (GGN) matrix based on its low-rank structure. Provided functionality includes computing

GGN eigenvalues
GGN eigenpairs (eigenvalues + eigenvector)
1ˢᵗ- and 2ⁿᵈ-order directional derivatives along GGN eigenvectors
Newton steps

These operations can also further approximate the GGN to reduce cost via sub-sampling, Monte-Carlo approximation, and block-diagonal approximation.

How does it work? ViViT uses and extends BackPACK for PyTorch. The described functionality is realized through a combination of existing and new BackPACK extensions and hooks into its backpropagation.

Installation

👷 🏗 👷 🏗 The PyPI release is coming soon. 👷 🏗 👷 🏗

For now, you need to install from GitHub via

pip install vivit-for-pytorch@git+https://github.com/f-dangel/vivit.git#egg=vivit-for-pytorch

Examples

👷 🏗 👷 🏗 Coming soon! 👷 🏗 👷 🏗

How to cite

If you are using ViViT, consider citing the paper

@misc{dangel2022vivit,
      title={{ViViT}: Curvature access through the generalized Gauss-Newton's low-rank structure},
      author={Felix Dangel and Lukas Tatzel and Philipp Hennig},
      year={2022},
      eprint={2106.02624},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Comments

[ADD] Warn about instabilities if eigenvalues are small

The directional gradient computation and transformation of the Newton step from Gram space into parameter space require division by the square root of the direction's eigenvalue. This is unstable if the eigenvalue is close to zero.

opened by f-dangel 1
[ADD] Clean `DirectionalDampedNewtonComputation`
Adds directionally damped Newton step computation with cleaned up API.

Fixes a bug in the eigenvalue criterion in the tests. It always picked one more eigenvalue than specified.
opened by f-dangel 1
[DOC] Add NTK example

Adds an example inspired by the functorch tutorial on NTKs. It demonstrates how to use vivit to compute empirical NTK matrices and makes a comparison with the functorch implementation.

opened by f-dangel 1
[ADD] Simplify `DirectionalDerivatives` API
Exotic features, like using different GGNs to compute directions and directional curvatures, as well as full control of which intermediate buffers to keep, have been deprecated in favor of a simpler API.

Remove Newton step computation for now as it was internally relying on DirectionalDerivatives

Remove many utilities and associated tests from the exotic features

Forbid duplicate indices in subsampling

Always delete intermediate buffers other than the target quantities
opened by f-dangel 1
[DOC] Set up `sphinx` and RTD

This PR adds a scaffold for the doc at https://vivit.readthedocs.io/en/latest/. Code examples are integrated via sphinx-gallery (I added a preliminary logo). Pull requests are built by the CI.

To build the docs, run make docs. You need to install the dependencies first, for example using pip install -e .[docs].

opened by f-dangel 1
Calculate Parameter Space Values of GGN Eigenvectors

The docs show how to calculate the gram matrix eigenvectors and the paper articulates that to translate from 'gram space' to parameter space we just need to multiply by the 'V' matrix.

What's the easiest way of implementing this?
question

opened by lk-wq 1
Detect loss function's `reduction`, error if unsupported
For now, the library only supports reduction='mean'. We rely on the user to use this reduction and raise awareness about this point in the documentation. It would be better to automatically have the library detect the reduction and error if it is unsupported.

This can be done via a hook into BackPACK.

[ ] Implement hook that determines the loss function reduction during backpropagation

[ ] Integrate the above hook into the *Computation and raise an exception if the reduction is not supported

[ ] Remove the comments about supported reductions in the documentation

enhancement
opened by f-dangel 0

Releases(1.0.0)

1.0.0(Jun 22, 2022)

First public release. Details about future releases will be documented in the changelog.
Source code(tar.gz)
Source code(zip)

Owner

Felix Dangel

Machine Learning PhD student at the University of Tübingen and the Max Planck Institute for Intelligent Systems.

GitHub Repository https://arxiv.org/abs/2106.02624

Image transformations designed for Scene Text Recognition (STR) data augmentation. Published at ICCV 2021 Workshop on Interactive Labeling and Data Augmentation for Vision.

Data Augmentation for Scene Text Recognition (ICCV 2021 Workshop) (Pronounced as "strog") Paper Arxiv Why it matters? Scene Text Recognition (STR) req

152 Dec 28, 2022

なりすまし検出(anti-spoof-mn3)のWebカメラ向けデモ

FaceDetection-Anti-Spoof-Demo なりすまし検出(anti-spoof-mn3)のWebカメラ向けデモです。モデルはPINTO_model_zoo/191_anti-spoof-mn3からONNX形式のモデルを使用しています。 Requirement mediapipe

8 Nov 18, 2022

Plug and play transformer you can find network structure and official complete code by clicking List

Plug-and-play Module Plug and play transformer you can find network structure and official complete code by clicking List The following is to quickly

8 Mar 27, 2022

Attention-guided gan for synthesizing IR images

SI-AGAN Attention-guided gan for synthesizing IR images This repository contains the Tensorflow code for "Pedestrian Gender Recognition by Style Trans

1 Oct 25, 2021

The official implementation of "Rethink Dilated Convolution for Real-time Semantic Segmentation"

RegSeg The official implementation of "Rethink Dilated Convolution for Real-time Semantic Segmentation" Paper: arxiv D block Decoder Setup Install the

61 Dec 27, 2022

NER for Indian languages

CL-NERIL: A Cross-Lingual Model for NER in Indian Languages Code for the paper - https://arxiv.org/abs/2111.11815 Setup Setup a virtual environment Th

0 Nov 24, 2021

A Pytorch implementation of "LegoNet: Efficient Convolutional Neural Networks with Lego Filters" (ICML 2019).

LegoNet This code is the implementation of ICML2019 paper LegoNet: Efficient Convolutional Neural Networks with Lego Filters Run python train.py You c

140 Sep 26, 2022

Awesome-google-colab - Google Colaboratory Notebooks and Repositories

Unofficial Google Colaboratory Notebook and Repository Gallery Please contact me to take over and revamp this repo (it gets around 30k views and 200k

1.2k Jan 03, 2023

Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

Hurdles to Progress in Long-form Question Answering This repository contains the official scripts and datasets accompanying our NAACL 2021 paper, "Hur

41 Nov 08, 2022

GEA - Code for Guided Evolution for Neural Architecture Search

Efficient Guided Evolution for Neural Architecture Search Usage Create a conda e

6 Jan 03, 2023

Predicting future trajectories of people in cameras of novel scenarios and views.

Pedestrian Trajectory Prediction Predicting future trajectories of pedestrians in cameras of novel scenarios and views. This repository contains the c

8 Sep 03, 2022

Reproducing Results from A Hybrid Approach to Targeting Social Assistance

title author date output Reproducing Results from A Hybrid Approach to Targeting Social Assistance Lendie Follett and Heath Henderson 12/28/2021 html_

0 Jan 06, 2022

[ICML 2021] A fast algorithm for fitting robust decision trees.

GROOT: Growing Robust Trees Growing Robust Trees (GROOT) is an algorithm that fits binary classification decision trees such that they are robust agai

17 Nov 21, 2022

DuBE: Duple-balanced Ensemble Learning from Skewed Data

DuBE: Duple-balanced Ensemble Learning from Skewed Data "Towards Inter-class and Intra-class Imbalance in Class-imbalanced Learning" (IEEE ICDE 2022 S

6 Nov 12, 2022

A task-agnostic vision-language architecture as a step towards General Purpose Vision

Towards General Purpose Vision Systems By Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem Overview Welcome to the official code base f

79 Dec 23, 2022

Face recognize and crop them

Face Recognize Cropping Module Source 아이디어 Face Alignment with OpenCV and Python Requirement 필요 라이브러리 imutil dlib python-opence (cv2) Usage 사용 방법 open

1 Feb 15, 2022

Locally Most Powerful Bayesian Test for Out-of-Distribution Detection using Deep Generative Models

LMPBT Supplementary code for the Paper entitled ``Locally Most Powerful Bayesian Test for Out-of-Distribution Detection using Deep Generative Models"

1 Sep 29, 2022

Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0

OpenGaze: Web Service for OpenFace Facial Behaviour Analysis Toolkit Overview OpenFace is a fantastic tool intended for computer vision and machine le

4 Nov 03, 2022

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

DeeBERT This is the code base for the paper DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference. Code in this repository is also available

132 Nov 14, 2022

A unified framework for machine learning with time series

Welcome to sktime A unified framework for machine learning with time series We provide specialized time series algorithms and scikit-learn compatible

6k Jan 08, 2023

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Related tags

Overview

ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure

Installation

Examples

How to cite

Comments

[ADD] Warn about instabilities if eigenvalues are small

[ADD] Clean `DirectionalDampedNewtonComputation`

[DOC] Add NTK example

[ADD] Simplify `DirectionalDerivatives` API

[DOC] Set up `sphinx` and RTD

Calculate Parameter Space Values of GGN Eigenvectors

Detect loss function's `reduction`, error if unsupported

Releases(1.0.0)

1.0.0(Jun 22, 2022)

Owner

Felix Dangel

Image transformations designed for Scene Text Recognition (STR) data augmentation. Published at ICCV 2021 Workshop on Interactive Labeling and Data Augmentation for Vision.

なりすまし検出(anti-spoof-mn3)のWebカメラ向けデモ

Plug and play transformer you can find network structure and official complete code by clicking List

Attention-guided gan for synthesizing IR images

The official implementation of "Rethink Dilated Convolution for Real-time Semantic Segmentation"

NER for Indian languages

A Pytorch implementation of "LegoNet: Efficient Convolutional Neural Networks with Lego Filters" (ICML 2019).

Awesome-google-colab - Google Colaboratory Notebooks and Repositories

Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

GEA - Code for Guided Evolution for Neural Architecture Search

Predicting future trajectories of people in cameras of novel scenarios and views.

Reproducing Results from A Hybrid Approach to Targeting Social Assistance

[ICML 2021] A fast algorithm for fitting robust decision trees.

DuBE: Duple-balanced Ensemble Learning from Skewed Data

A task-agnostic vision-language architecture as a step towards General Purpose Vision

Face recognize and crop them

Locally Most Powerful Bayesian Test for Out-of-Distribution Detection using Deep Generative Models

Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

A unified framework for machine learning with time series