Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Last update: Sep 08, 2022

Overview

Cross-framework Python Package for Evaluation of Latent-based Generative Models

Latte

Latte (for LATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.

Installation

For developers working on local clone, cd to the repo and replace latte with .. For example, pip install .[tests]

pip install latte-metrics           # core (numpy only)
pip install latte-metrics[pytorch]  # with torchmetrics wrapper
pip install latte-metrics[keras]    # with tensorflow wrapper
pip install latte-metrics[tests]    # for testing

Running tests locally

pip install .[tests]
pytest tests/ --cov=latte

Example

Functional API

import latte
from latte.functional.disentanglement.mutual_info import mig
import numpy as np

latte.seed(42)

z = np.random.randn(16, 8)
a = np.random.randn(16, 2)

mutual_info_gap = mig(z, a, discrete=False, reg_dim=[4, 3])

Modular API

import latte
from latte.metrics.core.disentanglement import MutualInformationGap
import numpy as np

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.compute()

TorchMetrics API

import latte
from latte.metrics.torch.disentanglement import MutualInformationGap
import torch

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update(z, attributes)

mig_val = mig.compute()

Keras Metric API

import latte
from latte.metrics.keras.disentanglement import MutualInformationGap
from tensorflow import keras as tfk

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.result()

Documentation

https://latte.readthedocs.io/en/latest

Supported metrics

Metric	Latte Functional	Latte Modular	TorchMetrics	Keras Metric
Disentanglement Metrics
📝 Mutual Information Gap (MIG)	🧪	🧪	🧪	??
📝 Dependency-blind Mutual Information Gap (DMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Mutual Information Gap (XMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Latent Information Gap (DLIG)	🧪	🧪	🧪	🧪
📝 Separate Attribute Predictability (SAP)	🧪	🧪	🧪	🧪
📝 Modularity	🧪	🧪	🧪	🧪
📝 β-VAE Score	👀	👀	👀	👀
📝 FactorVAE Score	👀	👀	👀	👀
📝 DCI Score	👀	👀	👀	👀
📝 Interventional Robustness Score (IRS)	👀	👀	👀	👀
📝 Consistency	👀	👀	👀	👀
📝 Restrictiveness	👀	👀	👀	👀
Interpolatability Metrics
📝 Smoothness	🧪	🧪	🧪	🧪
📝 Monotonicity	🧪	🧪	🧪	🧪
📝 Latent Density Ratio	🕣	🕣	🕣	🕣
📝 Linearity	👀	👀	👀	👀

Bundled metric modules

🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric Bundle	Latte Functional	Latte Modular	TorchMetrics	Keras Metric	Included
Dependency-aware Disentanglement	🧪	🧪	🧪	🧪	MIG, DMIG, XMIG, DLIG
LIAD-based Interpolatability	🧪	🧪	🧪	🧪	Smoothness, Monotonicity

Cite

For individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.

If you find our package useful please cite our repository and arXiv preprint as

@article{
  watcharasupat2021latte,
  author = {Watcharasupat, Karn N. and Lee, Junyoung and Lerch, Alexander},
  title = {{Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models}},
  eprint={2112.10638},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url = {https://github.com/karnwatcharasupat/latte}
  doi = {10.5281/zenodo.5786402}
}

Comments

Documentation: Metric Descriptions

Might be nice to provide a short description for each metric in addition to the paper links. The readme might get too long with it, but either some doc in the repo or maybe on a github.io page?
type: documentation priority: high

opened by alexanderlerch 2
Add Smoothness and Monotonicity support
Smoothness

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

Monotonicity

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add Modularity support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add SAP support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add DMIG, DLIG, XMIG support
DMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

XMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

DLIG

[ x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add MIG support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Support issue for on-the-fly computation in TF2 graph mode
The current delegate-to-NumPy technique used in TF is only compatible with TF2 eager mode since Tensor.numpy() would not work in graph mode. As a result, graph-mode users will only be able to use Latte in the evaluation stage when the model weights are no longer changing but not on-the-fly during the training stage.

However, certain computation steps required for some metrics (especially MI-based ones) necessarily require scikit-learn ops and there is no (maintainable) way to create consistent TF mirrors of those functions.

One potential solution is to wrap the core functions in tf.numpy_function or tf.py_function but we will have to figure out a way to make the wrapper less painful to implement/maintain since the variable args/kwargs option currently used by the dtype converter is not allowed in these functions. A naive workaround would be to make a tf.numpy_function wrapper for every highest-possible level function with fixed args but this would be considered a last-resort solution.

Links:

https://www.tensorflow.org/api_docs/python/tf/numpy_function

https://www.tensorflow.org/api_docs/python/tf/py_function

type: enhancement priority: medium !! needs more brains !!
opened by karnwatcharasupat 3

Releases(v0.0.1-alpha5)

v0.0.1-alpha5(Jan 20, 2022)
What's Changed

Add contributing guide by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/16

[ADD] add example notebooks by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/18

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha3...v0.0.1-alpha5
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha3(Dec 16, 2021)
Added metric bundle support

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha2...v0.0.1-alpha3
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha2(Dec 9, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha1...v0.0.1-alpha2
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha1(Dec 1, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-dev5...v0.0.1-alpha1
Source code(tar.gz)
Source code(zip)

Owner

Karn Watcharasupat

Lab Cat 🐱🌈 | Audio Signal Processing Research Student. NTU EEE Class of 2022. Georgia Tech Music Tech Visiting Researcher.

GitHub Repository https://latte.readthedocs.io/en/latest

Txt2Xml tool will help you convert from txt COCO format to VOC xml format in Object Detection Problem.

TXT 2 XML All codes assume running from root directory. Please update the sys path at the beginning of the codes before running. Over View Txt2Xml too

4 Nov 24, 2022

DANet for Tabular data classification/ regression.

Deep Abstract Networks A pyTorch implementation for AAAI-2022 paper DANets: Deep Abstract Networks for Tabular Data Classification and Regression. Bri

55 Sep 14, 2022

SHIFT15M: multiobjective large-scale fashion dataset with distributional shifts

[arXiv] The main motivation of the SHIFT15M project is to provide a dataset that contains natural dataset shifts collected from a web service IQON, wh

138 Nov 24, 2022

The official PyTorch code for NeurIPS 2021 ML4AD Paper, "Does Thermal data make the detection systems more reliable?"

MultiModal-Collaborative (MMC) Learning Framework for integrating RGB and Thermal spectral modalities This is the official code for NeurIPS 2021 Machi

12 Nov 02, 2022

PyTorch implementation of "ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context" (INTERSPEECH 2020)

ContextNet ContextNet has CNN-RNN-transducer architecture and features a fully convolutional encoder that incorporates global context information into

24 Nov 24, 2022

SBINN: Systems-biology informed neural network

SBINN: Systems-biology informed neural network The source code for the paper M. Daneker, Z. Zhang, G. E. Karniadakis, & L. Lu. Systems biology: Identi

15 Nov 19, 2022

[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)

PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision Kehong Gong*, Bingbing Li*, Jianfeng Zhang*, Ta

256 Dec 28, 2022

the code for paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration"

EOW-Softmax This code is for the paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration". Accepted by ICCV21. Usage Commnd exa

36 Dec 02, 2022

⚾🤖⚾ Automatic baseball pitching overlay in realtime

⚾ Automatically overlaying pitch motion and trajectory with machine learning! This project takes your baseball pitching clips and automatically genera

240 Dec 05, 2022

Code for T-Few from "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning"

T-Few This repository contains the official code for the paper: "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learni

220 Dec 31, 2022

RMTD: Robust Moving Target Defence Against False Data Injection Attacks in Power Grids

RMTD: Robust Moving Target Defence Against False Data Injection Attacks in Power Grids Real-time detection performance. This repo contains the code an

0 Nov 10, 2021

Pytorch code for "DPFM: Deep Partial Functional Maps" - 3DV 2021 (Oral)

DPFM Code for "DPFM: Deep Partial Functional Maps" - 3DV 2021 (Oral) Installation This implementation runs on python = 3.7, use pip to install depend

29 Oct 03, 2022

Towards End-to-end Video-based Eye Tracking

Towards End-to-end Video-based Eye Tracking The code accompanying our ECCV 2020 publication and dataset, EVE. Authors: Seonwook Park, Emre Aksan, Xuco

76 Dec 12, 2022

Art Project "Schrödinger's Game of Life"

Repo of the project "Team Creative Quantum AI: Schrödinger's Game of Life" Installation new conda env: conda create --name qcml python=3.8 conda activ

2 Sep 15, 2022

A PyTorch implementation of "ANEMONE: Graph Anomaly Detection with Multi-Scale Contrastive Learning", CIKM-21

ANEMONE A PyTorch implementation of "ANEMONE: Graph Anomaly Detection with Multi-Scale Contrastive Learning", CIKM-21 Dependencies python==3.6.1 dgl==

30 Dec 14, 2022

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection

Behind the Curtain: Learning Occluded Shapes for 3D Object Detection Acknowledgement We implement our model, BtcDet, based on [OpenPcdet 0.3.0]. Insta

163 Dec 19, 2022

This repository collects 100 papers related to negative sampling methods.

Negative-Sampling-Paper This repository collects 100 papers related to negative sampling methods, covering multiple research fields such as Recommenda

119 Dec 29, 2022

Build an Amazon SageMaker Pipeline to Transform Raw Texts to A Knowledge Graph

Build an Amazon SageMaker Pipeline to Transform Raw Texts to A Knowledge Graph This repository provides a pipeline to create a knowledge graph from ra

3 Jan 01, 2022

ICCV2021: Code for 'Spatial Uncertainty-Aware Semi-Supervised Crowd Counting'

14 May 13, 2022

The code for paper Efficiently Solve the Max-cut Problem via a Quantum Qubit Rotation Algorithm

Quantum Qubit Rotation Algorithm Single qubit rotation gates $$ U(\Theta)=\bigotimes_{i=1}^n R_x (\phi_i) $$ QQRA for the max-cut problem This code wa

0 Oct 18, 2021

Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Related tags

Overview

Latte

Installation

Running tests locally

Example

Functional API

Modular API

TorchMetrics API

Keras Metric API

Documentation

Supported metrics

Bundled metric modules

Cite

Comments

Documentation: Metric Descriptions

Add Smoothness and Monotonicity support

Smoothness

Monotonicity

Add Modularity support

Add SAP support

Add DMIG, DLIG, XMIG support

DMIG

XMIG

DLIG

Add MIG support

Support issue for on-the-fly computation in TF2 graph mode