Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Last update: Sep 08, 2022

Overview

Cross-framework Python Package for Evaluation of Latent-based Generative Models

Latte

Latte (for LATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.

Installation

For developers working on local clone, cd to the repo and replace latte with .. For example, pip install .[tests]

pip install latte-metrics           # core (numpy only)
pip install latte-metrics[pytorch]  # with torchmetrics wrapper
pip install latte-metrics[keras]    # with tensorflow wrapper
pip install latte-metrics[tests]    # for testing

Running tests locally

pip install .[tests]
pytest tests/ --cov=latte

Example

Functional API

import latte
from latte.functional.disentanglement.mutual_info import mig
import numpy as np

latte.seed(42)

z = np.random.randn(16, 8)
a = np.random.randn(16, 2)

mutual_info_gap = mig(z, a, discrete=False, reg_dim=[4, 3])

Modular API

import latte
from latte.metrics.core.disentanglement import MutualInformationGap
import numpy as np

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.compute()

TorchMetrics API

import latte
from latte.metrics.torch.disentanglement import MutualInformationGap
import torch

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update(z, attributes)

mig_val = mig.compute()

Keras Metric API

import latte
from latte.metrics.keras.disentanglement import MutualInformationGap
from tensorflow import keras as tfk

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.result()

Documentation

https://latte.readthedocs.io/en/latest

Supported metrics

Metric	Latte Functional	Latte Modular	TorchMetrics	Keras Metric
Disentanglement Metrics
📝 Mutual Information Gap (MIG)	🧪	🧪	🧪	??
📝 Dependency-blind Mutual Information Gap (DMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Mutual Information Gap (XMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Latent Information Gap (DLIG)	🧪	🧪	🧪	🧪
📝 Separate Attribute Predictability (SAP)	🧪	🧪	🧪	🧪
📝 Modularity	🧪	🧪	🧪	🧪
📝 β-VAE Score	👀	👀	👀	👀
📝 FactorVAE Score	👀	👀	👀	👀
📝 DCI Score	👀	👀	👀	👀
📝 Interventional Robustness Score (IRS)	👀	👀	👀	👀
📝 Consistency	👀	👀	👀	👀
📝 Restrictiveness	👀	👀	👀	👀
Interpolatability Metrics
📝 Smoothness	🧪	🧪	🧪	🧪
📝 Monotonicity	🧪	🧪	🧪	🧪
📝 Latent Density Ratio	🕣	🕣	🕣	🕣
📝 Linearity	👀	👀	👀	👀

Bundled metric modules

🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric Bundle	Latte Functional	Latte Modular	TorchMetrics	Keras Metric	Included
Dependency-aware Disentanglement	🧪	🧪	🧪	🧪	MIG, DMIG, XMIG, DLIG
LIAD-based Interpolatability	🧪	🧪	🧪	🧪	Smoothness, Monotonicity

Cite

For individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.

If you find our package useful please cite our repository and arXiv preprint as

@article{
  watcharasupat2021latte,
  author = {Watcharasupat, Karn N. and Lee, Junyoung and Lerch, Alexander},
  title = {{Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models}},
  eprint={2112.10638},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url = {https://github.com/karnwatcharasupat/latte}
  doi = {10.5281/zenodo.5786402}
}

Comments

Documentation: Metric Descriptions

Might be nice to provide a short description for each metric in addition to the paper links. The readme might get too long with it, but either some doc in the repo or maybe on a github.io page?
type: documentation priority: high

opened by alexanderlerch 2
Add Smoothness and Monotonicity support
Smoothness

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

Monotonicity

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add Modularity support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add SAP support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add DMIG, DLIG, XMIG support
DMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

XMIG

[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

DLIG

[ x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Add MIG support
[x] Functional API

[x] implementation

[x] tests

[x] Base API

[x] implementation

[x] tests

[x] Torch API

[x] implementation

[x] tests

[x] Keras API

[x] implementation

[x] tests

type: enhancement
opened by karnwatcharasupat 0
Support issue for on-the-fly computation in TF2 graph mode
The current delegate-to-NumPy technique used in TF is only compatible with TF2 eager mode since Tensor.numpy() would not work in graph mode. As a result, graph-mode users will only be able to use Latte in the evaluation stage when the model weights are no longer changing but not on-the-fly during the training stage.

However, certain computation steps required for some metrics (especially MI-based ones) necessarily require scikit-learn ops and there is no (maintainable) way to create consistent TF mirrors of those functions.

One potential solution is to wrap the core functions in tf.numpy_function or tf.py_function but we will have to figure out a way to make the wrapper less painful to implement/maintain since the variable args/kwargs option currently used by the dtype converter is not allowed in these functions. A naive workaround would be to make a tf.numpy_function wrapper for every highest-possible level function with fixed args but this would be considered a last-resort solution.

Links:

https://www.tensorflow.org/api_docs/python/tf/numpy_function

https://www.tensorflow.org/api_docs/python/tf/py_function

type: enhancement priority: medium !! needs more brains !!
opened by karnwatcharasupat 3

Releases(v0.0.1-alpha5)

v0.0.1-alpha5(Jan 20, 2022)
What's Changed

Add contributing guide by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/16

[ADD] add example notebooks by @karnwatcharasupat in https://github.com/karnwatcharasupat/latte/pull/18

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha3...v0.0.1-alpha5
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha3(Dec 16, 2021)
Added metric bundle support

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha2...v0.0.1-alpha3
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha2(Dec 9, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-alpha1...v0.0.1-alpha2
Source code(tar.gz)
Source code(zip)
v0.0.1-alpha1(Dec 1, 2021)

Full Changelog: https://github.com/karnwatcharasupat/latte/compare/v0.0.1-dev5...v0.0.1-alpha1
Source code(tar.gz)
Source code(zip)

Owner

Karn Watcharasupat

Lab Cat 🐱🌈 | Audio Signal Processing Research Student. NTU EEE Class of 2022. Georgia Tech Music Tech Visiting Researcher.

GitHub Repository https://latte.readthedocs.io/en/latest

Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Related tags

Overview

Latte

Installation

Running tests locally

Example

Functional API

Modular API

TorchMetrics API

Keras Metric API

Documentation

Supported metrics

Bundled metric modules

Cite

Comments

Documentation: Metric Descriptions

Add Smoothness and Monotonicity support

Smoothness

Monotonicity

Add Modularity support

Add SAP support

Add DMIG, DLIG, XMIG support

DMIG

XMIG

DLIG

Add MIG support

Support issue for on-the-fly computation in TF2 graph mode

Releases(v0.0.1-alpha5)

v0.0.1-alpha5(Jan 20, 2022)

What's Changed

v0.0.1-alpha3(Dec 16, 2021)

v0.0.1-alpha2(Dec 9, 2021)

v0.0.1-alpha1(Dec 1, 2021)

Owner

Karn Watcharasupat

A simple pytorch pipeline for semantic segmentation.

A scikit-learn compatible neural network library that wraps PyTorch

PyGCL: Graph Contrastive Learning Library for PyTorch

Adversarial Reweighting for Partial Domain Adaptation

Code base for "On-the-Fly Test-time Adaptation for Medical Image Segmentation"

Official implementation of NLOS-OT: Passive Non-Line-of-Sight Imaging Using Optimal Transport (IEEE TIP, accepted)

Joint project of the duo Hacker Ninjas

Deep learning with TensorFlow and earth observation data.

Starter kit for getting started in the Music Demixing Challenge.

DISTIL: Deep dIverSified inTeractIve Learning.

[Official] Exploring Temporal Coherence for More General Video Face Forgery Detection(ICCV 2021)

[CVPR2022] Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos

PyTorch implementation of the paper: Long-tail Learning via Logit Adjustment

Github for the conference paper GLOD-Gaussian Likelihood OOD detector

An unofficial PyTorch implementation of a federated learning algorithm, FedAvg.

Set of models for classifcation of 3D volumes

PyTorch Implementation of PIXOR: Real-time 3D Object Detection from Point Clouds

Dilated Convolution with Learnable Spacings PyTorch

A system for quickly generating training data with weak supervision

A Marvelous ChatBot implement using PyTorch.