Code to reproduce experiments in the paper "Explainability Requires Interactivity".

Overview

Explainability Requires Interactivity

This repository contains the code to train all custom models used in the paper Explainability Requires Interactivity as well as to create all static explanations (heat maps and generative). For our interactive framework, see the sister repositor.

Precomputed generative explanations are located at static_generative_explanations.

Requirements

Install the conda environment via conda env create -f env.yml (depending on your system you might need to change some versions, e.g. for pytorch, cudatoolkit and pytorch-lightning).

For some parts you will need the FairFace model, which can be downloaded from the authors' repo. You will only need the res34_fair_align_multi_7_20190809.pt file.

Training classification networks

CelebA dataset

You first need to download and decompress the CelebAMask-HQ dataset (or here). Then run the training with

python train.py --dset celeb --dset_path /PATH/TO/CelebAMask-HQ/ --classes_or_attr Smiling --target_path /PATH/TO/OUTPUT

/PATH/TO/FLOWERS102/ should contain a CelebAMask-HQ-attribute-anno.txt file and an CelebA-HQ-img directory. Any of the columns in CelebAMask-HQ-attribute-anno.txt can be used; in the paper we used Heavy_Makeup, Male, Smiling, and Young.

Flowers102 dataset

You first need to download and decompress the Flowers102 data. Then run the training with

python train.py --dset flowers102 --dset_path /PATH/TO/FLOWERS102/ --classes_or_attr 49-65 --target_path /PATH/TO/OUTPUT/

/PATH/TO/FLOWERS102/ should contain an imagelabels.mat file and an images directory. Classes 49 and 65 correspond to the "Oxeye daisy" and "California poppy", while 63 and 54 correspond to "Black-eyed Susan" and "Sunflower" as in the paper.

Generating heatmap explanations

Heatmap explanations are generated using the Captum library. After training, run explanations via

python static_exp.py --model_path /PATH/TO/MODEL.pt --img_path /PATH/TO/IMGS/ --model_name celeb --fig_dir /PATH/TO/OUTPUT/

/PATH/TO/IMGS/ contains (only) image files and can be omitted in order to run the default images exported by train.py. To run on FairFace, choose --model_name fairface and add --attr age or --attr gender. Other explanation methods can be easily added by modifying the explain_all function in static_exp.py. Explanations are saved to fig_dir. Only tested for the networks trained on the facial images data in the previous step, but any resnet18 with scalar output layer should work just as well.

Generating generative explanations

First, clone the original NVIDIA StyleGAN2-ada-pytorch repo. Make sure everything works as expected (e.g. run the getting started code). If the code is stuck at loading TODO, usually ctrl-C will let the model fall back to a smaller reference implementation which is good enough for our use case. Next, export the repo into your PYTHONPATH (e.g. via export PYTHONPATH=$PYTHONPATH:/PATH/TO/stylegan2-ada-pytorch/). To generate explanations, you will need to 0) train an image model (see above, or use the FairFace model); 1) create a dataset of latent codes + labels; 2) train a latent space logistic regression models; and 3) create the explanations. As each of the steps can be very slow, we split them up

Create labeled latent dataset

First, make sure to either train at least one image model as in the first step and/or download the FairFace model.

python generative_exp.py --phase 1 --attrs Smiling,ff-skin-color --base_dir /PATH/TO/BASE/ --generator_path /PATH/TO/STYLEGAN2.pkl --n_train 20000 --n_valid 5000

The base_dir is the directory where all files/sub-directories are stored and should be the same as the target_path from train.py (e.g., just .). It should contain e.g. the celeb-Smiling directory and the res34_fair_align_multi_7_20190809.pt file if using --attrs Smiling,ff-skin-color.

Train latent space model

After the first step, run

python generative_exp.py --phase 2 --attrs Smiling,ff-skin-color --base_dir /PATH/TO/BASE/ --epochs 50

with same base_dir and attrs.

Create generative explanations

Finally, you can generate generative explanations via

python generative_exp.py --phase 3 --base_dir /PATH/TO/BASE/ --eval_attr Smiling --generator_path /PATH/TO/STYLEGAN2.pkl --attrs Smiling,ff-skin-color --reconstruction_steps 1000 --ampl 0.09 --input_img_dir /PATH/TO/IMAGES/ --output_dir /PATH/TO/OUTPUT/

Here, eval_attr is the final evaluation model's class that you want to explain; attrs are the same as before, the directions in latent space; input_img_dir is a directory with (only) image files that are to be explained. Explanations are saved to output_dir.

Owner
Digital Health & Machine Learning
Digital Health & Machine Learning
Random Walk Graph Neural Networks

Random Walk Graph Neural Networks This repository is the official implementation of Random Walk Graph Neural Networks. Requirements Code is written in

Giannis Nikolentzos 38 Jan 02, 2023
A PyTorch implementation of the paper Mixup: Beyond Empirical Risk Minimization in PyTorch

Mixup: Beyond Empirical Risk Minimization in PyTorch This is an unofficial PyTorch implementation of mixup: Beyond Empirical Risk Minimization. The co

Harry Yang 121 Dec 17, 2022
Activating More Pixels in Image Super-Resolution Transformer

HAT [Paper Link] Activating More Pixels in Image Super-Resolution Transformer Xiangyu Chen, Xintao Wang, Jiantao Zhou and Chao Dong BibTeX @article{ch

XyChen 270 Dec 27, 2022
This repository contains all code and data for the Inside Out Visual Place Recognition task

Inside Out Visual Place Recognition This repository contains code and instructions to reproduce the results for the Inside Out Visual Place Recognitio

15 May 21, 2022
SuperSDR: multiplatform KiwiSDR + CAT transceiver integrator

SuperSDR SuperSDR integrates a realtime spectrum waterfall and audio receive from any KiwiSDR around the world, together with a local (or remote) cont

Marco Cogoni 30 Nov 29, 2022
Implementation of the ๐Ÿ˜‡ Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

HaloNet - Pytorch Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This re

Phil Wang 189 Nov 22, 2022
Face recognition project by matching the features extracted using SIFT.

MV_FaceDetectionWithSIFT Face recognition project by matching the features extracted using SIFT. By : Aria Radmehr Professor : Ali Amiri Dependencies

Aria Radmehr 4 May 31, 2022
A Blender python script for getting asset browser custom preview images for objects and collections.

asset_snapshot A Blender python script for getting asset browser custom preview images for objects and collections. Installation: Click the code butto

Johnny Matthews 44 Nov 29, 2022
The Official Repository for "Generalized OOD Detection: A Survey"

Generalized Out-of-Distribution Detection: A Survey 1. Overview This repository is with our survey paper: Title: Generalized Out-of-Distribution Detec

Jingkang Yang 338 Jan 03, 2023
Marine debris detection with commercial satellite imagery and deep learning.

Marine debris detection with commercial satellite imagery and deep learning. Floating marine debris is a global pollution problem which threatens mari

Inter Agency Implementation and Advanced Concepts 56 Dec 16, 2022
Barbershop: GAN-based Image Compositing using Segmentation Masks (SIGGRAPH Asia 2021)

Barbershop: GAN-based Image Compositing using Segmentation Masks Barbershop: GAN-based Image Compositing using Segmentation Masks Peihao Zhu, Rameen A

Peihao Zhu 928 Dec 30, 2022
Boostcamp AI Tech 3rd / Basic Paper reading w.r.t Embedding

Boostcamp AI Tech 3rd : Basic Paper Reading w.r.t Embedding TL;DR 1992๋…„๋ถ€ํ„ฐ 2018๋…„๋„๊นŒ์ง€ ์ด๋ฃจ์–ด์ง„ word/sentence embedding์˜ ์ค‘์š”ํ•œ ์ค„๊ธฐ๋ฅผ ์ด๋ฃจ๋Š” ๊ธฐ์ดˆ ๋…ผ๋ฌธ ์Šคํ„ฐ๋””๋ฅผ ์ง„ํ–‰ํ•˜๊ณ ์ž ํ•ฉ๋‹ˆ๋‹ค. ๋…ผ

Soyeon Kim 14 Nov 14, 2022
Understanding Convolution for Semantic Segmentation

TuSimple-DUC by Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cottrell. Introduction This repository is for Under

TuSimple 585 Dec 31, 2022
Official code for "Distributed Deep Learning in Open Collaborations" (NeurIPS 2021)

Distributed Deep Learning in Open Collaborations This repository contains the code for the NeurIPS 2021 paper "Distributed Deep Learning in Open Colla

Yandex Research 96 Sep 15, 2022
Implementation of our NeurIPS 2021 paper "A Bi-Level Framework for Learning to Solve Combinatorial Optimization on Graphs".

PPO-BiHyb This is the official implementation of our NeurIPS 2021 paper "A Bi-Level Framework for Learning to Solve Combinatorial Optimization on Grap

<a href=[email protected]"> 66 Nov 23, 2022
CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing

CapsuleVOS This is the code for the ICCV 2019 paper CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing. Arxiv Link: https://a

53 Oct 27, 2022
PED: DETR for Crowd Pedestrian Detection

PED: DETR for Crowd Pedestrian Detection Code for PED: DETR For (Crowd) Pedestrian Detection Paper PED: DETR for Crowd Pedestrian Detection Installati

36 Sep 13, 2022
A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

Website, Tutorials, and Docs โ€ƒโ€ƒ Uncertainty Toolbox A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualizatio

Uncertainty Toolbox 1.4k Dec 28, 2022
๐Ÿ›ฐ๏ธ List of earth observation companies and job sites

Earth Observation Companies & Jobs source Portals & Jobs Geospatial Geospatial jobs newsletter: ~biweekly newsletter with geospatial jobs by Ali Ahmad

Dahn 64 Dec 27, 2022
This is a Python wrapper for TA-LIB based on Cython instead of SWIG.

TA-Lib This is a Python wrapper for TA-LIB based on Cython instead of SWIG. From the homepage: TA-Lib is widely used by trading software developers re

John Benediktsson 7.3k Jan 03, 2023