This is the code for our paper DAAIN: Detection of Anomalous and AdversarialInput using Normalizing Flows

Last update: Oct 12, 2022

Related tags

Overview

Merantix-Labs: DAAIN

This is the code for our paper DAAIN: Detection of Anomalous and Adversarial Input using Normalizing Flows which can be found at arxiv.

Assumptions

There are assumptions:

The training data PerturbedDataset makes some assumptions about the data:
- the ignore_index is 255
- num_classes = 19
- the images are resized with size == 512

Module Overview

A selection of the files with some pointers what to find where

├── configs                                   # The yaml configs
│   ├── activation_spaces
│   │   └── esp_net_256_512.yaml
│   ├── backbone
│   │   ├── esp_dropout.yaml
│   │   └── esp_net.yaml
│   ├── dataset_paths
│   │   ├── bdd100k.yaml
│   │   └── cityscapes.yaml
│   ├── data_creation.yaml                    # Used to create the training and testing data in one go
│   ├── detection_inference.yaml              # Used for inference
│   ├── detection_training.yaml               # Used for training
│   ├── esp_dropout_training.yaml             # Used to train the MC dropout baseline
│   └── paths.yaml
├── README.md                                 # This file!
├── requirements.in                           # The requirements
├── setup.py
└── src
   └── daain
       ├── backbones                          # Definitions of the backbones, currently only a slighlty modified version
       │   │                                  # of the ESPNet was tested
       │   ├── esp_dropout_net
       │   │   ├── esp_dropout_net.py
       │   │   ├── __init__.py
       │   │   ├── lightning_module.py
       │   │   └── trainer
       │   │       ├── criteria.py
       │   │       ├── data.py
       │   │       ├── dataset_collate.py
       │   │       ├── data_statistics.py
       │   │       ├── __init__.py
       │   │       ├── iou_eval.py
       │   │       ├── README.md
       │   │       ├── trainer.py            # launch this file to train the ESPDropoutNet
       │   │       ├── transformations.py
       │   │       └── visualize_graph.py
       │   └── esp_net
       │       ├── espnet.py                 # Definition of the CustomESPNet
       │       └── layers.py
       ├── baseline
       │   ├── maximum_softmax_probability.py
       │   ├── max_logit.py
       │   └── monte_carlo_dropout.py
       ├── config_schema
       ├── constants.py                      # Some constants, the last thing to refactor...
       ├── data                              # General data classes
       │   ├── datasets
       │   │   ├── bdd100k_dataset.py
       │   │   ├── cityscapes_dataset.py
       │   │   ├── labels
       │   │   │   ├── bdd100k.py
       │   │   │   ├── cityscape.py
       │   │   └── semantic_segmentation_dataset.py
       │   ├── activations_dataset.py        # This class loads the recorded activations
       │   └── perturbed_dataset.py          # This class loads the attacked images
       ├── model
       │   ├── aggregation_mode.py           # Not interesting for inference
       │   ├── classifiers.py                # All classifiers used are defined here
       │   ├── model.py                      # Probably the most important module. Check this for an example on how
       │   │                                 # to used the detection model and how to load the parts
       │   │                                 # (normalising_flow & classifier)
       │   └── normalising_flow
       │       ├── coupling_blocks
       │       │   ├── attention_blocks
       │       │   ├── causal_coupling_bock.py  # WIP
       │       │   └── subnet_constructors.py
       │       └── lightning_module.py
       ├── scripts
       │   └── data_creation.py              # Use this file to create the training and testing data
       ├── trainer                           # Trainer of the full detection model
       │   ├── data.py                       # Loading the data...
       │   └── trainer.py
       ├── utils                             # General utils
       └── visualisations                    # Visualisation helpers

Parts

In general the model consists of two parts:

Normalising FLow
Classifier / Scoring method

Both have to be trained separately, depending on the classifier. Some are parameter free (except for the threshold).

The general idea can be summarised:

Record the activations of the backbone model at specific locations during a forward pass.
Transform the recorded activations using a normalising flow and map them to a standard Gaussian for each variable.
Apply some simple (mostly distance based) classifier on the transformed activations to get the anomaly score.

Training & Inference Process

Generate perturbed and adversarial images. We do not provide code for this step.
Generate the activations using src/daain/scripts/data_creation.py
Train the detection model using src/daain/trainer/trainer.py
Use src/daain/model/model.py to load the trained model and use it to get the anomaly score (the probability that the input was anomalous).

This is the code for our paper DAAIN: Detection of Anomalous and AdversarialInput using Normalizing Flows

Related tags

Overview

Merantix-Labs: DAAIN

Assumptions

Module Overview

Parts

Training & Inference Process

Owner

Merantix

EAST for ICPR MTWI 2018 Challenge II (Text detection of network images)

Using computer vision method to recognize and calcutate the features of the architecture.

Code for the AAAI 2018 publication "SEE: Towards Semi-Supervised End-to-End Scene Text Recognition"

Erosion and dialation using structure element in OpenCV python

With the virtual keyboard, you can write on the real time images by combining the thumb and index fingers on the letter you want.

Code for the "Sensing leg movement enhances wearable monitoring of energy expenditure" paper.

list all open dataset about ocr.

Solution for Problem 1 by team codesquad for AIDL 2020. Uses ML Kit for OCR and OpenCV for image processing

PyTorch Re-Implementation of EAST: An Efficient and Accurate Scene Text Detector

A python script based on opencv and paddleocr, which can automatically pick up tasks, make cookies, and receive rewards in the Destiny 2 Dawning Oven

CNN+Attention+Seq2Seq

Assignment work with webcam

Program created with opencv that allows you to automatically count your repetitions on several fitness exercises.

Play the Namibian game of Owela against a terrible AI. Built using Django and htmx.

Python package for handwriting and sketching in Jupyter cells

📷 This repository is focused on having various feature implementation of OpenCV in Python.

Code for the paper: Fusformer: A Transformer-based Fusion Approach for Hyperspectral Image Super-resolution

A Python wrapper for Google Tesseract

Driver Drowsiness Detection with OpenCV & Dlib