The official implementation of the IEEE S&P`22 paper "SoK: How Robust is Deep Neural Network Image Classification Watermarking".

Overview

Watermark-Robustness-Toolbox - Official PyTorch Implementation

contact Python 3.6 PyTorch 1.3.1 cuDNN 10.1.2 Website shields.io GPLv3 license

This repository contains the official PyTorch implementation of the following paper to appear at IEEE Security and Privacy 2022:

SoK: How Robust is Deep Neural Network Image Classification Watermarking?
Nils Lukas, Edward Jiang, Xinda Li, Florian Kerschbaum
https://arxiv.org/abs/2108.04974

Abstract: Deep Neural Network (DNN) watermarking is a method for provenance verification of DNN models. Watermarking should be robust against watermark removal attacks that derive a surrogate model that evades provenance verification. Many watermarking schemes that claim robustness have been proposed, but their robustness is only validated in isolation against a relatively small set of attacks. There is no systematic, empirical evaluation of these claims against a common, comprehensive set of removal attacks. This uncertainty about a watermarking scheme's robustness causes difficulty to trust their deployment in practice. In this paper, we evaluate whether recently proposed watermarking schemes that claim robustness are robust against a large set of removal attacks. We survey methods from the literature that (i) are known removal attacks, (ii) derive surrogate models but have not been evaluated as removal attacks, and (iii) novel removal attacks. Weight shifting, transfer learning and smooth retraining are novel removal attacks adapted to the DNN watermarking schemes surveyed in this paper. We propose taxonomies for watermarking schemes and removal attacks. Our empirical evaluation includes an ablation study over sets of parameters for each attack and watermarking scheme on the image classification datasets CIFAR-10 and ImageNet. Surprisingly, our study shows that none of the surveyed watermarking schemes is robust in practice. We find that schemes fail to withstand adaptive attacks and known methods for deriving surrogate models that have not been evaluated as removal attacks. This points to intrinsic flaws in how robustness is currently evaluated. Our evaluation includes a discussion of the runtime of each attack to underpin their practical relevance. While none of the schemes is robust against all attacks, none of the attacks removes all watermarks. We show that attacks can be combined and find combined attacks that remove all watermarks. We show that watermarking schemes need to be evaluated against a more extensive set of removal attacks with a more realistic adversary model. Our source code and a complete dataset of evaluation results will be made publicly available, which allows to independently verify our conclusions.

Features

All watermarking schemes and removal attacks are configured for the image classification datasets CIFAR-10 (32x32 pixels, 10 classes) and ImageNet (224x224 pixels, 1k classes). We implemented the following watermarking schemes, sorted by their categories:

.. and the following removal attacks, sorted by their categories:

Get Started

At this point, the Watermark-Robustness-Toolbox project is not available as a standalone pip package, but we are working on allowing an installation via pip. We describe a manual installation and usage. First, install all dependencies via pip.

$ pip install -r requirements.txt

The following four main scripts provide the entire toolbox's functionality:

  • train.py: Pre-trains an unmarked neural network.
  • embed.py: Embeds a watermark into a pre-trained neural network.
  • steal.py: Performs a removal attack against a watermarked neural network.
  • decision_threshold.py: Computes the decision threshold for a watermarking scheme.

We use the mlconfig library to pass configuration hyperparameters to each script. Configuration files used in our paper for CIFAR-10 and ImageNet can be found in the configs/ directory. Configuration files store all hyperparameters needed to reproduce an experiment.

Step 1: Pre-train a Model on CIFAR-10

$ python train.py --config configs/cifar10/train_configs/resnet.yaml

This creates an outputs directory and saves a model file at outputs/cifar10/null_models/resnet/.

Step 2: Embed an Adi Watermark

$ python embed.py --wm_config configs/cifar10/wm_configs/adi.yaml \
                  --filename outputs/cifar10/null_models/resnet/best.pth

This embeds an Adi watermark into the pre-trained model from 'Example 1' and saves (i) the watermarked model and (ii) all data to read the watermark under outputs/cifar10/wm/adi/00000_adi/.

Step 3: Attempt to Remove a Watermark

$ python steal.py --attack_config configs/cifar10/attack_configs/ftal.yaml \
                  --wm_dir outputs/cifar10/wm/adi/00000_adi/

This runs the Fine-Tuning (FTAL) removal attack against the watermarked model and creates a surrogate model stored under outputs/cifar10/attacks/ftal/. The directory also contains human-readable debug files, such as the surrogate model's watermark and test accuracies.

Datasets

Our toolbox currently implements custom data loaders (class WRTDataLoader) for the following datasets.

  • CIFAR-10
  • ImageNet (needs manual download)
  • Omniglot (needs manual download)
  • Open Images (needs manual download)

Documentation

We are actively working on documenting the parameters of each watermarking scheme and removal attack. At this point, we can only refer to the method's source code (at wrt/defenses/ and wrt/attacks/). Soon we will host a complete documentation of all parameters, so stay tuned!

Contribute

We encourage authors of watermarking schemes or removal attacks to implement their methods in the Watermark-Robustness-Toolbox to make them publicly accessible in a unified framework. Our aim is to improve reproducibility which makes it easier to evaluate a scheme's robustness. Any contributions or suggestions for improvements are welcome and greatly appreciated. This toolbox is maintained as part of a university project by graduate students.

Reference

The codebase has been based off an early version of the Adversarial-Robustness-Tooblox.

Cite our paper

@InProceedings{lukas2022watermarkingsok,
  title={SoK: How Robust is Deep Neural Network Image Classification Watermarking?}, 
  author={Lukas, Nils and Jiang, Edward and Li, Xinda and Kerschbaum, Florian},
  year={2022},
  booktitle={IEEE Symposium on Security and Privacy}
}
Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

============================================================================================================ `MILA will stop developing Theano https:

9.6k Jan 06, 2023
The source code and dataset for the RecGURU paper (WSDM 2022)

RecGURU About The Project Source code and baselines for the RecGURU paper "RecGURU: Adversarial Learning of Generalized User Representations for Cross

Chenglin Li 17 Jan 07, 2023
Official implementation of "OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association" in PyTorch.

openpifpaf Continuously tested on Linux, MacOS and Windows: New 2021 paper: OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Te

VITA lab at EPFL 50 Dec 29, 2022
kapre: Keras Audio Preprocessors

Kapre Keras Audio Preprocessors - compute STFT, ISTFT, Melspectrogram, and others on GPU real-time. Tested on Python 3.6 and 3.7 Why Kapre? vs. Pre-co

Keunwoo Choi 867 Dec 29, 2022
A simple program for training and testing vit

Vit This is a simple program for training and testing vit. Key requirements: torch, torchvision and timm. Dataset I put 5 categories of the cub classi

xiezhenyu 2 Oct 11, 2022
HEAM: High-Efficiency Approximate Multiplier Optimization for Deep Neural Networks

Approximate Multiplier by HEAM What's HEAM? HEAM is a general optimization method to generate high-efficiency approximate multipliers for specific app

4 Sep 11, 2022
Style-based Neural Drum Synthesis with GAN inversion

Style-based Drum Synthesis with GAN Inversion Demo TensorFlow implementation of a style-based version of the adversarial drum synth (ADS) from the pap

Sound and Music Analysis (SoMA) Group 29 Nov 19, 2022
Dynamic View Synthesis from Dynamic Monocular Video

Dynamic View Synthesis from Dynamic Monocular Video Project Website | Video | Paper Dynamic View Synthesis from Dynamic Monocular Video Chen Gao, Ayus

Chen Gao 139 Dec 28, 2022
Deep Sea Treasure Environment for Multi-Objective Optimization Research

DeepSeaTreasure Environment Installation In order to get started with this environment, you can install it using the following command: python3 -m pip

imec IDLab 6 Nov 14, 2022
Implementation of a Transformer using ReLA (Rectified Linear Attention)

ReLA (Rectified Linear Attention) Transformer Implementation of a Transformer using ReLA (Rectified Linear Attention). It will also contain an attempt

Phil Wang 49 Oct 14, 2022
CAMoE + Dual SoftMax Loss (DSL): Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

CAMoE + Dual SoftMax Loss (DSL): Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss This is official implement of "

程星 87 Dec 24, 2022
SlideGraph+: Whole Slide Image Level Graphs to Predict HER2 Status in Breast Cancer

SlideGraph+: Whole Slide Image Level Graphs to Predict HER2 Status in Breast Cancer A novel graph neural network (GNN) based model (termed SlideGraph+

28 Dec 24, 2022
Official Pytorch implementation of 6DRepNet: 6D Rotation representation for unconstrained head pose estimation.

6D Rotation Representation for Unconstrained Head Pose Estimation (Pytorch) Paper Thorsten Hempel and Ahmed A. Abdelrahman and Ayoub Al-Hamadi, "6D Ro

Thorsten Hempel 284 Dec 23, 2022
Code for the paper "Reinforced Active Learning for Image Segmentation"

Reinforced Active Learning for Image Segmentation (RALIS) Code for the paper Reinforced Active Learning for Image Segmentation Dependencies python 3.6

Arantxa Casanova 79 Dec 19, 2022
BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training

BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training By Likun Cai, Zhi Zhang, Yi Zhu, Li Zhang, Mu Li, Xiangyang Xue. This

290 Dec 29, 2022
An Open-Source Tool for Automatic Disease Diagnosis..

OpenMedicalChatbox An Open-Source Package for Automatic Disease Diagnosis. Overview Due to the lack of open source for existing RL-base automated diag

8 Nov 08, 2022
NVIDIA Deep Learning Examples for Tensor Cores

NVIDIA Deep Learning Examples for Tensor Cores Introduction This repository provides State-of-the-Art Deep Learning examples that are easy to train an

NVIDIA Corporation 10k Dec 31, 2022
Dynamic Capacity Networks using Tensorflow

Dynamic Capacity Networks using Tensorflow Dynamic Capacity Networks (DCN; http://arxiv.org/abs/1511.07838) implementation using Tensorflow. DCN reduc

Taeksoo Kim 8 Feb 23, 2021
This is an example of object detection on Micro bacterium tuberculosis using Mask-RCNN

Mask-RCNN on Mycobacterium tuberculosis This is an example of object detection on Mycobacterium Tuberculosis using Mask RCNN. Implement of Mask R-CNN

Jun-En Ding 1 Sep 16, 2021
Diffusion Normalizing Flow (DiffFlow) Neurips2021

Diffusion Normalizing Flow (DiffFlow) Reproduce setup environment The repo heavily depends on jam, a personal toolbox developed by Qsh.zh. The API may

76 Jan 01, 2023