The official implementation of the IEEE S&P`22 paper "SoK: How Robust is Deep Neural Network Image Classification Watermarking".

Overview

Watermark-Robustness-Toolbox - Official PyTorch Implementation

contact Python 3.6 PyTorch 1.3.1 cuDNN 10.1.2 Website shields.io GPLv3 license

This repository contains the official PyTorch implementation of the following paper to appear at IEEE Security and Privacy 2022:

SoK: How Robust is Deep Neural Network Image Classification Watermarking?
Nils Lukas, Edward Jiang, Xinda Li, Florian Kerschbaum
https://arxiv.org/abs/2108.04974

Abstract: Deep Neural Network (DNN) watermarking is a method for provenance verification of DNN models. Watermarking should be robust against watermark removal attacks that derive a surrogate model that evades provenance verification. Many watermarking schemes that claim robustness have been proposed, but their robustness is only validated in isolation against a relatively small set of attacks. There is no systematic, empirical evaluation of these claims against a common, comprehensive set of removal attacks. This uncertainty about a watermarking scheme's robustness causes difficulty to trust their deployment in practice. In this paper, we evaluate whether recently proposed watermarking schemes that claim robustness are robust against a large set of removal attacks. We survey methods from the literature that (i) are known removal attacks, (ii) derive surrogate models but have not been evaluated as removal attacks, and (iii) novel removal attacks. Weight shifting, transfer learning and smooth retraining are novel removal attacks adapted to the DNN watermarking schemes surveyed in this paper. We propose taxonomies for watermarking schemes and removal attacks. Our empirical evaluation includes an ablation study over sets of parameters for each attack and watermarking scheme on the image classification datasets CIFAR-10 and ImageNet. Surprisingly, our study shows that none of the surveyed watermarking schemes is robust in practice. We find that schemes fail to withstand adaptive attacks and known methods for deriving surrogate models that have not been evaluated as removal attacks. This points to intrinsic flaws in how robustness is currently evaluated. Our evaluation includes a discussion of the runtime of each attack to underpin their practical relevance. While none of the schemes is robust against all attacks, none of the attacks removes all watermarks. We show that attacks can be combined and find combined attacks that remove all watermarks. We show that watermarking schemes need to be evaluated against a more extensive set of removal attacks with a more realistic adversary model. Our source code and a complete dataset of evaluation results will be made publicly available, which allows to independently verify our conclusions.

Features

All watermarking schemes and removal attacks are configured for the image classification datasets CIFAR-10 (32x32 pixels, 10 classes) and ImageNet (224x224 pixels, 1k classes). We implemented the following watermarking schemes, sorted by their categories:

.. and the following removal attacks, sorted by their categories:

Get Started

At this point, the Watermark-Robustness-Toolbox project is not available as a standalone pip package, but we are working on allowing an installation via pip. We describe a manual installation and usage. First, install all dependencies via pip.

$ pip install -r requirements.txt

The following four main scripts provide the entire toolbox's functionality:

  • train.py: Pre-trains an unmarked neural network.
  • embed.py: Embeds a watermark into a pre-trained neural network.
  • steal.py: Performs a removal attack against a watermarked neural network.
  • decision_threshold.py: Computes the decision threshold for a watermarking scheme.

We use the mlconfig library to pass configuration hyperparameters to each script. Configuration files used in our paper for CIFAR-10 and ImageNet can be found in the configs/ directory. Configuration files store all hyperparameters needed to reproduce an experiment.

Step 1: Pre-train a Model on CIFAR-10

$ python train.py --config configs/cifar10/train_configs/resnet.yaml

This creates an outputs directory and saves a model file at outputs/cifar10/null_models/resnet/.

Step 2: Embed an Adi Watermark

$ python embed.py --wm_config configs/cifar10/wm_configs/adi.yaml \
                  --filename outputs/cifar10/null_models/resnet/best.pth

This embeds an Adi watermark into the pre-trained model from 'Example 1' and saves (i) the watermarked model and (ii) all data to read the watermark under outputs/cifar10/wm/adi/00000_adi/.

Step 3: Attempt to Remove a Watermark

$ python steal.py --attack_config configs/cifar10/attack_configs/ftal.yaml \
                  --wm_dir outputs/cifar10/wm/adi/00000_adi/

This runs the Fine-Tuning (FTAL) removal attack against the watermarked model and creates a surrogate model stored under outputs/cifar10/attacks/ftal/. The directory also contains human-readable debug files, such as the surrogate model's watermark and test accuracies.

Datasets

Our toolbox currently implements custom data loaders (class WRTDataLoader) for the following datasets.

  • CIFAR-10
  • ImageNet (needs manual download)
  • Omniglot (needs manual download)
  • Open Images (needs manual download)

Documentation

We are actively working on documenting the parameters of each watermarking scheme and removal attack. At this point, we can only refer to the method's source code (at wrt/defenses/ and wrt/attacks/). Soon we will host a complete documentation of all parameters, so stay tuned!

Contribute

We encourage authors of watermarking schemes or removal attacks to implement their methods in the Watermark-Robustness-Toolbox to make them publicly accessible in a unified framework. Our aim is to improve reproducibility which makes it easier to evaluate a scheme's robustness. Any contributions or suggestions for improvements are welcome and greatly appreciated. This toolbox is maintained as part of a university project by graduate students.

Reference

The codebase has been based off an early version of the Adversarial-Robustness-Tooblox.

Cite our paper

@InProceedings{lukas2022watermarkingsok,
  title={SoK: How Robust is Deep Neural Network Image Classification Watermarking?}, 
  author={Lukas, Nils and Jiang, Edward and Li, Xinda and Kerschbaum, Florian},
  year={2022},
  booktitle={IEEE Symposium on Security and Privacy}
}
Code and data accompanying our SVRHM'21 paper.

Code and data accompanying our SVRHM'21 paper. Requires tensorflow 1.13, python 3.7, scikit-learn, and pytorch 1.6.0 to be installed. Python scripts i

5 Nov 17, 2021
SplineConv implementation for Paddle.

SplineConv implementation for Paddle This module implements the SplineConv operators from Matthias Fey, Jan Eric Lenssen, Frank Weichert, Heinrich Mül

北海若 3 Dec 29, 2021
codes for "Scheduled Sampling Based on Decoding Steps for Neural Machine Translation" (long paper of EMNLP-2022)

Scheduled Sampling Based on Decoding Steps for Neural Machine Translation (EMNLP-2021 main conference) Contents Overview Background Quick to Use Furth

Adaxry 13 Jul 25, 2022
Code for Learning Manifold Patch-Based Representations of Man-Made Shapes, in ICLR 2021.

LearningPatches | Webpage | Paper | Video Learning Manifold Patch-Based Representations of Man-Made Shapes Dmitriy Smirnov, Mikhail Bessmeltsev, Justi

Dima Smirnov 22 Nov 14, 2022
Official Repsoitory for "Activate or Not: Learning Customized Activation." [CVPR 2021]

CVPR 2021 | Activate or Not: Learning Customized Activation. This repository contains the official Pytorch implementation of the paper Activate or Not

184 Dec 27, 2022
Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA for motion deblurring, image deraining, denoising (Gaussian/real data), and defocus deblurring.

Restormer: Efficient Transformer for High-Resolution Image Restoration Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan,

Syed Waqas Zamir 906 Dec 30, 2022
Code for "Learning Graph Cellular Automata"

Learning Graph Cellular Automata This code implements the experiments from the NeurIPS 2021 paper: "Learning Graph Cellular Automata" Daniele Grattaro

Daniele Grattarola 37 Oct 26, 2022
Semi-Supervised Learning, Object Detection, ICCV2021

End-to-End Semi-Supervised Object Detection with Soft Teacher By Mengde Xu*, Zheng Zhang*, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai,

Microsoft 789 Dec 27, 2022
This is an official pytorch implementation of Fast Fourier Convolution.

Fast Fourier Convolution (FFC) for Image Classification This is the official code of Fast Fourier Convolution for image classification on ImageNet. Ma

pkumi 199 Jan 03, 2023
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (NeurIPS 2021)

VRDP (NeurIPS 2021) Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language Mingyu Ding, Zhenfang Chen, Tao Du, Pin

Mingyu Ding 36 Sep 20, 2022
Deep Two-View Structure-from-Motion Revisited

Deep Two-View Structure-from-Motion Revisited This repository provides the code for our CVPR 2021 paper Deep Two-View Structure-from-Motion Revisited.

Jianyuan Wang 145 Jan 06, 2023
Exploration-Exploitation Dilemma Solving Methods

Exploration-Exploitation Dilemma Solving Methods Medium article for this repo - HERE In ths repo I implemented two techniques for tackling mentioned t

Aman Mishra 6 Jan 25, 2022
Locally Most Powerful Bayesian Test for Out-of-Distribution Detection using Deep Generative Models

LMPBT Supplementary code for the Paper entitled ``Locally Most Powerful Bayesian Test for Out-of-Distribution Detection using Deep Generative Models"

1 Sep 29, 2022
Lane assist for ETS2, built with the ultra-fast-lane-detection model.

Euro-Truck-Simulator-2-Lane-Assist Lane assist for ETS2, built with the ultra-fast-lane-detection model. This project was made possible by the amazing

36 Jan 05, 2023
Help you understand Manual and w/ Clutch point while driving.

简体中文 forza_auto_gear forza_auto_gear is a tool for Forza Horizon 5. It will help us understand the best gear shift point using Manual or w/ Clutch in

15 Oct 08, 2022
vit for few-shot classification

Few-Shot ViT Requirements PyTorch (= 1.9) TorchVision timm (latest) einops tqdm numpy scikit-learn scipy argparse tensorboardx Pretrained Checkpoints

Martin Dong 26 Nov 30, 2022
Neural network for stock price prediction

neural_network_for_stock_price_prediction Neural networks for stock price predic

2 Feb 04, 2022
Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics.

Bonnet: An Open-Source Training and Deployment Framework for Semantic Segmentation in Robotics. By Andres Milioto @ University of Bonn. (for the new P

Photogrammetry & Robotics Bonn 314 Dec 30, 2022
Adversarial Learning for Modeling Human Motion

Adversarial Learning for Modeling Human Motion This repository contains the open source code which reproduces the results for the paper: Adversarial l

wangqi 6 Jun 15, 2021
catch-22: CAnonical Time-series CHaracteristics

catch22 - CAnonical Time-series CHaracteristics About catch22 is a collection of 22 time-series features coded in C that can be run from Python, R, Ma

Carl H Lubba 229 Oct 21, 2022