Code to reproduce the experimental results for Statistically Robust Neural Network Classification, UAI 2021.
To reproduce the results of Experiment 6.1, run the following from the base directory:
python run_exp_1.py
This will:
- Train the NN classifier on MNIST using natural and corrupted training methods, as described in the paper;
- Evaluate the TSRM metric on each trained NN at a number of epsilon values;
- Collate the results and produce the plot of Figure 1.
Likewise, to reproduce the results of Experiment 6.2, run the following:
python run_exp_2.py
This will:
- Train the wide ResNet CNN classifier on CIFAR-10 using natural, corruption and adversarial training methods;
- Evaluate the trained networks on natural risk, SRR, and adversarial risk, outputting the results to a csv file (corresponding to results in Table 1).
Likewise, to reproduce the results of Experiment 6.3, run the following:
python run_exp_3.py
This will:
- Train the NN classifier on MNIST using natural and corrupted training methods (2 networks);
- Keep track of the natural and SRR weighted cross entropy loss during each epoch of training for both networks;
- Produce the plot of Figure 2.
Likewise, to reproduce the results of the experiment in Appendix A, run the following (AFTER running Experiment 6.1):
python run_exp_estimation.py
This will:
- Load the naturally trained NN classifier on MNIST from Experiment 6.1;
- Evaluate the TSRM using both adaptive sampling and monte carlo for this network and 100 datapoints from the MNIST test set;
- Produce the plot of Figure 3.