F-Principle

This is an exercise problem of the digital signal processing (DSP) course at School of Artificial Intelligence at the Nanjing University (NJU), teaching by Han-Jia Ye. The course homepage is at DSP. This exercise is written by Jia-Qi Yang. Please feel free to contact me by mailing [email protected] if you have any questions.

Problem 1: Understanding F-Principle (35pt)

Read following articles:

1. F-Principle
2. Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Then, answer following questions:

1. What is F-Principle ? (5pt)
2. Why F-Principle is important ? (5pt)
3. What are the differences between response frequency and input frequency ? Which one is used in F-Principle ? (5pt)
4. How is frequency defined in high-dimensional functions ? Why ? (10pt)
5. How does the authors verify F-Principle experimentally ? (10pt)

Problem 2: Reproducing F-Principle (65pt)

Code to reproduce F-Principle by the authors is published at F-Principle Github.

You may modify F-Principle Github to conduct following experiments. However, this implementation is based on tf1.x, and the high-dim experiments are not implemented. You may also choose to extend pytorch training scripts provided in src/.

2.1 Low-dim Experiment (25pt)

Read F-Principle in low-dim experiments.

1. Plot training procedure in Spatial Domain, i.e. the first figure in F-Principle in low-dim experiments. (10pt)
2. Plot training procedure in Fourier Domain, i.e. the second figure in F-Principle in low-dim experiments. (10pt)

You may plot several figures instead of gifs in F-Principle in low-dim experiments.

2.2 High-dim Experiment (30pt)

Read F-Principle in high-dim experiments.

1. Implement the projection method or the filtering method on MNIST dataset.
2. Describe the procedure of your method using pseudo-code.
3. Inspect how each response frequency component (e.g. high-frequency and low-frequency) converges. You may plot figures or using tables to demostrate your results.

2.3 Summay (10pt)

1. What did you learn from this practice problem ? (5pt)
2. What problems did you encounter and how did you solve them ? (5pt)