Optimization techniques II

We discuss adaptive methods for SGD such as RMSprop and ADAM. We also talk about normalization layers and their effects on the neural network training process. Finally, we discuss a real-world example of neural nets being used in industry to make MRI scans faster and more efficient.

Please let me know if any changes need to be made before merging.

This:

• Moves PyTorch from 0.4 to 1.1 (one tiny code change)
• Moves Python from 3.6 to 3.7 (no changes to code, just env)
• Moves 1-2 requirements out of notebooks and into environment (potential nasty scipy pip install from librosa avoided!)
• Uses conda kernels so the correct environment kernel is available (all notebooks rerun to pick up proper kernel)
• Adds JuptyerLab (not required, but nice) - the interactive backend in the final notebook is still best in the classic interface. Try out built-in dark mode!

All notebooks seem to run (except noted minor issue with JupyterLab)

Hi Alf :wave:,

As indicated in my last email, I can't afford to wait for Yann's return without a big delay on my side. So here are the subtitle files:

• For English, it is the addition of the unicode. In practice:

1. The list of files not modified during this review of the unicode: practinum1 (didn't need unicode), practinum4 (the file contains blocks of 3 instead of 2 for the others), for lecture 12 (the only file I didn't translate into French)

2. The list of finished files (full English review + unicode) : lecture 6 & 9

3. The list of about clean files (partial English review + unicode) : lecture 1-3,10,11 + practinum 1-3, 7-8, 10

4. The list of not clean files (no English review + unicode): lecture 5-9,12-15 + practinum 5-6,9,11-15

• For French, these are all the subtitles (except for lecture 12 where I have huge problems understanding Mike Lewis's accent and so I preferred not to put anything than to translate badly).

I also added a disclamer for the V2 of the French translation of the website which should arrive this month. It should be my next and last PR closing the French translation work :boom:

Loïck

The following image links are broken:

• [x] Figure 2(a)
• [x] Figure 2(b)
• [x] Figure 3(a)
• [x] Figure 3(b)

See https://atcold.github.io/pytorch-Deep-Learning/en/week03/03-3/

I think the images were originally obtained from this presentation: 02 - CNN.pdf

See pages 10-11

Also, small suggestions:

• [x] Change Figure 4 to include R^7 and R^2 as in Slide 20 . This would better match the text for Figure 4.

• [x] Include Figure (4b maybe?) with that on Slide 21 to show what Padding is doing

Hi @Atcold ! I would like to know how and where should I commit markdown files in Portuguese? I recall that you have commented something with @ebetica .

Hi @JonathanSum ! Just for your info, There seems to be some inline latex broken on lecture 13-3:

The rest of the lectures I've checked seem to be fine.

This PR reorganizes the website structure, so we now have:

en/
  index.md
  about.dm
  week01/
  week02/
  ...
zh/
  index.md
  about.md
  week01/
  week02/
  ...
...

Hopefully it's less messy and easier to work with.

After this is merged, I will pull the images out into a global directory as well.

Also fixes some broken links in zh/index.md

I found an error visualizing on the github.io page the file /docs/es/week02/02-1.md.

The english version of the file appears before some parts and the layout of the spanish parts after the english parts gets a bit messed up.

A PR that fixes the timers of the sbv files that I couldn't correct in PR #660 to avoid conflicts.

I also took the opportunity to correct the few errors I caught when translating the lecture10.

I also noticed that the sbv files of the practinums of weeks 14 and 15 were missing.

I have translated the top 50% of the RNN(06-02) in Chinese.

I passed the course on deep learning.ai and I also wrote a few notebooks to help students in the coursera Tensorflow time series seq2seq notebook.

Poornima and I have compared an LSTM and RNN and visualized the gradients with respect to the input. We see that the gradients for the RNN are much smaller compared to the LSTM.

We are able to train MNIST for a large input sequence with an LSTM and failed to do so with an RNN.

Hope this is useful. If we need to make any chances, please let us know !

Hi there,

I hope these tips can help you: Using Docker, with torchtext version 0.9.0 and PyTorch version 1.8.0.

Please note that PyTorch 1.8 may not have good support for CUDA versions newer than 11. If you are using a newer version of CUDA, you may want to consider using the CPU instead.

Bunch of minor "theoretical" changes in the evaluation function:

1. test_data_gen was used as the data generator in the evaluation, instead of data_generator, thereby evaluating the net on the test set used for training (not an actual issue here given the sequences are randomized and not sampled from existing datasets, but in principle would lead to a data leak in realistic scenarios);
2. the correct sequences printed were a sampling (with reinsertion) of the first 10 evaluated, instead of 10 sampled from the whole set of correct ones;
3. the condition for printing the incorrectly classified sequences would declare the absence of misclassifications if verbose==False, independently of their actual presence;

Added

• new truck methods for randomizing state within contraints
• new truck methods for seeing if truck is at dock or offscreen
• Training script for optimizing controller

Note

I currently have not successfully trained the controller to convergence. I have based the training off of this. On the website, they mention that the controller is hard to train. I have tried training it on the website with no success, so it seems like even their lessons are difficult to train. However, the code for training should be very similar to the code on the website. You may also alter the amount of lessons, max time steps, learning rate etc. to see if the model converges. I have been trying for over a week and have not succeeded yet.

I found that the Chinese version was basically machine translated, which caused the latex syntax to be broken. Of course, there are a lot of unreasonable translation. This PR is mainly about fixing broken latex. I also did my best to fix some of the translations that were too much bullshit.

I would question some translations in the dictionary for Russian: I've graduated this year and we haven't really translated everything. For example, it will be more understandable if I say "one-hot" in Russian as it is, rather than "унитарный код". Basically, I've never heard anyone calling it "унитарный код", to be honest...

So I guess there is a choice between being academically strict or being understood.

I think source activate is a few years old now and isn't supported anymore. https://stackoverflow.com/questions/49600611/python-anaconda-should-i-use-conda-activate-or-source-activate-in-linux