A collection of various deep learning architectures, models, and tips

Overview

Python 3.7

Deep Learning Models

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Traditional Machine Learning

Multilayer Perceptrons

Convolutional Neural Networks

Basic

Concepts

  • Replacing Fully-Connnected by Equivalent Convolutional Layers
       [PyTorch: GitHub | Nbviewer]

AlexNet

DenseNet

  • DenseNet-121 Digit Classifier Trained on MNIST
       [PyTorch: GitHub | Nbviewer]
  • DenseNet-121 Image Classifier Trained on CIFAR-10
       [PyTorch: GitHub | Nbviewer]

Fully Convolutional

  • Fully Convolutional Neural Network
       [PyTorch: GitHub | Nbviewer]

LeNet

MobileNet

Network in Network

  • Network in Network CIFAR-10 Classifier
       [PyTorch: GitHub | Nbviewer]

VGG

ResNet

  • ResNet and Residual Blocks
       [PyTorch: GitHub | Nbviewer]
  • ResNet-18 Digit Classifier Trained on MNIST
       [PyTorch: GitHub | Nbviewer]
  • ResNet-18 Gender Classifier Trained on CelebA
       [PyTorch: GitHub | Nbviewer]
  • ResNet-34 Digit Classifier Trained on MNIST
       [PyTorch: GitHub | Nbviewer]
  • ResNet-34 Object Classifier Trained on QuickDraw
       [PyTorch: GitHub | Nbviewer]
  • ResNet-34 Gender Classifier Trained on CelebA
       [PyTorch: GitHub | Nbviewer]
  • ResNet-50 Digit Classifier Trained on MNIST
       [PyTorch: GitHub | Nbviewer]
  • ResNet-50 Gender Classifier Trained on CelebA
       [PyTorch: GitHub | Nbviewer]
  • ResNet-101 Gender Classifier Trained on CelebA
       [PyTorch: GitHub | Nbviewer]
  • ResNet-101 Trained on CIFAR-10
       [PyTorch: GitHub | Nbviewer]
  • ResNet-152 Gender Classifier Trained on CelebA
       [PyTorch: GitHub | Nbviewer]

Normalization Layers

  • BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier
       [PyTorch: GitHub | Nbviewer]
  • Filter Response Normalization for Network-in-Network CIFAR-10 Classifier
       [PyTorch: GitHub | Nbviewer]

Metric Learning

  • Siamese Network with Multilayer Perceptrons
       [TensorFlow 1: GitHub | Nbviewer]

Autoencoders

Fully-connected Autoencoders

Convolutional Autoencoders

  • Convolutional Autoencoder with Deconvolutions / Transposed Convolutions
       [TensorFlow 1: GitHub | Nbviewer]
       [PyTorch: GitHub | Nbviewer]
  • Convolutional Autoencoder with Deconvolutions and Continuous Jaccard Distance
       [PyTorch: GitHub | Nbviewer]
  • Convolutional Autoencoder with Deconvolutions (without pooling operations)
       [PyTorch: GitHub | Nbviewer]
  • Convolutional Autoencoder with Nearest-neighbor Interpolation
       [TensorFlow 1: GitHub | Nbviewer]
       [PyTorch: GitHub | Nbviewer]
  • Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA
       [PyTorch: GitHub | Nbviewer]
  • Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw
       [PyTorch: GitHub | Nbviewer]

Variational Autoencoders

Conditional Variational Autoencoders

  • Conditional Variational Autoencoder (with labels in reconstruction loss)
       [PyTorch: GitHub | Nbviewer]
  • Conditional Variational Autoencoder (without labels in reconstruction loss)
       [PyTorch: GitHub | Nbviewer]
  • Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss)
       [PyTorch: GitHub | Nbviewer]
  • Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss)
       [PyTorch: GitHub | Nbviewer]

Generative Adversarial Networks (GANs)

Graph Neural Networks (GNNs)

  • Most Basic Graph Neural Network with Gaussian Filter on MNIST
       [PyTorch: GitHub | Nbviewer]
  • Basic Graph Neural Network with Edge Prediction on MNIST
       [PyTorch: GitHub | Nbviewer]
  • Basic Graph Neural Network with Spectral Graph Convolution on MNIST
       [PyTorch: GitHub | Nbviewer]

Recurrent Neural Networks (RNNs)

Many-to-one: Sentiment Analysis / Classification

  • A simple single-layer RNN (IMDB)
       [PyTorch: GitHub | Nbviewer]
  • A simple single-layer RNN with packed sequences to ignore padding characters (IMDB)
       [PyTorch: GitHub | Nbviewer]
  • RNN with LSTM cells (IMDB)
       [PyTorch: GitHub | Nbviewer]
  • RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors
       [PyTorch: GitHub | Nbviewer]
  • RNN with LSTM cells and Own Dataset in CSV Format (IMDB)
       [PyTorch: GitHub | Nbviewer]
  • RNN with GRU cells (IMDB)
       [PyTorch: GitHub | Nbviewer]
  • Multilayer bi-directional RNN (IMDB)
       [PyTorch: GitHub | Nbviewer]
  • Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News)
       [PyTorch: GitHub | Nbviewer]

Many-to-Many / Sequence-to-Sequence

  • A simple character RNN to generate new text (Charles Dickens)
       [PyTorch: GitHub | Nbviewer]

Ordinal Regression

  • Ordinal Regression CNN -- CORAL w. ResNet34 on AFAD-Lite
       [PyTorch: GitHub | Nbviewer]
  • Ordinal Regression CNN -- Niu et al. 2016 w. ResNet34 on AFAD-Lite
       [PyTorch: GitHub | Nbviewer]
  • Ordinal Regression CNN -- Beckham and Pal 2016 w. ResNet34 on AFAD-Lite
       [PyTorch: GitHub | Nbviewer]

Tips and Tricks

  • Cyclical Learning Rate
       [PyTorch: GitHub | Nbviewer]
  • Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet)
       [PyTorch: GitHub | Nbviewer]
  • Gradient Clipping (w. MLP on MNIST)
       [PyTorch: GitHub | Nbviewer]

Transfer Learning

  • Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10)
       [PyTorch: GitHub | Nbviewer]

Visualization and Interpretation

  • Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images)
       [PyTorch: GitHub | Nbviewer]
  • Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images)
       [PyTorch: GitHub | Nbviewer]

PyTorch Workflows and Mechanics

Custom Datasets

  • Custom Data Loader Example for PNG Files
       [PyTorch: GitHub | Nbviewer]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5
       [PyTorch: GitHub | Nbviewer]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA
       [PyTorch: GitHub | Nbviewer]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw
       [PyTorch: GitHub | Nbviewer]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset
       [PyTorch: GitHub | Nbviewer]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD)
       [PyTorch: GitHub | Nbviewer]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color Images
       [PyTorch: GitHub | Nbviewer]
  • Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNIST
       [PyTorch: GitHub | Nbviewer]

Training and Preprocessing

Improving Memory Efficiency

  • Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10)
       [PyTorch: GitHub | Nbviewer]

Parallel Computing

  • Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA
       [PyTorch: GitHub | Nbviewer]
  • Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example)    [PyTorch: GitHub | Nbviewer]

Other

  • PyTorch with and without Deterministic Behavior -- Runtime Benchmark
       [PyTorch: GitHub | Nbviewer]
  • Sequential API and hooks
       [PyTorch: GitHub | Nbviewer]
  • Weight Sharing Within a Layer
       [PyTorch: GitHub | Nbviewer]
  • Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib
       [PyTorch: GitHub | Nbviewer]

Autograd

  • Getting Gradients of an Intermediate Variable in PyTorch
       [PyTorch: GitHub | Nbviewer]

TensorFlow Workflows and Mechanics

Custom Datasets

  • Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives
       [TensorFlow 1: GitHub | Nbviewer]
  • Storing an Image Dataset for Minibatch Training using HDF5
       [TensorFlow 1: GitHub | Nbviewer]
  • Using Input Pipelines to Read Data from TFRecords Files
       [TensorFlow 1: GitHub | Nbviewer]
  • Using Queue Runners to Feed Images Directly from Disk
       [TensorFlow 1: GitHub | Nbviewer]
  • Using TensorFlow's Dataset API
       [TensorFlow 1: GitHub | Nbviewer]

Training and Preprocessing

  • Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives
       [TensorFlow 1: GitHub | Nbviewer]
Owner
Sebastian Raschka
Machine Learning researcher & passionate open source contributor. Author of the "Python Machine Learning" book.
Sebastian Raschka
Image captioning - Tensorflow implementation of Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Introduction This neural system for image captioning is roughly based on the paper "Show, Attend and Tell: Neural Image Caption Generation with Visual

Guoming Wang 749 Dec 28, 2022
PyTorch Implementation of Fully Convolutional Networks. (Training code to reproduce the original result is available.)

pytorch-fcn PyTorch implementation of Fully Convolutional Networks. Requirements pytorch = 0.2.0 torchvision = 0.1.8 fcn = 6.1.5 Pillow scipy tqdm

Kentaro Wada 1.6k Jan 04, 2023
C++ Implementation of PyTorch Tutorials for Everyone

C++ Implementation of PyTorch Tutorials for Everyone OS (Compiler)\LibTorch 1.9.0 macOS (clang 10.0, 11.0, 12.0) Linux (gcc 8, 9, 10, 11) Windows (msv

Omkar Prabhu 1.5k Jan 04, 2023
simple generative adversarial network (GAN) using PyTorch

Generative Adversarial Networks (GANs) in PyTorch Running Run the sample code by typing: ./gan_pytorch.py ...and you'll train two nets to battle it o

vanguard_space 32 Jun 14, 2020
PyTorch Tutorial for Deep Learning Researchers

This repository provides tutorial code for deep learning researchers to learn PyTorch. In the tutorial, most of the models were implemented with less

Yunjey Choi 25.4k Jan 05, 2023
Open source guides/codes for mastering deep learning to deploying deep learning in production in PyTorch, Python, C++ and more.

Deep Learning Materials by Deep Learning Wizard Start Learning Now Please head to www.deeplearningwizard.com to start learning! It is mobile/tablet fr

Ritchie Ng 572 Dec 28, 2022
Simple examples to introduce PyTorch

This repository introduces the fundamental concepts of PyTorch through self-contained examples. At its core, PyTorch provides two main features: An n-

Justin Johnson 4.4k Jan 07, 2023
PyTorch tutorials and best practices.

Effective PyTorch Table of Contents Part I: PyTorch Fundamentals PyTorch basics Encapsulate your model with Modules Broadcasting the good and the ugly

Vahid Kazemi 1.5k Jan 04, 2023
Pytorch implementations of various Deep NLP models in cs-224n(Stanford Univ)

DeepNLP-models-Pytorch Pytorch implementations of various Deep NLP models in cs-224n(Stanford Univ: NLP with Deep Learning) This is not for Pytorch be

Kim SungDong 2.9k Dec 24, 2022
A collection of various deep learning architectures, models, and tips

Deep Learning Models A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks. Traditiona

Sebastian Raschka 15.5k Jan 07, 2023
Minimal tutorials for PyTorch

Minimal tutorials for PyTorch adapted from Alec Radford's Theano tutorials. Tensor multiplication Linear Regression Logistic Regression Neural Network

Vinh Khuc 321 Oct 25, 2022
An IPython Notebook tutorial on deep learning for natural language processing, including structure prediction.

Table of Contents: Introduction to Torch's Tensor Library Computation Graphs and Automatic Differentiation Deep Learning Building Blocks: Affine maps,

Robert 1.8k Jan 04, 2023
PyTorch tutorials.

PyTorch Tutorials All the tutorials are now presented as sphinx style documentation at: https://pytorch.org/tutorials Contributing We use sphinx-galle

6.6k Jan 02, 2023
This is a gentle introductin on how to start using an awesome library called Weights and Biases.

🪄 W&B Minimal PyTorch Tutorial This tutorial is also accompanied with a PyTorch source code, it can be found in src folder. Furthermore, all plots an

Nauryzbay K 8 Aug 20, 2022
Interactive deep learning book with multi-framework code, math, and discussions. Adopted at 200 universities.

D2L.ai: Interactive Deep Learning Book with Multi-Framework Code, Math, and Discussions Book website | STAT 157 Course at UC Berkeley | Latest version

Dive into Deep Learning (D2L.ai) 16k Jan 03, 2023
Deep Learning (with PyTorch)

Deep Learning (with PyTorch) This notebook repository now has a companion website, where all the course material can be found in video and textual for

Alfredo Canziani 6.2k Jan 02, 2023
The Hitchiker's Guide to PyTorch

The Hitchiker's Guide to PyTorch

Kai Arulkumaran 1k Dec 20, 2022
Simple PyTorch Tutorials Zero to ALL!

PyTorchZeroToAll Quick 3~4 day lecture materials for HKUST students. Video Lectures: (RNN TBA) Youtube Bilibili Slides Lecture Slides @GoogleDrive If

Sung Kim 3.7k Dec 30, 2022
Example of network fine-tuning in pytorch for the kaggle competition Dogs vs. Cats Redux: Kernels Edition

Example of network fine-tuning in pytorch for the kaggle competition Dogs vs. Cats Redux: Kernels Edition Currently

bobby 70 Sep 22, 2022
A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

PyTorch Examples WARNING: if you fork this repo, github actions will run daily on it. To disable this, go to /examples/settings/actions and Disable Ac

19.4k Jan 01, 2023