D2L has gone 1.0.0-beta0! We thank all the 296 contributors for making this happen!

Forthcoming on Cambridge University Press

Chapter 1--11 will be forthcoming on Cambridge University Press (early 2023).

New JAX Implementation

We added new JAX implementation. Get started with import jax at https://d2l.ai/chapter_preliminaries/ndarray.html

Thank @AnirudhDagar!

New Vol.2 Chapter on Reinforcement Learning

With the advent of ChatGPT (sibling model to InstructGPT fine-tuned using reinforcement learning), you may get curious about how to enable ML to take decisions sequentially:

17. Reinforcement Learning 17.1. Markov Decision Process (MDP) 17.2. Value Iteration 17.3. Q-Learning

Thank Pratik Chaudhari (University of Pennsylvania and Amazon), Rasool Fakoor @rasoolfa (Amazon), and Kavosh Asadi (Amazon)!

New Vol.2 Chapter on Gaussian Processes

“Everything is a special case of a Gaussian process.” Gaussian processes and deep neural networks are highly complementary and can be combined to great effect:

18. Gaussian Processes 18.1. Introduction to Gaussian Processes 18.2. Gaussian Process Priors 18.3. Gaussian Process Inference

Thank Andrew Gordon Wilson @andrewgordonwilson (New York University and Amazon)!

New Vol.2 Chapter on Hyperparameter Optimization

Tired of setting hyperparameters in a trial-and-error manner? You may wish to check out the systematic hyperparameter optimization approach:

19. Hyperparameter Optimization 19.1. What Is Hyperparameter Optimization? 19.2. Hyperparameter Optimization API 19.3. Asynchronous Random Search 19.4. Multi-Fidelity Hyperparameter Optimization 19.5. Asynchronous Successive Halving

Thank Aaron Klein @aaronkl (Amazon), Matthias Seeger @mseeger (Amazon), and Cedric Archambeau (Amazon)!

Fixes and Improvements

Thank @finale80 @JojiJoseph @gab-chen @Excelsior7 @shanmo @kxxt @vBarbaros @gui-miotto @bolded @atishaygarg @tuelwer @gopalakrishna-r @qingfengtommy @Mohamad-Jaallouk @biswajitsahoo1111 @315930399 for improving this book!

Add PaddlePaddle implementation for the d2l library (compatible with classic.d2l.ai). See the PR in d2l-zh.

We are happy to release D2L 1.0.0-alpha1.post0 We thank all the contributors who have made this open-source textbook better for everyone.

This minor release includes the following updates:

• Build PDFs using Cambridge Latex Style (#2187)
• Feat: Add dynamic preview/stable version tab (#2264)
• Chapter Modern RNN is heavily improved, refactoring the text and reorganizing content (#2241)
• Chapter Modern CNN is refactored and is more polished (#2249)
• Bump supported version to Python3.9 (#2231)

It also comes with the following bug-fixes:

• Fix #2250: Depend on matplotlib-inline instead of ipython to avoid colab warning (#2279)
• Fix broken preview version pdf links (#2264)
• Fix #2247: Tensorflow explicitly squeeze image to support matplotlib<3.3.0 (#2248)
• Fix PyTorch moving average computation for batch norm (#2213)
• Fix PyTorch module types (LazyLinear -> Linear) (#2225)
• Fix QR code being overridden by the section rule issue (#2251)
• Fix torch.meshgrid user warning (7d921558a8cb063af05c6246b0aa8cbb2fe9d222)

We are excited to announce the release of D2L 1.0.0-alpha0! We thank all the 265 contributors who have made this open-source textbook better for everyone.

New Topics and Revision

We have added the following new topics, with discussions of more recent methods such as ResNeXt, RegNet, ConvNeXt, Vision Transformer, Swin Transformer, T5, GPT-1/2/3, zero-shot, one-shot, few-shot, Gato, Imagen, Minerva, and Parti.

• Object-Oriented Design for Implementation
• Synthetic Regression Data
• Generalization
• The Base Classification Model
• Generalization in Classification
• Generalization in Deep Learning
• ResNeXt
• Designing Convolution Network Architectures
• Transformers for Vision
• Large-Scale Pretraining with Transformers

Besides new topics, we have significantly revised all the topics up to transformers. For example, the previous Linear Neural Networks and Multilayer Perceptrons chapters have been revamped as new chapters of Linear Neural Networks for Regression, Linear Neural Networks for Classification, and Multilayer Perceptrons.

New API

Throughout the book we repeatedly walk through various components including the data, the model, the loss function, and the optimization algorithm. Treating components in deep learning as objects, we can define classes for these objects and their interactions. This object-oriented design for implementation will greatly streamline the presentation. Therefore, inspired by open-source libraries such as PyTorch Lightning, we have re-designed the API with three core classes:

• Module contains models, losses, and optimization methods;
• DataModule provides data loaders for training and validation;
• Both classes are combined using the Trainer class, which allows us to train models on a variety of hardware platforms.

For example, with the classic API in previous releases:

model = # Multilayer perceptron definition
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=256)
loss = nn.CrossEntropyLoss(reduction='none')
trainer = torch.optim.SGD(net.parameters(), lr=0.1)
d2l.train_ch3(model, train_iter, test_iter, loss, num_epochs=10, trainer)

With the new API:

model = # Multilayer perceptron definition
data = d2l.FashionMNIST(batch_size=256)
trainer = d2l.Trainer(max_epochs=10)
trainer.fit(model, data)

Lazy Layers in PyTorch

Since v1.8.0, PyTorch offers "lazy" layers where input shape specification is no longer required. For simplicity we will use "lazy" layers whenever we can, such as:

class LinearRegression(d2l.Module):
    def __init__(self, lr):
        super().__init__()
        self.save_hyperparameters()
        self.net = nn.LazyLinear(1)  # Lazy layer with output dimension only
        self.net.weight.data.normal_(0, 0.01)
        self.net.bias.data.fill_(0)

Ongoing Translations

Join us to improve ongoing translations in:

• Chinese (Git repo)
• Portuguese (Git repo)
• Turkish (Git repo)
• Vietnamese (Git repo)
• Korean (Git repo)
• Japanese (Git repo)

This release fixes issues when installing d2l package and running d2l notebooks on Google Colab with Python 3.7 and updates PyTorch & TensorFlow to their respective latest versions.

More concretely, this release includes the following upgrades/fixes:

• Update TensorFlow==2.8.0 (#2055)
• Update PyTorch: torch==1.11.0 & torchvision==0.12.0 (#2063)
• Rollback NumPy==1.21.5 & Support Python>=3.7 (#2066)
• Fix MXNet plots; NumPy auto coercion & Unpin matplotlib==3.4 dependency (#2078)
• Fix the broken download link for MovieLens dataset (#2074)
• Fix iPython deprecation warning of set_matplotlib_formats (#2065)
• Fix Densenet PyTorch implementation using nn.AdaptiveAvgPool2d (f6b1dd0053a5caeb8a53c81f97eb929c27fb868e)
• Fix the hotdog class in section Fine Tuning for imagenet, which is number 934 instead of 713 (#2009)
• Use reduction=none in PyTorch loss for train_epoch_ch3 (#2007)
• Fix argument test_feature->test_features of train_and_pred in kaggle house price section (#1982)
• Fix TypeError: can’t convert CUDA tensor to numpy, explicitly moving torch tensor to cpu before plotting (#1966)

This release supports running the book with SageMaker Studio Lab for free and introduces several fixes:

• Fix data synchronization for multi-GPU training in PyTorch (https://github.com/d2l-ai/d2l-en/pull/1978)
• Fix token sampling in BERT datasets (https://github.com/d2l-ai/d2l-en/pull/1979/)
• Fix semantic segmentation normalization in PyTorch (https://github.com/d2l-ai/d2l-en/pull/1980/)
• Fix mean square loss calculation in PyTorch and TensorFlow (https://github.com/d2l-ai/d2l-en/pull/1984)
• Fix broken paragraphs (https://github.com/d2l-ai/d2l-en/commit/8e0fe4ba54b6e2a0aa0f15f58a1e81f7fef1cdd7)

Dive into Deep Learning is now available on arxiv!

Framework Adaptation

We have added TensorFlow implementations up to Chapter 11 (Optimization Algorithms).

Towards v1.0

The following chapters have been significantly improved for v1.0:

• Optimization (the first 4 sections)
• Computational Performance
• Computer Vision
• Natural Language Processing: Pretraining
• Natural Language Processing: Applications

Finalized chapters are being translated into Chinese (d2l-zh v2)

Other Improvements

• Add BLEU uniform weights from the original paper
• Revise the normalization trick in LogSumExp
• Revise data standardization
• Prove convexity using second derivatives for one-dimensional and multi-dimensional cases
• Improve d2l.train_2d function
• Improve convergence analysis of Newton's method
• Improve SGD convergence analysis for convex objectives 1
• Improve Convergence Analysis for Convex Objectives
• Reorganize comparisons of network partitioning, layer-wise partitioning, and data parallelism
• Improve d2l.box_iou function
• Improve the "Labeling Classes and Offsets" subsection
• Add discussions of issues of non-maximum suppression
• Reorganize multiscale anchor boxes and multiscale detection
• Highlight layerwise representations via deep nets in multiscale object detection
• Connect SSD downsampling blocks to VGG blocks
• Refer to YOLO and a recent survey on object detection
• Fix legend issues in Kaggle CIFAR-10 and ImageNet Dogs
• Improve performance on the Kaggle small-scale CIFAR-10 dataset
• Improve performance on the Kaggle small-scale ImageNet Dog dataset
• Improve the function to build the mapping from RGB to class indices for VOC labels
• Revise motivations for transposed convolution
• Rewrite basic transposed convolution operation
• Add relations between transposed convolution and regular convolution implementations
• Improve explanations of the pretrained backbone for the fully convolutional network
• Improve the output synthesized image of style transfer
• Add d2l.show_list_len_pair_hist
• Fix d2l.get_negatives
• Improve efficiency of d2l.Vocab
• Exclude unknown tokens when training word embeddings
• Add self-supervised learning
• Add discussions of self-supervised learning in NLP
• Revise the notation table

Brand-New Attention Chapter

We have added the brand-new Chapter: Attention Mechanisms:

• Attention Cues

  • Attention Cues in Biology
  • Queries, Keys, and Values
  • Visualization of Attention

• Attention Pooling: Nadaraya-Watson Kernel Regression

  • Generating the Dataset
  • Average Pooling
  • Nonparametric Attention Pooling
  • Parametric Attention Pooling

• Attention Scoring Functions

  • Masked Softmax Operation
  • Additive Attention
  • Scaled Dot-Product Attention

• Bahdanau Attention

  • Model
  • Defining the Decoder with Attention
  • Training

• Multi-Head Attention

  • Model
  • Implementation

• Self-Attention and Positional Encoding

  • Self-Attention
  • Comparing CNNs, RNNs, and Self-Attention
  • Positional Encoding

• Transformer

  • Model
  • Positionwise Feed-Forward Networks
  • Residual Connection and Layer Normalization
  • Encoder
  • Decoder
  • Training

PyTorch Adaptation Completed

We have completed PyTorch implementations for Vol.1 (Chapter 1--15).

Towards v1.0

The following chapters have been significantly improved for v1.0:

• Introduction
• Modern Recurrent Neural Networks

Chinese Translation

The following chapters have been translated into Chinese (d2l-zh v2 Git repo, Web preview):

• Introduction
• Preliminaries
• Linear Neural Networks
• Multilayer Perceptrons
• Deep Learning Computation
• Convolutional Neural Networks
• Modern Convolutional Neural Networks

Turkish Translation

The community are translating the book into Turkish (d2l-tr Git repo, Web preview). The first draft of Chapter 1--7 is complete.

Framework Adaptation

We have added PyTorch implementations up to Chapter 11 (Optimization Algorithms). Chapter 1--7 and Chapter 11 have also been adapted to TensorFlow.

Towards v1.0

The following chapters have been significantly improved for v1.0:

• Linear Neural Networks
• Multilayer Perceptrons
• Deep Learning Computation
• Convolutional Neural Networks
• Modern Convolutional Neural Networks
• Recurrent Neural Networks

Finalized chapters are being translated into Chinese (d2l-zh v2)

Other Improvements

• Fixed issues of not showing all the equation numbers in the HTML and PDF
• Consistently used f-string
• Revised overfitting experiments
• Fixed implementation errors for weight decay experiments
• Improved layer index style
• Revised "breaking the symmetry"
• Revised descriptions of covariate and label shift
• Fixed mathematical errors in covariate shift correction
• Added true risk, empirical risk, and (weighted) empirical risk minimization
• Improved variable naming style for matrices and tensors
• Improved consistency of mathematical notation for tensors of order two or higher
• Improved mathematical descriptions of convolution
• Revised descriptions of cross-correlation
• Added feature maps and receptive fields
• Revised mathematical descriptions of batch normalization
• Added more details to Markov models
• Fixed implementations of k-step-ahead predictions in sequence modeling
• Fixed mathematical descriptions in language modeling
• Improved the d2l.Vocab API
• Fixed mathematical descriptions and figure illustrations for deep RNNs
• Added BLEU
• Improved machine translation application results
• Improved the animation plot function in the all the training loops

Highlights

We have added both PyTorch and TensorFlow implementations up to Chapter 7 (Modern CNNs).

Improvements

• We updated the text to be framework neutral, such as now we call ndarray as tensor.
• Readers can click the tab in the HTML version to switch between frameworks, both colab button and discussion thread will change properly.
• We changed the release process, d2l.ai will host the latest release (i.e. the release branch), instead of the contents from the master branch. We unified the version number of both text and the d2l package. That's why we jumped from v0.8 to v0.14.0
• The notebook zip contains three folders, mxnet, pytorch and tensorflow (though we only build the PDF for mxnet yet).

Highlights

D2L is now runnable on Amazon SageMaker and Google Colab.

New Contents

The following chapters are re-organized:

• Natural Language Processing: Pretraining
• Natural Language Processing: Applications

The following sections are added:

• Subword Embedding (Byte-pair encoding)
• Bidirectional Encoder Representations from Transformers (BERT)
• The Dataset for Pretraining BERT
• Pretraining BERT
• Natural Language Inference and the Dataset
• Natural Language Inference: Using Attention
• Fine-Tuning BERT for Sequence-Level and Token-Level Applications
• Natural Language Inference: Fine-Tuning BERT

Improvements

There have been many light revisions and improvements throughout the book.

Highlights

• D2L is now based on the NumPy interface. All the code samples are rewritten.

New Contents

• Recommender Systems

  • Overview of Recommender Systems
  • The MovieLens Dataset
  • Matrix Factorization
  • AutoRec: Rating Prediction with Autoencoders
  • Personalized Ranking for Recommender Systems
  • Neural Collaborative Filtering for Personalized Ranking
  • Sequence-Aware Recommender Systems
  • Feature-Rich Recommender Systems
  • Factorization Machines
  • Deep Factorization Machines

• Appendix: Mathematics for Deep Learning

  • Geometry and Linear Algebraic Operations
  • Eigendecompositions
  • Single Variable Calculus
  • Multivariable Calculus
  • Integral Calculus
  • Random Variables
  • Maximum Likelihood
  • Distributions
  • Naive Bayes
  • Statistics
  • Information Theory

• Attention Mechanisms

  • Attention Mechanism
  • Sequence to Sequence with Attention Mechanism
  • Transformer

• Generative Adversarial Networks

  • Generative Adversarial Networks
  • Deep Convolutional Generative Adversarial Networks

• Preliminaries

  • Data Preprocessing
  • Calculus

Improvements

• The Preliminaries chapter is improved.
• More theoretical analysis is added to the Optimization chapter.

Preview Version

Hard copies of a D2L preview version based on this release (excluding chapters of Recommender Systems and Generative Adversarial Networks) are distributed at AWS re:Invent 2019 and NeurIPS 2019.

Change of Contents

We heavily revised the following chapters, especially during teaching STAT 157 at Berkeley.

• Preface
• Installation
• Introduction
• The Preliminaries: A Crashcourse
• Linear Neural Networks
• Multilayer Perceptrons
• Recurrent Neural Networks

The Community Are Translating D2L into Korean and Japanese

d2l-ko in Korean (website: ko.d2l.ai) joins d2l.ai! Thank Muhyun Kim, Kyoungsu Lee, Ji hye Seo, Jiyang Kang and many other contributors!

d2l-ja in Japanese (website: ja.d2l.ai) joins d2l.ai! Thank Masaki Samejima!

Thanks to Our Contributors

@alxnorden, @avinashingit, @bowen0701, @brettkoonce, Chaitanya Prakash Bapat, @cryptonaut, Davide Fiocco, @edgarroman, @gkutiel, John Mitro, Liang Pu, Rahul Agarwal, @mohamed-ali, @mstewart141, Mike Müller, @NRauschmayr, @Prakhar Srivastav, @sad-, @sfermigier, Sheng Zha, @sundeepteki, @topecongiro, @tpdi, @vermicelli, Vishaal Kapoor, @vishwesh5, @YaYaB, Yuhong Chen, Evgeniy Smirnov, @lgov, Simon Corston-Oliver, @IgorDzreyev, @trungha-ngx, @pmuens, @alukovenko, @senorcinco, @vfdev-5, @dsweet, Mohammad Mahdi Rahimi, Abhishek Gupta, @uwsd, @DomKM, Lisa Oakley, @vfdev-5, @bowen0701, @arush15june, @prasanth5reddy.

Contents

• Translated contents from https://github.com/d2l-ai/d2l-zh, including the following chapters

  • Introduction
  • A Taste of Deep Learning
  • Deep Learning Basics
  • Deep Learning Computation
  • Convolutional Neural Networks
  • Recurrent Neural Networks
  • Optimization Algorithms
  • Computational Performance
  • Computer Vision
  • Natural Language Processing
  • Appendix

• Added new contents in the following chapters

  • Introduction
  • A Taste of Deep Learning
  • Deep Learning Basics
  • Deep Learning Computation
  • Convolutional Neural Networks

Style

• Improved HTML styles
• Improved PDF styles

Chinese Version

v1.0.0-rc0 is released: https://github.com/d2l-ai/d2l-zh/releases/tag/v1.0.0-rc0 The physical book will be published soon.

Thanks to Our Contributors

alxnorden, avinashingit, bowen0701, brettkoonce, Chaitanya Prakash Bapat, cryptonaut, Davide Fiocco, edgarroman, gkutiel, John Mitro, Liang Pu, Rahul Agarwal, mohamed-ali, mstewart141, Mike Müller, NRauschmayr, Prakhar Srivastav, sad-, sfermigier, Sheng Zha, sundeepteki, topecongiro, tpdi, vermicelli, Vishaal Kapoor, vishwesh5, YaYaB