Catalog

One 、 Introduce

Two 、 Basic knowledge of

3、 ... and 、Kaiming Initialization assumptions  

Four 、Kaiming Simple formula derivation of initialization

1. Forward propagation

2. Back propagation

5、 ... and 、Pytorch Realization

Deep learning parameter initialization series ：

（ One ）Xavier initialization With code

（ Two ）Kaiming initialization With code

One 、 Introduce

        Kaiming Initialize the paper address ：https://arxiv.org/abs/1502.01852

        Xavier Initialize at ReLU Layer performance is not good , The main reason is relu The layer maps negative numbers to 0, Affect the overall variance . and Xavier The activation function applicable to the initialization method is limited ： Requirements about 0 symmetry ; linear . and ReLU The activation function does not meet these conditions , Experiments can also verify Xavier Initialization does not apply to ReLU Activation function . So he Kaiming has made improvements , Put forward Kaiming initialization , At first, it was mainly used in computer vision 、 Convolution network .

Two 、 Basic knowledge of

1. Suppose that random variables X And random variables Y Are independent of each other , Then there are

        (1) 

 2. A formula for finding variance by expectation , The expectation that the variance is equal to the square minus the square of the expectation .

                (2)

 3. Independent variable product formula

        (3)

4. Continuous random variable X The probability density function of is f(x), If integral converges absolutely , The expected formula is as follows ：

                        (4)

3、 ... and 、Kaiming Initialization assumptions  

        And Xavier Initialization is similar ,Kaiming Initialization also applies Glorot Conditions , That is, our initialization strategy should make the activation value of each layer consistent with the variance of the state gradient in the propagation process ;Kaiming The initialized parameters still meet the mean value of 0, And the average weight during the update process has always been 0.

        And Xavier Initialize different ,Kaiming Initialization no longer requires that the average output of each layer be 0（ because Relu Such an activation function cannot be done ）; Of course, it is no longer required f′(0)=1.

        Kaiming In the initialization , Forward propagation and back propagation use their own initialization strategies , But ensure that the variance of each layer in forward propagation and the variance of gradient in back propagation are 1.

Four 、Kaiming The initialization of the Simple formula derivation

         We use convolution to derive , And the activation function uses ReLU.

1. Forward propagation

         For a layer of convolution , Yes ：

                        (5)

          among Is the output before activating the function , Is the number of weights , Weight. , It's input .

         according to (3) type , Can be (4) The formula is deduced as ：

        (6)

          Based on assumptions , however It is the upper layer that passes ReLU Got , therefore , be ：

                （7）

  adopt (2) Type available , be （7） The formula is deduced as ：

                        (8)

According to the expectation formula （4）, Pass the first Layer output to find this expectation , We have , among Express ReLU function .

                         (9)

among Represents the probability density function , because When , So you can remove less than 0 The range of , And greater than 0 When , Can be launched ：

                   （10）

because It is assumed that 0 It is symmetrically distributed around and the mean value is 0, therefore Also in 0 The nearby distribution is symmetrical , And the mean is 0( Assume here that the offset is 0), be

       （11） 

therefore The expectation is ：

              （12）

  According to the formula (2), because Our expectations are equal to 0, So there is ：

The type （12） Derived as ：

                        （13）

take (13) Type in (8) type ：

                        (14)

Carry out forward propagation from the first floor , The variance of a certain layer can be obtained as  :

there Is the input sample , We will normalize it , therefore , Now let the output variance of each layer be equal to 1, namely :

So when it spreads forward ,Kaiming The implementation of initialization is the following uniform distribution ：

Gaussian distribution ：

2. Back propagation

Because back propagation

                (15)

  among Means that the loss function takes its derivative .  Is the parameter

according to （3） type ：

  among Indicates the number of output channels during back propagation , At last, it comes to

So back propagation ,Kaiming The implementation of initialization is the following uniform distribution ：

Gaussian distribution ：

5、 ... and 、Pytorch Realization

import torch

class DemoNet(torch.nn.Module):
    def __init__(self):
        super(DemoNet, self).__init__()
        self.conv1 = torch.nn.Conv2d(1, 1, 3)
        print('random init:', self.conv1.weight)
        '''
            kaiming  The initialization method obeys uniform distribution  U～（-bound, bound）, bound = sqrt(6/(1+a^2)*fan_in)
            a  Is the slope of the negative half axis of the activation function ,relu  yes  0
            mode-  Optional  fan_in  or  fan_out, fan_in  When propagating forward , The variance is consistent ; fan_out  When making back propagation , The variance is consistent 
            nonlinearity-  Optional  relu  and  leaky_relu , The default value is  . leaky_relu
        '''
        torch.nn.init.kaiming_uniform_(self.conv1.weight, a=0, mode='fan_out')
        print('xavier_uniform_:', self.conv1.weight)

        '''
            kaiming  The initialization method obeys the normal distribution , This is a  0  The normal distribution of the mean ,N～ (0,std), among  std = sqrt(2/(1+a^2)*fan_in)
            a  Is the slope of the negative half axis of the activation function ,relu  yes  0
            mode-  Optional  fan_in  or  fan_out, fan_in  When propagating forward , The variance is consistent ;fan_out  When making back propagation , The variance is consistent 
            nonlinearity-  Optional  relu  and  leaky_relu , The default value is  . leaky_relu
        '''
        torch.nn.init.kaiming_normal_(self.conv1.weight, a=0, mode='fan_out')
        print('kaiming_normal_:', self.conv1.weight)


if __name__ == '__main__':
    demoNet = DemoNet()