当前位置：网站首页>Machine learning BP (back propagation) neural network

Machine learning BP (back propagation) neural network

2022-07-18 21:16:00 【InfoQ】

Author：AXYZdong

Automation Engineering Male

A little bit of thinking , Have a little idea , A little bit rational ！

Set a small goal , Try to be a habit ！ Meet a better self in the most beautiful years ！

Preface

Neural network control is 20 century 80 s , In artificial neural networks （Artificial Neural Networks,ANN） One of the frontier disciplines in the field of automatic control developed on the basis of the breakthrough in research . It is a new branch of intelligent control , To solve the complicated

nonlinear 、 Not sure 、 Unascertained

The control of the system opens up a new way .

1. What is neural network ？

Neural networks include

Biological neural networks

and

Artificial neural network

Biological neural networks , Generally refers to biological brain neurons 、 cells 、 A network of contacts, etc , Used to produce biological consciousness 、 Help organisms think and act .

Artificial neural network (Artificial Neural Networks, Shorthand for ANN) Also referred to as neural network , It's a kind of imitation of animal neural network behavior characteristics , A mathematical model for distributed parallel information processing . This network depends on the complexity of the system , By adjusting the interconnecting relationships between a large number of internal nodes , In order to achieve the purpose of processing information .

2. BP Theoretical basis of neural network

BP (Back Propagation) Neural network is 1986 Year by year Rumelhart and McClelland The concept put forward by the leading scientist , It is a kind of multilayer feedforward neural network trained according to the error back propagation algorithm , It is the most widely used neural network .

2.1 perceptron （Perceptron） The Internet

perceptron （Perceptron） Network is the result of early bionics research , stay 1950s from Frank Rosenblatt First introduction , It's a neural network model .

The retina of the human eye is composed of light sensing elements arranged in a matrix , The output of these sensing elements is connected to some neurons . When the input reaches a certain level or a certain type of input is generated , Neurons give output .

This is the basic idea of perceptron , That is, when the input activity exceeds a certain internal threshold , Neurons are activated , When the input has certain characteristics , The departure rate also increases .

perceptron ——BP A single node in a neural network

[ ] By input 、 The weight 、 bias 、 Activation function 、 The output consists of .

Input node ：

The weight ：

bias ：

Activation function ：

Output node ：

2.2 BP The structure and propagation rules of Neural Networks

BP Neural network is a typical nonlinear algorithm .

BP The neural network consists of

Input layer

、

Hidden layer

（ Also known as the middle tier ） and

Output layer

constitute , There are one or more hidden layers . Each node can have several layers . The connection state of nodes between layers passes

The weight

To embody .

There is only one hidden layer ： Traditional shallow neural network ; There are multiple hidden layers ： Deep learning neural network .

▲ A typical BP The structure of neural networks

BP The core steps of neural network are as follows . among , Solid lines represent forward propagation , The dotted line represents back propagation .

Forward propagation data （ Information 、 The signal ） Input from the input end , Along the direction of the network , Multiply by the corresponding weight and add , Then the result is calculated in the activation function as input , Pass the calculated result as input to the next node . Sequentially calculate , Until you get the final result . Through the perceptron of each layer , Layer upon layer calculation , Get the output , The output of each node is used as the input of the next node . This process is forward propagation .

Back propagation compares the output result with the expected output result , The error generated by the comparison is back propagated by the network , The essence is a “ Negative feedback ” The process of . Through many iterations , Constantly adjust the weight of each node on the network （ to update ）, Weight adjustment （ to update ） Gradient descent method .

sigmoid function

characteristic ：

For the sake of simplicity , The activation functions involved in this paper are sigmoid function

From the perspective of machine learning , Our task is ： Given a set of data sets , It contains input data

And the real output

, How to find the best set of neural network parameters , Make the inferred value calculated by the network

Can be compared with the real value

The highest degree of anastomosis ?

2.3 Gradient descent learning

In the process of forward propagation , There is one

Are you satisfied with the expected results

Link , In this link, there will be an error between the actual output and the expected output , In order to reduce this error , This translates into an optimization process , For any optimization problem , There will always be an objective function (objective function), The objective function is

Loss function （Loss function）

In order to minimize the error between the actual output and the expected output , We need to find the minimum value of this function .

Find the minimum value of the function through iterative method

Analytic solution ： Push down the calculation through strict publicity , The exact solution of the equation given , Satisfy the equation with any accuracy . Numerical solution ： under certain conditions , The solution obtained by some approximate calculation , It can satisfy the equation with given accuracy . Iterative method to find the minimum value of the function It is through

gradient descent + iteration

To find numerical solutions .

Iterative methods usually go through many times , Because finding the minimum value of a function may require many iterations , And in every iteration , The weights between nodes of each layer will also be updated iteratively .

： Adjustment amount ;

Smooth items .

In this way, each iteration will produce a weight update , Then, the updated weights are propagated forward with the training samples , If the result is not satisfactory , Back propagation , Continue iteration . So back and forth , Until you get a satisfactory result .

Gradient descent learning , Some alpine skiers are always looking for the area with the largest slope to slide down . When he is in A Point position , Descend along the maximum gradient route , Reach the local minimum , Then stop sliding ; If he is from B Start sliding down at , Then he will finally reach the global minimum .

▲ Schematic diagram of gradient descent of error function

BP Why choose gradient descent method for neural network back propagation ？

Gradient descent method is a general method for training neural networks and linear classifiers . The slope is the derivative of the function .

In fact ,

It may not be a scalar , It's a vector . Refer to the knowledge of multivariate functions , A gradient is a vector of partial derivatives . Each element on the gradient will indicate the slope of the function in all directions at that point , Therefore, the gradient points to the direction in which the function changes fastest . That is, point to the fastest direction of enlargement and the fastest direction of diminution , Corresponding to positive gradient and negative gradient .

A lot of deep learning is actually computing gradients , Then iterate through the gradient , Update parameter vector .

2.4 Improvement of learning algorithm

Introduce the momentum term

, For learning rate ;

Is the momentum factor ,

Learning rate of variable step method

Bad choice , If it is too small , Convergence is too slow ; If you choose too big , It is possible to over correct , Cause oscillation and even divergence . therefore , The variable step size method can be used to improve .

When the gradient direction is the same for two consecutive iterations , Indicates that the descent is too slow , At this time, the step length can be doubled ; When the gradient direction of two consecutive iterations is opposite , Indicates excessive descent , At this time, the step length can be halved .

The combination of momentum term and variable step size method

2.5 BP Application of neural network

BP The field of neural network application BP Neural networks are widely used in many research fields , Like the environment 、 biological 、 Medical Science 、 meteorological 、 astronomical 、 Geography 、 economics 、 Management 、 Industry 、 statistical 、 Computer 、 mobile communication 、 space flight 、 information technology 、 automation 、 energy 、 The new material 、 Marine and other fields .

BP The main aspects of neural network application
forecast
： Power operation load 、 Hemoglobin concentration 、 housing price 、 The stock market 、 Water demand 、 The wind speed 、 Geological disasters, etc .
evaluation
： City Safety 、 The river is healthy 、 The quality of teaching 、 Network security 、 water environment 、 ecological risk 、 Disaster losses, etc .
The image processing
： Image denoising 、 Image classification 、 Image restoration 、 Image compression 、 Image correction 、 Image segmentation 、 Image encryption, etc .
Simulation
： Mobile robot obstacle avoidance simulation 、 Logistics traffic simulation 、 UAV attitude adaptive simulation 、 Industrial control simulation .

3. 9 Line code implementation BP neural network

Code from github：

https://github.com/miloharper/simple-neural-network

from numpy import exp, array, random, dot 
# from numpy Call in library exp( Exponential function ）、array( Array 〉、random（ Random function )、dot( Matrix multiplication function ).
training_set_inputs = array([[0, 0, 1], [1, 1, 1], [1, 0, 1], [0, 1, 1]])
#bp Input of neural network training part .
training_set_outputs = array([[0, 1, 1, 0]]).T
#bp Output of neural network training part ,.T Represents matrix transpose .
random.seed(1)
# Use random functions to generate random numbers , Use seed Function can ensure that the random numbers generated each time are consistent .
synaptic_weights = 2 * random.random((3, 1)) - 1
# Generate a random array , The array format is 3 That's ok 1 Column , Used to store initial weights .
for iteration in range(10000):
 output = 1 / (1 + exp(-(dot(training_set_inputs, synaptic_weights))))
 # Use for Sentence loop 10000 Time , The input and weight of the training set are dot Do matrix multiplication , Enter the result of multiplication into sigmoid function , Then assign the result to output.
 synaptic_weights += dot(training_set_inputs.T, (training_set_outputs - output) * output * (1 - output))
 # Weighted i Adjustment and adoption “ Error weighted derivative &quot;&quot; The formula .
print (1 / (1 + exp(-(dot(array([1, 0, 0]), synaptic_weights)))))
#synaptic_weights Is the final weight after adjustment , Array （ matrix 〉[1,0,0] With this weight matrix dot Function to multiply , The result of multiplication is introduced into sigmoid function , Get the final result .

Running results ：

4. summary

BP The propagation process of neural network includes forward propagation and back propagation , The essence of back propagation is
“ Negative feedback ”
. This is similar to the closed-loop system in control , Through feedback , Use the deviation to correct the deviation , So as to achieve satisfactory output effect ;

Dealing with errors , Take advantage of
Gradient descent method + Multiple iterations
The way , Find the smallest error . In the process , Every iteration , The weights between nodes in different layers will be updated once . Because of the dynamic update of weights , The error obtained by each forward propagation is also dynamically updated , Until the desired output effect is obtained ;

Mastering the principle is the foundation , Code implementation is the key
. At the end of the article 9 Line code implementation BP neural network , It reflects BP The principle of neural network , It is a single neural network （simple-neural-network）, If you want to realize multiple neural networks , It may be very complicated .