Stochastic gradient decline (Stochastic gradient descent)
Batch gradient decline (Batch gradient descent)
Gradient decrease (GD) is a common method of minimizing risk functions, loss function, random gradient drop and batch gradient declines are two iterative solution ideas, and the following is analyzed from formulas and achievements, if any aspect Write wrong, I hope the netizen corrects.
The following H (X) is a function of the combination, the Theta loss function, theta is the parameter, it is an iterative value, the Theta solves the function h (the Theta) that is ultimately fitted. Where m is the number of records of the training set, J is the number of parameters.
Be
Be
1. The solution idea of decline in mass gradient is as follows:
(1) Sete the Theta to be biased to get Jeta to get the gradient corresponding to each Theta.
Be
(2) Since it is to minimize the risk function, the Theta is updated in accordance with each of the gradients of each parameter.
(3) From the above formula, it can be noted that it is a global optimal solution, but each iteration is one step, it is necessary to use all the data for training set, if M is large, then you can know the iteration of this method. speed! ! Therefore, this introduces another method, and the random gradient decreases.
2. The solution idea of the decline in random gradient is as follows:
(1) The above risk function can be written as such a form, and the loss function corresponds to the particle size of each sample in the training set, and the above mass gradient decreases to all training samples:
Be
(2) The loss function of each sample is to get the corresponding gradient for the Theta, to update the Theta
(3) Random gradient decline is updated once through each sample, if the sample is large (for example, hundreds of thousand), then only tens of thousands or thousands of samples have been used, it has been itered. To the optimal solution, compare the above-mentioned batch gradient decline, iteration needs to use more than 100,000 training samples, one iteration is impossible, if it iterates 10 times, it is necessary to traverse the training sample 10 times. However, one of SGD accompanying is that there is more noise than BGD, so that SGD is not per-optimization direction every time it iterates.
3. For the above Linear Regression problem, will the random gradient decrease in a batch gradient?
(1) Batch gradient decline --- minimize the loss function of all training samples, so that the final solution is the global optimal solution, that is, the solicity parameter is minimized.
(2) Random gradient decrease --- minimize the loss function of each sample, although not the loss function of each iteration, the overall direction is to the global optimal solution, ultimately The result is often near the global optimal solution.
4, gradient decline is used to seek the best solution, which issues can be obtained global optimal? What questions may be partially optimal?
For the above Linear Regression problem, the optimization problem is unimodal to the Theta distribution, that is, only one peak on the top of the graph, so the gradient decline is finally obtained. The global optimal solution. However, for Multimodal issues, because there is a plurality of peak values, it is very likely that the final result of the gradient decline is to be topically optimal.
5, realization difference of random gradient and mass gradient
In the past, the NMF implementation in a blog post was taken as an example, and the difference between the two (Note: In fact, the code corresponding to Python is more intuitive, and later to write more Python!
[Java] View Plain
// Random gradient decline, update parameters, read full text, technology area
Ma Huateng proposed "Tencent Chupo Ecological Partner Program" to build a digital ecological community
Can the company now interact with humans by artificial intelligence?
Which elements of artificial intelligence changed the phone?
After splitting Windows, is Microsoft really play the return of the king?
Huichun Technology: Display IOT, the artificial intelligence field is a major breakthrough, strong push gesture recognition!
Our other product:
