深入理解bp反向传播算法推导：从数学原理到代码实现

1次阅读

共计 1929 个字符，预计需要花费 5 分钟才能阅读完成。

反向传播（Backpropagation）是神经网络训练的核心算法，通过计算损失函数对网络参数的梯度，指导参数更新方向。其本质是链式法则的递归应用，将误差从输出层逐层反向传播至输入层。

$x^{(i)}$: 第 i 层输入
$W^{(i)}$: 第 i 层权重矩阵
$z^{(i)} = W^{(i)}x^{(i-1)} + b^{(i)}$: 加权输入
$a^{(i)} = \sigma(z^{(i)})$: 激活输出
$L$: 损失函数

输出层误差：
$$\delta^{(L)} = \frac{\partial L}{\partial z^{(L)}} = \frac{\partial L}{\partial a^{(L)}} \odot \sigma'(z^{(L)})$$

隐藏层误差传播：
$$\delta^{(l)} = ((W^{(l+1)})^T \delta^{(l+1)}) \odot \sigma'(z^{(l)})$$

权重梯度：
$$\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T$$

偏置梯度：
$$\frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}$$

import numpy as np

class NeuralNetwork:
    def __init__(self, layers):
        self.weights = [np.random.randn(y, x) * 0.1 
                       for x, y in zip(layers[:-1], layers[1:])]
        self.biases = [np.zeros((y, 1)) for y in layers[1:]]

    def forward(self, x):
        """前向传播"""
        self.activations = [x]
        self.zs = []
        for w, b in zip(self.weights, self.biases):
            z = np.dot(w, self.activations[-1]) + b
            self.zs.append(z)
            self.activations.append(self.sigmoid(z))
        return self.activations[-1]

    def backward(self, x, y):
        """反向传播"""
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        nabla_b = [np.zeros(b.shape) for b in self.biases]

        # 输出层误差
        delta = (self.activations[-1] - y) * self.sigmoid_prime(self.zs[-1])
        nabla_b[-1] = delta
        nabla_w[-1] = np.dot(delta, self.activations[-2].T)

        # 隐藏层误差反向传播
        for l in range(2, len(self.weights)+1):
            z = self.zs[-l]
            sp = self.sigmoid_prime(z)
            delta = np.dot(self.weights[-l+1].T, delta) * sp
            nabla_b[-l] = delta
            nabla_w[-l] = np.dot(delta, self.activations[-l-1].T)

        return nabla_w, nabla_b

    def sigmoid(self, z):
        return 1.0/(1.0+np.exp(-z))

    def sigmoid_prime(self, z):
        return self.sigmoid(z)*(1-self.sigmoid(z))