深入解析bp反向传播算法：从数学原理到Python实现

1次阅读

共计 2480 个字符，预计需要花费 7 分钟才能阅读完成。

反向传播 (Backpropagation) 是训练神经网络的核心算法，其本质是链式法则的递归应用。设神经网络有 $L$ 层，第 $l$ 层的权重矩阵为 $W^l$，偏置为 $b^l$，激活函数为 $\sigma^l$。对于单个样本 $(x,y)$，前向传播过程为：

$$
\begin{aligned}
z^l &= W^l a^{l-1} + b^l \
a^l &= \sigma^l(z^l)
\end{aligned}
$$

定义损失函数 $J$ 后，反向传播通过计算 $\frac{\partial J}{\partial W^l}$ 和 $\frac{\partial J}{\partial b^l}$ 来更新参数。关键推导步骤如下：

计算输出层误差：
$$
\delta^L = \frac{\partial J}{\partial a^L} \odot \sigma’^L(z^L)
$$
反向传播误差：
$$
\delta^l = ((W^{l+1})^T \delta^{l+1}) \odot \sigma’^l(z^l)
$$
计算梯度：
$$
\frac{\partial J}{\partial W^l} = \delta^l (a^{l-1})^T, \quad \frac{\partial J}{\partial b^l} = \delta^l
$$

不同优化器在反向传播中的参数更新策略差异显著：

SGD：
$$
W \leftarrow W – \eta \nabla_W J
$$
简单但容易震荡，收敛慢
Momentum：
$$
\begin{aligned}
v &\leftarrow \gamma v + \eta \nabla_W J \
W &\leftarrow W – v
\end{aligned}
$$
引入动量项减轻震荡
Adam：
$$
\begin{aligned}
m &\leftarrow \beta_1 m + (1-\beta_1)\nabla_W J \
v &\leftarrow \beta_2 v + (1-\beta_2)(\nabla_W J)^2 \
\hat{m} &= m/(1-\beta_1^t) \
\hat{v} &= v/(1-\beta_2^t) \
W &\leftarrow W – \eta \hat{m}/(\sqrt{\hat{v}}+\epsilon)
\end{aligned}
$$
自适应学习率，实践中表现优异

import numpy as np

class NeuralNetwork:
    def __init__(self, layers):
        self.weights = [np.random.randn(y,x)*0.01 
                        for x,y in zip(layers[:-1], layers[1:])]
        self.biases = [np.zeros((y,1)) for y in layers[1:]]

    def forward(self, x):
        """前向传播"""
        a = x
        for w, b in zip(self.weights, self.biases):
            z = np.dot(w, a) + b
            a = 1/(1+np.exp(-z))  # Sigmoid
        return a

    def backward(self, x, y):
        """反向传播"""
        # 前向计算各层激活值
        activations = [x]
        zs = []
        for w, b in zip(self.weights, self.biases):
            z = np.dot(w, activations[-1]) + b
            zs.append(z)
            activations.append(1/(1+np.exp(-z)))

        # 反向传播误差
        delta = (activations[-1] - y) * activations[-1] * (1-activations[-1])
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w[-1] = np.dot(delta, activations[-2].T)
        nabla_b[-1] = delta

        for l in range(2, len(self.weights)+1):
            delta = np.dot(self.weights[-l+1].T, delta) * \
                    activations[-l] * (1-activations[-l])
            nabla_w[-l] = np.dot(delta, activations[-l-1].T)
            nabla_b[-l] = delta

        return nabla_w, nabla_b