深度学习入门：从零理解bp误差反向传播算法

1次阅读

没有评论

共计 2473 个字符，预计需要花费 7 分钟才能阅读完成。

神经网络通过堆叠神经元层实现复杂函数拟合。前向传播计算预测值时，每个神经元的输出为：

$$
a_j^l = \sigma(\sum_k w_{jk}^l a_k^{l-1} + b_j^l)
$$

其中 $\sigma$ 为激活函数，$w$ 和 $b$ 分别代表权重和偏置。训练的核心目标是调整这些参数使得损失函数 $L$ 最小化，这正是反向传播算法的使命。

反向传播实质是复合函数求导的链式法则应用。以平方损失函数为例：

输出层误差项：
$$\delta^L = \frac{\partial L}{\partial a^L} \odot \sigma'(z^L)$$
隐藏层误差传播：
$$\delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l)$$
参数梯度计算：
$$\frac{\partial L}{\partial w^l} = \delta^l (a^{l-1})^T$$
$$\frac{\partial L}{\partial b^l} = \delta^l$$

import numpy as np

class NeuralNetwork:
    def __init__(self, layers):
        # 初始化权重（He 初始化）self.weights = [np.random.randn(y, x)/np.sqrt(x) 
                        for x, y in zip(layers[:-1], layers[1:])]
        self.biases = [np.zeros((y, 1)) for y in layers[1:]]

    def forward(self, x):
        """前向传播计算"""
        for w, b in zip(self.weights, self.biases):
            x = sigmoid(np.dot(w, x) + b)
        return x

    def backprop(self, x, y):
        """反向传播计算梯度"""
        # 存储各层激活值和加权输入
        activations = [x]
        zs = []

        # 前向传播
        for w, b in zip(self.weights, self.biases):
            z = np.dot(w, activations[-1]) + b
            zs.append(z)
            activations.append(sigmoid(z))

        # 反向传播
        delta = (activations[-1] - y) * sigmoid_prime(zs[-1])
        nabla_w = [np.zeros_like(w) for w in self.weights]
        nabla_b = [np.zeros_like(b) for b in self.biases]
        nabla_w[-1] = np.dot(delta, activations[-2].T)
        nabla_b[-1] = delta

        for l in range(2, len(self.weights)+1):
            z = zs[-l]
            delta = np.dot(self.weights[-l+1].T, delta) * sigmoid_prime(z)
            nabla_w[-l] = np.dot(delta, activations[-l-1].T)
            nabla_b[-l] = delta

        return nabla_w, nabla_b

def sigmoid(z):
    return 1.0/(1.0+np.exp(-z))

def sigmoid_prime(z):
    return sigmoid(z)*(1-sigmoid(z))

数值梯度验证是调试的关键步骤：

def gradient_check(network, x, y, epsilon=1e-7):
    # 计算反向传播梯度
    nabla_w, nabla_b = network.backprop(x, y)

    # 数值梯度计算
    for layer in range(len(network.weights)):
        for i in range(network.weights[layer].shape[0]):
            for j in range(network.weights[layer].shape[1]):
                # 正向扰动
                network.weights[layer][i,j] += epsilon
                loss_plus = np.sum((network.forward(x) - y)**2)
                # 负向扰动
                network.weights[layer][i,j] -= 2*epsilon
                loss_minus = np.sum((network.forward(x) - y)**2)
                # 恢复原值
                network.weights[layer][i,j] += epsilon
                # 数值梯度
                numeric_grad = (loss_plus - loss_minus)/(2*epsilon)
                # 比较差异
                assert abs(nabla_w[layer][i,j] - numeric_grad) < 1e-5

当网络层数较深时，梯度可能呈现指数级衰减或增长。以 sigmoid 函数为例，其导数最大值为 0.25，经过 $n$ 层传播后梯度最多缩小 $0.25^n$。解决方案包括：