BP神经网络反向传播算法详解：从数学推导到Python实现

1次阅读

共计 3353 个字符，预计需要花费 9 分钟才能阅读完成。

让我们从一个经典的 XOR（异或）问题入手。假设我们有以下训练数据：

输入 1	输入 2	输出
0	0	0
0	1	1
1	0	1
1	1	0

一个简单的 2 层神经网络（输入层 2 个神经元，隐藏层 2 个神经元，输出层 1 个神经元）可以解决这个问题。前向传播的过程就是数据从输入层流向输出层的过程：

输入层到隐藏层：
$$h_1 = w_{11}x_1 + w_{12}x_2 + b_1$$
$$h_2 = w_{21}x_1 + w_{22}x_2 + b_2$$
$$a_1 = \sigma(h_1)$$
$$a_2 = \sigma(h_2)$$
隐藏层到输出层：
$$o = w_{31}a_1 + w_{32}a_2 + b_3$$
$$\hat{y} = \sigma(o)$$

其中 σ 是激活函数（如 sigmoid），$\hat{y}$ 是预测输出。

反向传播的核心是通过链式法则计算损失函数对各参数的梯度。以均方误差 (MSE) 损失为例：

$$L = \frac{1}{2}(y – \hat{y})^2$$

计算输出层梯度：
$$\frac{\partial L}{\partial w_{3j}} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial o} \cdot \frac{\partial o}{\partial w_{3j}} = -(y-\hat{y}) \cdot \sigma'(o) \cdot a_j$$
计算隐藏层梯度：
$$\frac{\partial L}{\partial w_{ij}} = \frac{\partial L}{\partial a_j} \cdot \frac{\partial a_j}{\partial h_j} \cdot \frac{\partial h_j}{\partial w_{ij}}$$
$$\frac{\partial L}{\partial a_j} = \sum_{k} \frac{\partial L}{\partial o_k} \cdot \frac{\partial o_k}{\partial a_j}$$
更新参数（梯度下降）：
$$w \leftarrow w – \eta \cdot \frac{\partial L}{\partial w}$$

import numpy as np
import matplotlib.pyplot as plt

class NeuralNetwork:
    def __init__(self):
        # 初始化权重（输入层 2 个神经元，隐藏层 2 个神经元，输出层 1 个神经元）self.weights1 = np.random.randn(2, 2)
        self.weights2 = np.random.randn(2, 1)
        self.bias1 = np.zeros((1, 2))
        self.bias2 = np.zeros((1, 1))

    def sigmoid(self, x):
        return 1 / (1 + np.exp(-x))

    def sigmoid_derivative(self, x):
        return x * (1 - x)

    def forward(self, X):
        self.hidden = self.sigmoid(np.dot(X, self.weights1) + self.bias1)
        self.output = self.sigmoid(np.dot(self.hidden, self.weights2) + self.bias2)
        return self.output

    def backward(self, X, y, output, learning_rate):
        # 计算输出层误差
        output_error = y - output
        output_delta = output_error * self.sigmoid_derivative(output)

        # 计算隐藏层误差
        hidden_error = output_delta.dot(self.weights2.T)
        hidden_delta = hidden_error * self.sigmoid_derivative(self.hidden)

        # 更新权重和偏置
        self.weights2 += self.hidden.T.dot(output_delta) * learning_rate
        self.bias2 += np.sum(output_delta, axis=0, keepdims=True) * learning_rate
        self.weights1 += X.T.dot(hidden_delta) * learning_rate
        self.bias1 += np.sum(hidden_delta, axis=0, keepdims=True) * learning_rate

    def train(self, X, y, epochs=10000, learning_rate=0.1):
        loss_history = []
        for epoch in range(epochs):
            # 前向传播
            output = self.forward(X)

            # 计算损失
            loss = np.mean(np.square(y - output))
            loss_history.append(loss)

            # 反向传播
            self.backward(X, y, output, learning_rate)

            if epoch % 1000 == 0:
                print(f'Epoch {epoch}, Loss: {loss}')

        # 绘制损失曲线
        plt.plot(loss_history)
        plt.title('Training Loss')
        plt.xlabel('Epoch')
        plt.ylabel('Loss')
        plt.show()

# 测试 XOR 问题
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

nn = NeuralNetwork()
nn.train(X, y)

for x in X:
    print(f'Input: {x}, Output: {nn.forward(np.array([x]))}')

学习率选择：
太大：可能导致震荡甚至发散
太小：收敛速度过慢
建议：从 0.1 开始尝试，观察损失曲线

梯度检查：

def gradient_check(network, X, y, epsilon=1e-7):
    # 计算数值梯度
    params = network.get_params()
    num_grad = np.zeros_like(params)

    for i in range(len(params)):
        params_plus = params.copy()
        params_plus[i] += epsilon
        network.set_params(params_plus)
        loss_plus = network.compute_loss(X, y)

        params_minus = params.copy()
        params_minus[i] -= epsilon
        network.set_params(params_minus)
        loss_minus = network.compute_loss(X, y)

        num_grad[i] = (loss_plus - loss_minus) / (2 * epsilon)

    # 与反向传播计算的梯度比较
    network.set_params(params)
    network.forward(X)
    network.backward(X, y)
    backprop_grad = network.get_gradients()

    diff = np.linalg.norm(num_grad - backprop_grad) / \
           np.linalg.norm(num_grad + backprop_grad)
    print(f'Gradient difference: {diff}')
    return diff < 1e-7