BP神经网络推导入门：从数学原理到Python实现

1次阅读

共计 4399 个字符，预计需要花费 11 分钟才能阅读完成。

假设我们有一个最简单的单层感知机，输入为 $x$，权重为 $w$，偏置为 $b$，激活函数为 $\sigma$，则输出为：

$$ y = \sigma(wx + b) $$

对于平方损失函数 $L = \frac{1}{2}(y – t)^2$（t 为真实值），我们需要计算损失对权重 $w$ 的偏导数：

$$ \frac{\partial L}{\partial w} = \frac{\partial L}{\partial y} \frac{\partial y}{\partial (wx+b)} \frac{\partial (wx+b)}{\partial w} $$

展开就是：

$$ \frac{\partial L}{\partial w} = (y – t) \cdot \sigma'(wx + b) \cdot x $$

这就是最基本的链式法则应用。理解了这一点，就能更好地扩展到多层网络。

现在考虑一个 3 层网络（输入层、隐藏层、输出层）。定义以下符号：

$W^{(1)}$: 输入层到隐藏层的权重矩阵
$b^{(1)}$: 隐藏层的偏置
$W^{(2)}$: 隐藏层到输出层的权重矩阵
$b^{(2)}$: 输出层的偏置
$\sigma$: Sigmoid 激活函数

前向传播过程：

隐藏层输入：$z^{(1)} = W^{(1)}x + b^{(1)}$
隐藏层输出：$a^{(1)} = \sigma(z^{(1)})$
输出层输入：$z^{(2)} = W^{(2)}a^{(1)} + b^{(2)}$
最终输出：$a^{(2)} = \sigma(z^{(2)})$

损失函数（以平方误差为例）：

$$ L = \frac{1}{2} \sum (a^{(2)} – t)^2 $$

关键是要计算损失对每一层参数的梯度。我们采用反向传播算法，从输出层开始逐层计算。

计算损失对输出层权重 $W^{(2)}$ 的偏导：

$$ \frac{\partial L}{\partial W^{(2)}} = \frac{\partial L}{\partial a^{(2)}} \frac{\partial a^{(2)}}{\partial z^{(2)}} \frac{\partial z^{(2)}}{\partial W^{(2)}} $$

计算各项：

$\frac{\partial L}{\partial a^{(2)}} = (a^{(2)} – t)$
$\frac{\partial a^{(2)}}{\partial z^{(2)}} = \sigma'(z^{(2)}) = a^{(2)}(1 – a^{(2)})$（Sigmoid 导数）
$\frac{\partial z^{(2)}}{\partial W^{(2)}} = a^{(1)T}$

所以最终：

$$ \delta^{(2)} = (a^{(2)} – t) \odot a^{(2)}(1 – a^{(2)}) $$
$$ \frac{\partial L}{\partial W^{(2)}} = a^{(1)T} \delta^{(2)} $$

类似地，计算隐藏层权重 $W^{(1)}$ 的梯度：

$$ \delta^{(1)} = (W^{(2)T} \delta^{(2)}) \odot a^{(1)}(1 – a^{(1)}) $$
$$ \frac{\partial L}{\partial W^{(1)}} = x^T \delta^{(1)} $$

下面是完整的 NumPy 实现：

import numpy as np

class NeuralNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        # 初始化权重（使用较小的随机值）self.W1 = np.random.randn(input_size, hidden_size) * 0.01
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size) * 0.01
        self.b2 = np.zeros((1, output_size))

    def sigmoid(self, x):
        return 1 / (1 + np.exp(-x))

    def sigmoid_derivative(self, x):
        return x * (1 - x)

    def forward(self, X):
        # 前向传播
        self.z1 = np.dot(X, self.W1) + self.b1
        self.a1 = self.sigmoid(self.z1)
        self.z2 = np.dot(self.a1, self.W2) + self.b2
        self.a2 = self.sigmoid(self.z2)
        return self.a2

    def backward(self, X, y, output, learning_rate):
        # 反向传播
        m = X.shape[0]  # 样本数

        # 输出层误差
        delta2 = (output - y) * self.sigmoid_derivative(output)
        dW2 = np.dot(self.a1.T, delta2) / m
        db2 = np.sum(delta2, axis=0, keepdims=True) / m

        # 隐藏层误差
        delta1 = np.dot(delta2, self.W2.T) * self.sigmoid_derivative(self.a1)
        dW1 = np.dot(X.T, delta1) / m
        db1 = np.sum(delta1, axis=0, keepdims=True) / m

        # 更新参数
        self.W1 -= learning_rate * dW1
        self.b1 -= learning_rate * db1
        self.W2 -= learning_rate * dW2
        self.b2 -= learning_rate * db2

    def train(self, X, y, epochs, learning_rate):
        for i in range(epochs):
            output = self.forward(X)
            self.backward(X, y, output, learning_rate)

            # 每 100 次迭代打印损失
            if i % 100 == 0:
                loss = np.mean(0.5 * (y - output) ** 2)
                print(f"Epoch {i}, Loss: {loss}")

# 使用示例
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])  # 输入
Y = np.array([[0], [1], [1], [0]])  # 异或问题输出

nn = NeuralNetwork(2, 4, 1)  # 输入 2 个节点，隐藏层 4 个节点，输出 1 个节点
nn.train(X, Y, epochs=10000, learning_rate=0.1)

为了验证我们的反向传播实现是否正确，可以实施梯度检查：

def gradient_check(nn, X, y, epsilon=1e-7):
    # 获取所有参数
    parameters = {"W1": nn.W1, "b1": nn.b1, "W2": nn.W2, "b2": nn.b2}

    # 计算数值梯度
    for param_name in parameters:
        param = parameters[param_name]
        grad_numerical = np.zeros_like(param)

        for i in range(param.shape[0]):
            for j in range(param.shape[1]):
                # 保存原始值
                original_value = param[i,j]

                # 计算 f(theta + epsilon)
                param[i,j] = original_value + epsilon
                output_plus = nn.forward(X)
                loss_plus = np.mean(0.5 * (y - output_plus) ** 2)

                # 计算 f(theta - epsilon)
                param[i,j] = original_value - epsilon
                output_minus = nn.forward(X)
                loss_minus = np.mean(0.5 * (y - output_minus) ** 2)

                # 恢复原始值
                param[i,j] = original_value

                # 计算数值梯度
                grad_numerical[i,j] = (loss_plus - loss_minus) / (2 * epsilon)

        # 获取反向传播计算的梯度
        output = nn.forward(X)
        nn.backward(X, y, output, learning_rate=0.1)  # 学习率不影响梯度计算

        if param_name == "W1":
            grad_backprop = nn.dW1
        elif param_name == "b1":
            grad_backprop = nn.db1
        elif param_name == "W2":
            grad_backprop = nn.dW2
        else:
            grad_backprop = nn.db2

        # 计算相对误差
        numerator = np.linalg.norm(grad_backprop - grad_numerical)
        denominator = np.linalg.norm(grad_backprop) + np.linalg.norm(grad_numerical)
        relative_error = numerator / denominator

        print(f"Gradient check for {param_name}: relative error = {relative_error}")
        if relative_error > 1e-7:
            print("WARNING: Backprop implementation may be incorrect!")

# 执行梯度检查
nn = NeuralNetwork(2, 4, 1)
gradient_check(nn, X, Y)