BP神经网络原理详解：前向传播、反向传播与链式求导实战

1次阅读

没有评论

共计 3388 个字符，预计需要花费 9 分钟才能阅读完成。

BP（Back Propagation）神经网络是一种多层前馈神经网络，通过反向传播算法调整权重和偏置。其核心包含两个过程：

前向传播 ：输入数据通过各层加权求和并激活，最终得到预测输出
反向传播 ：根据预测输出与真实值的误差，逆向计算梯度并更新参数

以单隐藏层网络为例，设输入层维度 $n$，隐藏层 $h$，输出层 $m$：

初始化参数矩阵：
$W_1 \in \mathbb{R}^{n×h}$（输入层到隐藏层权重）
$b_1 \in \mathbb{R}^h$（隐藏层偏置）
$W_2 \in \mathbb{R}^{h×m}$（隐藏层到输出层权重）
$b_2 \in \mathbb{R}^m$（输出层偏置）
前向计算流程：
隐藏层输出：$z_1 = XW_1 + b_1$，$a_1 = \sigma(z_1)$
输出层结果：$z_2 = a_1W_2 + b_2$，$\hat{y} = softmax(z_2)$
其中 $\sigma$ 为 ReLU 激活函数：$\sigma(x) = max(0,x)$

使用交叉熵损失：
$$ L = -\frac{1}{N}\sum_{i=1}^N\sum_{j=1}^m y_j^{(i)}\log(\hat{y}_j^{(i)}) $$

通过链式法则逐层求导：

输出层梯度：
$$ \frac{\partial L}{\partial z_2} = \hat{y} – y $$
隐藏层梯度：
$$ \frac{\partial L}{\partial W_2} = a_1^T \frac{\partial L}{\partial z_2} $$
$$ \frac{\partial L}{\partial b_2} = \sum \frac{\partial L}{\partial z_2} $$
$$ \frac{\partial L}{\partial a_1} = \frac{\partial L}{\partial z_2} W_2^T $$
$$ \frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial a_1} \odot \sigma'(z_1) $$
输入层梯度：
$$ \frac{\partial L}{\partial W_1} = X^T \frac{\partial L}{\partial z_1} $$
$$ \frac{\partial L}{\partial b_1} = \sum \frac{\partial L}{\partial z_1} $$

import numpy as np
from sklearn.datasets import make_classification

class BPNetwork:
    def __init__(self, input_size, hidden_size, output_size):
        # 参数初始化
        self.W1 = np.random.randn(input_size, hidden_size) * 0.01
        self.b1 = np.zeros(hidden_size)
        self.W2 = np.random.randn(hidden_size, output_size) * 0.01
        self.b2 = np.zeros(output_size)

    def relu(self, x):
        return np.maximum(0, x)

    def softmax(self, x):
        exps = np.exp(x - np.max(x, axis=1, keepdims=True))
        return exps / np.sum(exps, axis=1, keepdims=True)

    def forward(self, X):
        self.z1 = np.dot(X, self.W1) + self.b1
        self.a1 = self.relu(self.z1)
        self.z2 = np.dot(self.a1, self.W2) + self.b2
        self.y_hat = self.softmax(self.z2)
        return self.y_hat

    def backward(self, X, y, lr=0.01):
        m = X.shape[0]

        # 输出层梯度
        dz2 = self.y_hat - y
        dw2 = np.dot(self.a1.T, dz2) / m
        db2 = np.sum(dz2, axis=0) / m

        # 隐藏层梯度
        da1 = np.dot(dz2, self.W2.T)
        dz1 = da1 * (self.z1 > 0)  # ReLU 导数
        dw1 = np.dot(X.T, dz1) / m
        db1 = np.sum(dz1, axis=0) / m

        # 参数更新
        self.W1 -= lr * dw1
        self.b1 -= lr * db1
        self.W2 -= lr * dw2
        self.b2 -= lr * db2

    def train(self, X, y, epochs=100, lr=0.01):
        for epoch in range(epochs):
            y_hat = self.forward(X)
            loss = -np.mean(y * np.log(y_hat + 1e-8))
            self.backward(X, y, lr)
            if epoch % 10 == 0:
                print(f'Epoch {epoch}, Loss: {loss:.4f}')

# 数据准备
X, y = make_classification(n_samples=1000, n_features=20, n_classes=3, n_informative=15)
y_onehot = np.eye(3)[y]

# 模型训练
model = BPNetwork(20, 64, 3)
model.train(X, y_onehot, epochs=100, lr=0.1)

梯度消失 ：
使用 ReLU 激活函数替代 sigmoid
残差连接（ResNet 结构）
梯度裁剪（Gradient Clipping）
梯度爆炸 ：
权重初始化调整（如 He 初始化）
批量归一化（BatchNorm）
梯度裁剪阈值设置

初始学习率建议范围：0.001~0.1

使用学习率衰减策略：

lr = initial_lr * (1 / (1 + decay_rate * epoch))

自适应优化器推荐：Adam, RMSprop

class BatchNorm:
    def __init__(self, dim, eps=1e-5, momentum=0.9):
        self.gamma = np.ones(dim)
        self.beta = np.zeros(dim)
        self.eps = eps
        self.momentum = momentum
        self.running_mean = np.zeros(dim)
        self.running_var = np.ones(dim)

    def __call__(self, x, train=True):
        if train:
            batch_mean = np.mean(x, axis=0)
            batch_var = np.var(x, axis=0)
            self.running_mean = self.momentum * self.running_mean + (1 - self.momentum) * batch_mean
            self.running_var = self.momentum * self.running_var + (1 - self.momentum) * batch_var
            x_norm = (x - batch_mean) / np.sqrt(batch_var + self.eps)
        else:
            x_norm = (x - self.running_mean) / np.sqrt(self.running_var + self.eps)
        return self.gamma * x_norm + self.beta