BP神经网络原理详解与Python实现：从数学推导到代码实战

1次阅读

没有评论

共计 3366 个字符，预计需要花费 9 分钟才能阅读完成。

BP（Back Propagation）神经网络是一种多层前馈神经网络，通过误差反向传播算法进行训练。它的核心是通过链式法则计算损失函数对每个参数的梯度，然后利用梯度下降法更新权重。

前向传播是数据从输入层流向输出层的过程。对于第 l 层的第 j 个神经元，其输出为：

$$ a_j^l = \sigma(z_j^l) = \sigma(\sum_{i} w_{ji}^l a_i^{l-1} + b_j^l) $$

其中 $\sigma$ 是激活函数，$w_{ji}^l$ 是连接权重，$b_j^l$ 是偏置项。

反向传播算法基于链式法则计算梯度。定义损失函数为 $E$，则：

输出层误差：
$$ \delta_j^L = \frac{\partial E}{\partial z_j^L} = \frac{\partial E}{\partial a_j^L} \sigma'(z_j^L) $$
隐藏层误差：
$$ \delta_j^l = \sum_{k} w_{kj}^{l+1} \delta_k^{l+1} \sigma'(z_j^l) $$
权重梯度：
$$ \frac{\partial E}{\partial w_{ji}^l} = a_i^{l-1} \delta_j^l $$
$$ \frac{\partial E}{\partial b_j^l} = \delta_j^l $$

Sigmoid：
$$ \sigma(z) = \frac{1}{1+e^{-z}} $$
优点：输出在 (0,1) 之间，适合概率输出
缺点：容易导致梯度消失
ReLU：
$$ f(z) = max(0,z) $$
优点：计算简单，缓解梯度消失
缺点：可能导致神经元 ” 死亡 ”
Tanh：
$$ tanh(z) = \frac{e^z – e^{-z}}{e^z + e^{-z}} $$
优点：输出在 (-1,1) 之间，中心对称
缺点：同样存在梯度消失问题

import numpy as np

class NeuralNetwork:
    def __init__(self, layers, activation='sigmoid'):
        # 初始化权重
        self.weights = []
        self.biases = []
        for i in range(len(layers)-1):
            w = np.random.randn(layers[i+1], layers[i]) * 0.1
            b = np.zeros((layers[i+1], 1))
            self.weights.append(w)
            self.biases.append(b)

        # 设置激活函数
        if activation == 'sigmoid':
            self.activation = self._sigmoid
            self.activation_deriv = self._sigmoid_deriv
        elif activation == 'relu':
            self.activation = self._relu
            self.activation_deriv = self._relu_deriv

    def _sigmoid(self, x):
        return 1/(1+np.exp(-x))

    def _sigmoid_deriv(self, x):
        return self._sigmoid(x)*(1-self._sigmoid(x))

    def _relu(self, x):
        return np.maximum(0, x)

    def _relu_deriv(self, x):
        return (x > 0).astype(float)

    def forward(self, x):
        """前向传播"""
        self.activations = [x]
        self.zs = []

        for w, b in zip(self.weights, self.biases):
            z = np.dot(w, self.activations[-1]) + b
            a = self.activation(z)
            self.zs.append(z)
            self.activations.append(a)

        return self.activations[-1]

    def backward(self, y, learning_rate):
        """反向传播"""
        # 计算输出层误差
        delta = (self.activations[-1] - y) * self.activation_deriv(self.zs[-1])

        # 更新输出层权重
        m = self.activations[-2].shape[1]
        self.weights[-1] -= learning_rate * np.dot(delta, self.activations[-2].T) / m
        self.biases[-1] -= learning_rate * np.mean(delta, axis=1, keepdims=True)

        # 反向传播误差
        for l in range(len(self.weights)-2, -1, -1):
            delta = np.dot(self.weights[l+1].T, delta) * self.activation_deriv(self.zs[l])

            # 更新权重
            self.weights[l] -= learning_rate * np.dot(delta, self.activations[l].T) / m
            self.biases[l] -= learning_rate * np.mean(delta, axis=1, keepdims=True)

    def train(self, X, y, epochs=1000, learning_rate=0.1, batch_size=32):
        """训练函数"""
        n_samples = X.shape[1]

        for epoch in range(epochs):
            # 小批量训练
            for i in range(0, n_samples, batch_size):
                X_batch = X[:, i:i+batch_size]
                y_batch = y[:, i:i+batch_size]

                # 前向传播
                self.forward(X_batch)
                # 反向传播
                self.backward(y_batch, learning_rate)

            # 每 100 轮打印损失
            if epoch % 100 == 0:
                pred = self.forward(X)
                loss = np.mean((pred - y)**2)
                print(f"Epoch {epoch}, Loss: {loss:.4f}")

固定学习率：简单但需要手动调整
学习率衰减：随着训练逐渐减小学习率
自适应方法：如 Adam、RMSprop 等

使用 ReLU 等激活函数
采用 Batch Normalization
使用残差连接

增加 Dropout 层
使用 L1/L2 正则化
早停法(Early Stopping)

from sklearn.datasets import load_digits
from sklearn.preprocessing import OneHotEncoder

# 加载数据
digits = load_digits()
X = digits.data.T / 16.0  # 归一化
y = OneHotEncoder(sparse=False).fit_transform(digits.target.reshape(-1,1)).T

# 创建网络
nn = NeuralNetwork(layers=[64, 32, 10], activation='sigmoid')

# 训练
nn.train(X, y, epochs=1000, learning_rate=0.1, batch_size=32)

# 测试
pred = np.argmax(nn.forward(X), axis=0)
accuracy = np.mean(pred == digits.target)
print(f"Accuracy: {accuracy*100:.2f}%")