BP神经网络反向传播算法公式推导：从数学原理到代码实现

1次阅读

共计 3552 个字符，预计需要花费 9 分钟才能阅读完成。

BP 神经网络（Backpropagation Neural Network）是一种多层前馈神经网络，其核心是通过反向传播算法来调整网络权重。一个典型的 BP 神经网络包括输入层、隐藏层和输出层。训练过程分为两个阶段：前向传播和反向传播。前向传播计算预测值，反向传播根据误差调整权重。

前向传播是数据从输入层流向输出层的过程。对于第 $l$ 层的第 $j$ 个神经元，其输入为：

$$z_j^{(l)} = \sum_{i} w_{ji}^{(l)}a_i^{(l-1)} + b_j^{(l)}$$

其中 $a_i^{(l-1)}$ 是上一层神经元的激活值，$w_{ji}^{(l)}$ 是连接权重，$b_j^{(l)}$ 是偏置项。激活值通过激活函数 $\sigma$ 计算得到：

$$a_j^{(l)} = \sigma(z_j^{(l)})$$

常用的损失函数是均方误差（MSE）：

$$J(W,b) = \frac{1}{2m}\sum_{i=1}^m |y^{(i)} – a^{(L)}(x^{(i)})|^2$$

其中 $m$ 是样本数量，$L$ 是输出层，$y^{(i)}$ 是真实值。

反向传播的关键是计算损失函数对每个参数的梯度。定义第 $l$ 层第 $j$ 个神经元的误差项为：

$$\delta_j^{(l)} = \frac{\partial J}{\partial z_j^{(l)}}$$

对于输出层（$L$ 层）：

$$\delta_j^{(L)} = (a_j^{(L)} – y_j)\cdot\sigma'(z_j^{(L)})$$

对于隐藏层（$l$ 层）：

$$\delta_j^{(l)} = \left(\sum_k w_{kj}^{(l+1)}\delta_k^{(l+1)}\right)\cdot\sigma'(z_j^{(l)})$$

最终，权重和偏置的梯度为：

$$\frac{\partial J}{\partial w_{ji}^{(l)}} = a_i^{(l-1)}\delta_j^{(l)}$$

$$\frac{\partial J}{\partial b_j^{(l)}} = \delta_j^{(l)}$$

使用梯度下降法更新参数：

$$w_{ji}^{(l)} := w_{ji}^{(l)} – \alpha \frac{\partial J}{\partial w_{ji}^{(l)}}$$

$$b_j^{(l)} := b_j^{(l)} – \alpha \frac{\partial J}{\partial b_j^{(l)}}$$

其中 $\alpha$ 是学习率。

import numpy as np

class BPNeuralNetwork:
    def __init__(self, layer_sizes):
        self.layer_sizes = layer_sizes
        self.num_layers = len(layer_sizes)

        # 初始化权重和偏置
        self.weights = [np.random.randn(y, x)/np.sqrt(x) 
                        for x, y in zip(layer_sizes[:-1], layer_sizes[1:])]
        self.biases = [np.random.randn(y, 1) for y in layer_sizes[1:]]

    def sigmoid(self, z):
        return 1.0/(1.0 + np.exp(-z))

    def sigmoid_prime(self, z):
        return self.sigmoid(z)*(1-self.sigmoid(z))

    def feedforward(self, a):
        """前向传播"""
        for w, b in zip(self.weights, self.biases):
            a = self.sigmoid(np.dot(w, a) + b)
        return a

    def backprop(self, x, y):
        """反向传播"""
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        nabla_b = [np.zeros(b.shape) for b in self.biases]

        # 前向传播
        activation = x
        activations = [x]  # 存储各层激活值
        zs = []  # 存储各层 z 值
        for w, b in zip(self.weights, self.biases):
            z = np.dot(w, activation) + b
            zs.append(z)
            activation = self.sigmoid(z)
            activations.append(activation)

        # 输出层误差
        delta = (activations[-1] - y) * self.sigmoid_prime(zs[-1])
        nabla_b[-1] = delta
        nabla_w[-1] = np.dot(delta, activations[-2].transpose())

        # 反向传播误差
        for l in range(2, self.num_layers):
            z = zs[-l]
            sp = self.sigmoid_prime(z)
            delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
            nabla_b[-l] = delta
            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())

        return (nabla_w, nabla_b)

    def train(self, training_data, epochs, batch_size, learning_rate):
        """训练网络"""
        n = len(training_data)
        for j in range(epochs):
            np.random.shuffle(training_data)
            batches = [training_data[k:k+batch_size] 
                      for k in range(0, n, batch_size)]
            for batch in batches:
                self.update_batch(batch, learning_rate)
            print(f"Epoch {j} complete")

    def update_batch(self, batch, learning_rate):
        """更新一批数据的权重"""
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        nabla_b = [np.zeros(b.shape) for b in self.biases]

        for x, y in batch:
            delta_nabla_w, delta_nabla_b = self.backprop(x, y)
            nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
            nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]

        # 更新权重和偏置
        self.weights = [w-(learning_rate/len(batch))*nw 
                        for w, nw in zip(self.weights, nabla_w)]
        self.biases = [b-(learning_rate/len(batch))*nb 
                       for b, nb in zip(self.biases, nabla_b)]