BP神经网络优化实战：自适应学习率与动量因子的协同调参指南

1次阅读

共计 4250 个字符，预计需要花费 11 分钟才能阅读完成。

传统 BP 神经网络使用固定学习率时，经常会遇到两个典型问题：

学习率设得太小：收敛速度慢，训练过程像蜗牛爬坡
学习率设得太大：损失函数在最优值附近震荡，甚至发散

举个例子，假设我们用一个简单的二次函数模拟损失函数：

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-2, 2, 100)
y = x**2  # 模拟损失函数

# 固定学习率的影响演示
lr_list = [0.1, 0.5, 1.2]
for lr in lr_list:
    path = [1.8]  # 初始点
    for _ in range(20):
        grad = 2 * path[-1]  # 梯度计算
        path.append(path[-1] - lr * grad)
    plt.plot(path, label=f'lr={lr}')

plt.xlabel('Steps')
plt.ylabel('Parameter Value')
plt.legend()
plt.show()

运行这段代码会看到：lr=0.1 时收敛慢；lr=0.5 时稳定收敛；lr=1.2 时直接发散了。这就是我们需要自适应学习率和动量因子的根本原因。

AdaGrad 的核心思想是：对频繁出现的特征使用较小的学习率，对罕见特征使用较大的学习率。数学表达为：

$$
\theta_{t+1} = \theta_t – \frac{\eta}{\sqrt{G_t + \epsilon}} \odot g_t
$$

其中 $G_t$ 是历史梯度平方和，$\epsilon$ 防止除零。但 AdaGrad 有个缺陷：随着训练进行，学习率会单调下降直至接近 0。

RMSProp 改进方案：

$$
E[g^2]t = \gamma E[g^2] + (1-\gamma)g_t^2
$$

$$
\theta_{t+1} = \theta_t – \frac{\eta}{\sqrt{E[g^2]_t + \epsilon}} g_t
$$

通过引入衰减系数 $\gamma$，解决了学习率过早衰减的问题。

动量因子的物理意义类似于小球滚下山坡时的惯性：

$$
v_t = \beta v_{t-1} + (1-\beta)g_t
$$

$$
\theta_{t+1} = \theta_t – \eta v_t
$$

其中 $\beta$ 通常取 0.9。这相当于对梯度做指数加权平均，能有效平滑更新方向。

Adam 结合了两种策略的优点：

计算梯度的一阶矩估计（动量）
计算梯度的二阶矩估计（自适应学习率）
进行偏差修正

更新公式：

$$
m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t
$$

$$
v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2
$$

$$
\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}
$$

$$
\theta_{t+1} = \theta_t – \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t
$$

import numpy as np
from tensorflow.keras.datasets import mnist
from sklearn.preprocessing import OneHotEncoder

# 数据准备
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(-1, 784) / 255.0
X_test = X_test.reshape(-1, 784) / 255.0
encoder = OneHotEncoder(sparse=False)
y_train = encoder.fit_transform(y_train.reshape(-1, 1))
y_test = encoder.transform(y_test.reshape(-1, 1))

# 网络参数
input_dim = 784
hidden_dim = 256
output_dim = 10

# Adam 优化器实现
class AdamOptimizer:
    def __init__(self, lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8):
        self.lr = lr
        self.beta1 = beta1
        self.beta2 = beta2
        self.eps = eps
        self.m = None
        self.v = None
        self.t = 0

    def update(self, params, grads):
        if self.m is None:
            self.m = [np.zeros_like(p) for p in params]
            self.v = [np.zeros_like(p) for p in params]

        self.t += 1
        updated_params = []
        for i, (param, grad) in enumerate(zip(params, grads)):
            # 更新一阶矩估计
            self.m[i] = self.beta1 * self.m[i] + (1 - self.beta1) * grad
            # 更新二阶矩估计
            self.v[i] = self.beta2 * self.v[i] + (1 - self.beta2) * (grad ** 2)
            # 偏差修正
            m_hat = self.m[i] / (1 - self.beta1 ** self.t)
            v_hat = self.v[i] / (1 - self.beta2 ** self.t)
            # 参数更新
            updated_params.append(param - self.lr * m_hat / (np.sqrt(v_hat) + self.eps))
        return updated_params

# 3 层神经网络实现
class NeuralNetwork:
    def __init__(self):
        self.W1 = np.random.randn(input_dim, hidden_dim) * 0.01
        self.b1 = np.zeros(hidden_dim)
        self.W2 = np.random.randn(hidden_dim, output_dim) * 0.01
        self.b2 = np.zeros(output_dim)

    def forward(self, X):
        self.z1 = np.dot(X, self.W1) + self.b1
        self.a1 = np.maximum(0, self.z1)  # ReLU
        self.z2 = np.dot(self.a1, self.W2) + self.b2
        exp_z = np.exp(self.z2 - np.max(self.z2, axis=1, keepdims=True))
        self.probs = exp_z / np.sum(exp_z, axis=1, keepdims=True)
        return self.probs

    def backward(self, X, y, l2_reg=0.01):
        batch_size = X.shape[0]

        # 输出层梯度
        dz2 = self.probs - y
        dW2 = np.dot(self.a1.T, dz2) / batch_size
        db2 = np.sum(dz2, axis=0) / batch_size

        # 隐藏层梯度
        da1 = np.dot(dz2, self.W2.T)
        dz1 = da1 * (self.z1 > 0)  # ReLU 导数
        dW1 = np.dot(X.T, dz1) / batch_size
        db1 = np.sum(dz1, axis=0) / batch_size

        # L2 正则化
        dW2 += l2_reg * self.W2
        dW1 += l2_reg * self.W1

        return [dW1, db1, dW2, db2]

# 训练过程
nn = NeuralNetwork()
optimizer = AdamOptimizer(lr=0.001)
batch_size = 128
epochs = 20

for epoch in range(epochs):
    # 学习率衰减
    if epoch == 10:
        optimizer.lr *= 0.1

    for i in range(0, len(X_train), batch_size):
        X_batch = X_train[i:i+batch_size]
        y_batch = y_train[i:i+batch_size]

        # 前向传播
        probs = nn.forward(X_batch)

        # 计算损失
        loss = -np.mean(y_batch * np.log(probs + 1e-15))

        # 反向传播
        grads = nn.backward(X_batch, y_batch)

        # 参数更新
        params = [nn.W1, nn.b1, nn.W2, nn.b2]
        updated_params = optimizer.update(params, grads)
        nn.W1, nn.b1, nn.W2, nn.b2 = updated_params

    # 验证集评估
    test_probs = nn.forward(X_test)
    test_preds = np.argmax(test_probs, axis=1)
    test_acc = np.mean(test_preds == np.argmax(y_test, axis=1))
    print(f'Epoch {epoch+1}, Loss: {loss:.4f}, Test Acc: {test_acc:.4f}')