深度解析残差网络：如何通过跳跃连接解决梯度消失问题

1次阅读

共计 2460 个字符，预计需要花费 7 分钟才能阅读完成。

当神经网络层数加深时，使用 sigmoid 激活函数会导致梯度在反向传播时呈指数级衰减，这是因为 sigmoid 的导数最大值为 0.25。根据链式法则，多层梯度连乘后数值可能趋近于零，使得浅层参数无法有效更新。这种现象在超过 15 层的传统 CNN 中尤为明显，严重限制了模型的深度和性能。

传统 CNN 的梯度流动可以表示为：
$$\frac{\partial L}{\partial x_l} = \frac{\partial L}{\partial x_{l+1}} \cdot \frac{\partial F(x_l, W_l)}{\partial x_l}$$

而 ResNet 引入跳跃连接后：
$$\frac{\partial L}{\partial x_l} = \frac{\partial L}{\partial x_{l+1}} \cdot (1 + \frac{\partial F(x_l, W_l)}{\partial x_l})$$

这个 ”1+” 项保证了梯度至少能以恒等路径回传，从根本上避免了梯度消失。实验表明，在 34 层网络上，ResNet 的梯度幅值比传统 CNN 高 2 - 3 个数量级。

BasicBlock（适合浅层网络）：

class BasicBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm2d(out_channels)

        # 处理维度变化的跳跃连接
        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride),
                nn.BatchNorm2d(out_channels)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)  # 关键跳跃连接
        return F.relu(out)

Bottleneck（适合深层网络）：

class Bottleneck(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1, expansion=4):
        super().__init__()
        mid_channels = out_channels // expansion
        self.conv1 = nn.Conv2d(in_channels, mid_channels, kernel_size=1)
        self.bn1 = nn.BatchNorm2d(mid_channels)
        self.conv2 = nn.Conv2d(mid_channels, mid_channels, kernel_size=3, stride=stride, padding=1)
        self.bn2 = nn.BatchNorm2d(mid_channels)
        self.conv3 = nn.Conv2d(mid_channels, out_channels, kernel_size=1)
        self.bn3 = nn.BatchNorm2d(out_channels)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride),
                nn.BatchNorm2d(out_channels)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = F.relu(self.bn2(self.conv2(out)))
        out = self.bn3(self.conv3(out))
        out += self.shortcut(x)
        return F.relu(out)

在 CIFAR-10 数据集上的实验表明：