深入解析CBOW词嵌入：从原理到工程实践

1次阅读

共计 3221 个字符，预计需要花费 9 分钟才能阅读完成。

词嵌入技术是自然语言处理（NLP）的基础，而 CBOW（Continuous Bag of Words）模型作为经典词嵌入方法之一，在实际应用中开发者常遇到以下问题：

窗口大小选择困难：窗口过大导致训练效率低下，过小则丢失上下文信息
低频词处理不佳：低频词因样本不足导致嵌入质量差
训练速度瓶颈：传统 softmax 计算在词汇量大时效率极低
词向量不稳定：相同参数多次训练结果差异显著

CBOW 特性：
用上下文预测中心词
训练速度快（尤其适合高频词）
对小数据集更友好
Skip-gram 特性：
用中心词预测上下文
对低频词表现更好
需要更多训练数据
选择建议：
语料规模较小时优先 CBOW
需要捕获细粒度语义时考虑 Skip-gram

前向传播：
$$
\mathbf{h} = \frac{1}{C}\sum_{c=1}^C \mathbf{W}{input}^T \mathbf{x}_c
$$
$$
\mathbf{u}_o = \mathbf{W}
$$
$$
P(w_o|w_{c1},…,w_{cC}) = \frac{\exp(u_{o})}{\sum_{i=1}^V \exp(u_i)}
$$} \mathbf{h

反向传播（交叉熵损失）：
$$
\frac{\partial L}{\partial \mathbf{W}{output}} = (\hat{y} – y)\mathbf{h}^T
$$
$$
\frac{\partial L}{\partial \mathbf{W}_c^T
$$}} = \sum_{c=1}^C \mathbf{W}_{output}^T(\hat{y} – y)\mathbf{x

import torch
import torch.nn as nn
import torch.optim as optim

class CBOW(nn.Module):
    def __init__(self, vocab_size, embedding_dim):
        super().__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear = nn.Linear(embedding_dim, vocab_size)

    def forward(self, inputs):
        # inputs 形状: (batch_size, window_size*2)
        embedded = self.embeddings(inputs)  # (batch_size, window_size*2, emb_dim)
        averaged = torch.mean(embedded, dim=1)  # (batch_size, emb_dim)
        outputs = self.linear(averaged)  # (batch_size, vocab_size)
        return outputs

# 示例训练循环
def train_model(corpus, window_size=2, emb_dim=100, epochs=5):
    # 数据预处理（需实现 build_vocab 和 create_dataset）vocab, word_to_idx = build_vocab(corpus)
    train_data = create_dataset(corpus, word_to_idx, window_size)

    model = CBOW(len(vocab), emb_dim)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters())

    for epoch in range(epochs):
        total_loss = 0
        for context, target in train_data:
            optimizer.zero_grad()
            log_probs = model(context)
            loss = criterion(log_probs, target)
            loss.backward()
            optimizer.step()
            total_loss += loss.item()
        print(f"Epoch {epoch}, Loss: {total_loss}")

实现原理：
将多分类问题转化为二分类
对每个正样本采样 k 个负样本

PyTorch 示例：

class NegativeSamplingLoss(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, input_vectors, output_vectors, noise_vectors):
        batch_size, emb_dim = input_vectors.shape
        input_vectors = input_vectors.view(batch_size, emb_dim, 1)

        # 正样本得分
        pos_score = torch.bmm(output_vectors, input_vectors).sigmoid().log()

        # 负样本得分
        neg_score = torch.bmm(noise_vectors.neg(), input_vectors).sigmoid().log()

        return -(pos_score + neg_score).mean()

线性预热：前 10% 训练步从 0 线性增加到初始学习率
余弦退火：后续训练步按余弦曲线下降

相似度计算：

from sklearn.metrics.pairwise import cosine_similarity

def most_similar(word, word_to_idx, embeddings, topn=5):
    vec = embeddings[word_to_idx[word]]
    sims = cosine_similarity([vec], embeddings)[0]
    return np.argsort(sims)[-topn-1:-1][::-1]

可视化工具：

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

def visualize(embeddings, words):
    tsne = TSNE(n_components=2)
    reduced = tsne.fit_transform(embeddings)
    plt.scatter(reduced[:,0], reduced[:,1])
    for i, word in enumerate(words):
        plt.annotate(word, xy=(reduced[i,0], reduced[i,1]))
    plt.show()