AI大模型运维入门指南：从零搭建到生产环境避坑

1次阅读

共计 2262 个字符，预计需要花费 6 分钟才能阅读完成。

显存管理成为核心瓶颈 ：传统应用关注 CPU 和内存，而大模型运维需要精细管理 GPU 显存，防止 OOM（Out Of Memory）错误，例如通过torch.cuda.empty_cache() 主动清理碎片
长时任务成为常态：单次推理可能持续分钟级，需要设计心跳检测和任务超时机制，避免僵尸进程占用资源
模型文件巨大：单个模型权重文件常达数百 GB，传统文件传输方式失效，需结合分块校验和断点续传（如使用rsync --partial）

指标	A100 80GB	H100 80GB	适用场景
FP32 算力	19.5 TFLOPS	30 TFLOPS	训练任务首选
显存带宽	2039 GB/s	3000 GB/s	大 batch 推理场景
NVLink 速度	600 GB/s	900 GB/s	多卡并行训练
每日租赁价	$3.5-$4.5	$8-$10	预算敏感型项目

# 基础镜像选择建议
FROM nvcr.io/nvidia/pytorch:23.05-py3

# 必须安装的组件
RUN apt-get update && apt-get install -y \
    cuda-toolkit-11-7 \
    nvidia-container-toolkit \
    && rm -rf /var/lib/apt/lists/*

# 关键环境变量配置
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
ENV CUDA_VISIBLE_DEVICES=0

import mlflow.pyfunc

class WrapperModel(mlflow.pyfunc.PythonModel):
    def load_context(self, context):
        # 带错误重试的模型加载
        for _ in range(3):  # 最大重试次数
            try:
                self.model = torch.load(context.artifacts["model_path"], 
                                      map_location='cuda:0')
                break
            except RuntimeError as e:
                if "CUDA out of memory" in str(e):
                    torch.cuda.empty_cache()
                    continue
                raise

    def predict(self, context, model_input):
        with torch.no_grad():
            return self.model(model_input)

# 记录模型版本
mlflow.pyfunc.log_model(
    artifact_path="gpt3",
    python_model=WrapperModel(),
    artifacts={"model_path": "/models/gpt3-v1.pt"},
    registered_model_name="GPT-3"
)

from concurrent.futures import ThreadPoolExecutor
import numpy as np

class DynamicBatcher:
    def __init__(self, max_batch_size=8, timeout=0.1):
        self.executor = ThreadPoolExecutor(max_workers=4)
        self.batch_buffer = []
        self.timeout = timeout  # 等待新请求的最大时间(秒)

    def process_request(self, requests):
        # 根据输入长度自动调整 batch
        sorted_requests = sorted(requests, key=lambda x: len(x["input"]))
        batches = np.array_split(
            sorted_requests, 
            max(1, len(requests)//self.max_batch_size)
        )
        return [self.executor.submit(self._inference, b) for b in batches]

{
  "panels": [{
    "title": "GPU 显存监控",
    "type": "graph",
    "targets": [{"expr": "sum(container_memory_usage_bytes{device=~'gpu.*'}) by (pod)",
      "legendFormat": "{{pod}}显存占用"
    }],
    "thresholds": [{
      "colorMode": "critical",
      "fill": true,
      "line": true,
      "op": "gt",
      "value": 0.9
    }]
  }]
}