RAG 发布门禁生产实战：Release Manifest、灰度、回滚与审计闭环

代码合并
→ CI 构建通过
→ 部署新容器
→ 发布完成

Change Request
→ Risk Classification
→ Candidate Build
→ Immutable Release Manifest
→ Offline Paired Evaluation
→ Hard / Soft Gate
→ Shadow
→ Canary 1% / 5% / 10%
→ Full Rollout
→ Online Evaluation
→ Incident / Bad Case
→ Regression Dataset

PASS
→ 可以进入下一阶段

CONDITIONAL PASS
→ 带明确风险、Owner 和观察条件进入灰度

PAUSE
→ 停止扩大流量，继续观察或补充样本

BLOCK / ROLLBACK
→ 不允许发布，或立即切回稳定版本

Build
Test
Deploy
Health Check
Rollback Binary

大概是下午发布的；
好像换过 Reranker；
Prompt 应该还是 v7；
索引可能已经切到 v4；
Embedding 用的是哪个 Endpoint 不确定。

{
  "release_id": "rag-prod-2026-08-05.3",
  "created_at": "2026-08-05T18:30:00+08:00",
  "owner": "rag-platform",
  "change_ticket": "RAG-1842",
  "risk_level": "high",
  "application": {
    "repository": "chicogong/rag-service",
    "commit": "abc123def456",
    "image": "registry.example.com/rag-service:abc123"
  },
  "query": {
    "classifier_version": "query-classifier-v5",
    "planner_prompt_version": "query-plan-v8",
    "rewrite_prompt_version": "rewrite-v11",
    "scope_compiler_version": "scope-v4"
  },
  "retrieval": {
    "index_alias": "kb_chunks",
    "physical_indices": ["kb_chunks_v4"],
    "projection_schema_version": 6,
    "embedding_model_key": "bge-m3|endpoint-a|rev-20260801",
    "bm25_config_version": "bm25-v9",
    "rrf_config_version": "rrf-v7",
    "top_k": {
      "bm25": 80,
      "dense": 80,
      "fusion": 100
    }
  },
  "rerank": {
    "model_key": "bge-reranker-v2-m3|gpu-pool-a|rev-3",
    "config_version": "rerank-v10",
    "window": 50,
    "output_count": 8
  },
  "context": {
    "packer_version": "context-v7",
    "token_budget": 6000,
    "max_chunks_per_document": 2
  },
  "generation": {
    "prompt_version": "answer-v16",
    "model_route": "answer-stable-202608",
    "guardrail_version": "guard-v5",
    "citation_mapper_version": "citation-v4"
  },
  "evaluation": {
    "datasets": [
      "rag-eval-core@2026-08-05.1",
      "rag-eval-regression@2026-08-05.7",
      "rag-eval-safety@2026-08-05.3"
    ],
    "judge_version": "judge-v11",
    "metric_schema_version": "rag-metrics-v6",
    "baseline_release_id": "rag-prod-2026-07-29.2",
    "report_uri": "artifact://rag-release/rag-prod-2026-08-05.3/report.json"
  },
  "rollback": {
    "target_release_id": "rag-prod-2026-07-29.2",
    "runbook": "runbook://rag/release-rollback-v4",
    "verified_at": "2026-08-05T17:30:00+08:00"
  }
}

answer_temperature: 0.1 → 0.0

acl_filter_mode: pre_filter → post_filter

为什么改？
解决哪个 Bad Case 或业务目标？
改动了哪些资产？
预计影响哪些 Query Cohort？
可能退化什么？
如何观察？
如何回滚？
谁是 Owner？

change_request:
  id: RAG-1842
  title: 增加 Reranker Window 以改善多证据问题
  owner: retrieval-team
  risk_level: high

  hypothesis:
    problem: 多证据问题正确 Chunk 已进入 RRF Top-50，但未进入 Rerank Window
    expected_gain: multi_evidence_context_recall 提升
    acceptable_cost: p95 增加不超过 80ms

  affected_assets:
    - reranker_config
    - context_packer

  target_cohorts:
    - multi_evidence
    - policy_comparison

  known_risks:
    - GPU queue time 增加
    - 短 Query 可能无收益
    - 长候选文本导致截断比例上升

  rollback:
    target_release_id: rag-prod-2026-07-29.2
    estimated_seconds: 30

release-manifest.json
change-request.yaml
offline-eval-report.json
offline-eval-report.md
case-level-diff.jsonl
safety-report.json
latency-cost-report.json
index-manifest.json
prompt-bundle/
retrieval-config.json
reranker-config.json
rollback-plan.yaml
approval-record.json

Baseline
Candidate

Improved
Regressed
Unchanged
New Failure
Fixed Bad Case
Judge Inconsistent

高频 FAQ +5%
Critical 权限问题出现 1 次越权

{
  "case_id": "rag_eval_0042",
  "severity": "high",
  "cohort": "multi_evidence",
  "baseline": {
    "retrieval_recall_at_20": 0.5,
    "faithfulness": 1.0,
    "latency_ms": 910
  },
  "candidate": {
    "retrieval_recall_at_20": 1.0,
    "faithfulness": 1.0,
    "latency_ms": 1040
  },
  "classification": "improved",
  "root_delta": "rerank window 30 → 50"
}

Unauthorized Retrieval Rate = 0
Unauthorized Context Rate = 0
Unauthorized Citation Rate = 0
Critical Safety Pass Rate = 100%
Invalid Citation ID Rate = 0
Trace Contract Pass Rate = 100%
Schema Validation Pass Rate = 100%
High Severity Regression Count = 0
P99 < Hard Deadline
Rollback Verification = passed

不允许扩大 Scope
不允许跳过 ACL
不允许用缓存中的旧权限继续回答

Exact ID
Error Code
Concept
Multi-evidence
Multi-hop
Temporal
Permission-sensitive
Long Context
No Evidence

release_gate_version: rag-gate-v5

release:
  id: rag-prod-2026-08-05.3
  baseline_release_id: rag-prod-2026-07-29.2
  risk_level: high
  owner: rag-platform

required_artifacts:
  - release-manifest.json
  - change-request.yaml
  - evaluation/summary.json
  - evaluation/case-level-diff.jsonl
  - evaluation/safety-report.json
  - runtime/index-manifest.json
  - rollback/rollback-plan.yaml
  - rollback/rollback-verification.json

datasets:
  - rag-eval-core@2026-08-05.1
  - rag-eval-regression@2026-08-05.7
  - rag-eval-safety@2026-08-05.3

hard_gates:
  unauthorized_retrieval_rate:
    max: 0
  unauthorized_context_rate:
    max: 0
  unauthorized_citation_rate:
    max: 0
  critical_case_pass_rate:
    min: 1.0
  high_severity_regression_count:
    max: 0
  invalid_citation_rate:
    max: 0
  trace_contract_pass_rate:
    min: 1.0
  schema_validation_pass_rate:
    min: 1.0
  rollback_verification:
    equals: passed
  p99_latency_ms:
    max: 5000

soft_gates:
  evidence_recall_at_10:
    max_drop: 0.005
  ndcg_at_10:
    max_drop: 0.01
  context_precision:
    max_drop: 0.01
  faithfulness:
    max_drop: 0.005
  citation_accuracy:
    max_drop: 0.005
  p95_latency_ms:
    max_increase_ratio: 0.15
  cost_per_request:
    max_increase_ratio: 0.10

cohort_gates:
  permission_sensitive:
    pass_rate:
      min: 1.0
  exact_id:
    hit_at_1:
      max_drop: 0
  multi_hop:
    complete_chain_rate:
      max_drop: 0.02

waiver_policy:
  allowed_for:
    - soft_gate
  forbidden_for:
    - unauthorized_access
    - critical_safety
    - invalid_schema
  required_approvers: 2
  max_valid_hours: 24

Faithfulness 不变
NDCG +4%
P95 +12%
成本 +11%

waiver:
  release_id: rag-prod-2026-08-05.3
  gate: cost_per_request
  observed: 0.021
  threshold: 0.020
  reason: 多跳问题完整证据链提升 8.4%
  scope: multi_hop_queries_only
  expires_at: 2026-08-06T18:00:00+08:00
  owner: rag-quality
  approvers:
    - retrieval-lead
    - product-owner
  online_guardrail:
    cost_per_request_max: 0.024
    auto_pause: true

Shadow
→ 1%
→ 5%
→ 10%
→ 25%
→ 50%
→ 100%

最小样本量
最短观察窗口
推进条件
暂停条件
回滚条件
Owner

canary:
  stages:
    - name: shadow
      traffic_percent: 0
      min_requests: 10000
      min_duration_minutes: 30

    - name: canary_1
      traffic_percent: 1
      min_requests: 2000
      min_duration_minutes: 30

    - name: canary_10
      traffic_percent: 10
      min_requests: 10000
      min_duration_minutes: 60

    - name: canary_50
      traffic_percent: 50
      min_requests: 50000
      min_duration_minutes: 120

    - name: rollout
      traffic_percent: 100

  advance_when:
    hard_gates_passed: true
    high_severity_regressions: 0
    p95_latency_ratio_max: 1.15
    cost_ratio_max: 1.10

  pause_when:
    sample_size_insufficient: true
    judge_disagreement_rate_max: 0.08
    soft_gate_warning_count_max: 2

  rollback_when:
    unauthorized_access_count_min: 1
    critical_failure_count_min: 1
    p99_latency_ms_min: 5000
    error_rate_ratio_min: 2.0

发布 15 分钟没报警
→ 自动从 1% 升到 50%

Duration
+
Request Count
+
Cohort Coverage
+
Hard Gate
+
Soft Gate

Empty Retrieval Rate
Evidence Recall Proxy
Citation Coverage
Invalid Citation Rate
Refusal Rate
Clarification Rate
User Negative Feedback
Human Escalation

Unauthorized Retrieval
Unauthorized Context
Unauthorized Citation
Prompt Injection Detection
Sensitive Data Exposure
Cross-tenant Cache Hit

P50 / P95 / P99
Timeout Rate
Queue Time
Embedding Latency
Reranker Latency
LLM Time to First Token
Fallback Rate

Embedding Cost
Rerank Pair Count
Rerank GPU Time
LLM Input / Output Tokens
Cache Read / Write
Artifact Storage
Cost per Successful Answer

Tenant
Language
Query Cohort
Knowledge Base
Data Source
Model Route
Region
Document Type

旧索引还存在吗？
旧 Prompt 能读取吗？
旧模型 Endpoint 还在线吗？
Alias 切换权限可用吗？
Feature Flag 能立即生效吗？
缓存是否带 Release ID？

{
  "rollback_target": "rag-prod-2026-07-29.2",
  "verified": true,
  "verified_at": "2026-08-05T17:30:00+08:00",
  "steps": {
    "prompt_switch": "passed",
    "index_alias": "passed",
    "model_route": "passed",
    "feature_flag": "passed"
  },
  "estimated_seconds": 28
}

kb_chunks

kb_chunks_v3
kb_chunks_v4

POST _aliases
{
  "actions": [
    {
      "remove": {
        "index": "kb_chunks_v3",
        "alias": "kb_chunks"
      }
    },
    {
      "add": {
        "index": "kb_chunks_v4",
        "alias": "kb_chunks",
        "is_write_index": true
      }
    }
  ]
}

Canary 观察窗口
+
回归发现延迟
+
Incident 分析周期
+
安全缓冲

旧模型 + v3 索引
新模型 + v4 索引

同一个 dense_vector 字段混写不同模型生成的向量

{
  "prompt_name": "rag-answer",
  "active_version": "v16",
  "stable_version": "v15",
  "versions": {
    "v15": {
      "sha256": "...",
      "created_at": "2026-07-29T10:00:00Z"
    },
    "v16": {
      "sha256": "...",
      "created_at": "2026-08-05T10:00:00Z"
    }
  }
}

routes:
  answer-stable:
    primary: model-x-rev-3
    fallback: model-y-rev-2

  answer-candidate:
    primary: model-x-rev-4
    fallback: model-x-rev-3

cache_key =
  tenant_id
  + normalized_query
  + scope_hash
  + release_id

query_rewrite_enabled
multi_query_enabled
reranker_enabled
new_index_enabled
new_prompt_enabled
new_model_route_enabled
web_search_enabled
agentic_retrieval_enabled

{
  "release_id": "rag-prod-2026-08-05.3",
  "feature_flags": {
    "query_rewrite_enabled": true,
    "reranker_enabled": true,
    "new_index_enabled": true
  }
}

release_id
baseline_release_id
canary_stage
experiment_cohort
application_commit
index_manifest_version
prompt_version
model_route
reranker_version

Baseline vs Candidate
Stable vs Canary
Release A vs Release B

stable
candidate
canary_stage

Gate Evaluation Success Rate
Candidate Build Success Rate
Canary Auto-Pause Latency
Rollback Completion Time
Release Artifact Completeness

Trace Export Success Rate
Release ID Coverage
Candidate / Baseline Correlation Rate
Metric Freshness
Artifact Availability

release-gate
├── validate change request
├── build candidate manifest
├── validate required artifacts
├── run baseline
├── run candidate
├── compute deterministic metrics
├── run calibrated judges
├── compare paired cases
├── check hard gates
├── check soft gates
├── verify rollback
├── generate report
└── create deployment approval

name: rag-release-gate

on:
  pull_request:
    paths:
      - "prompts/**"
      - "retrieval/**"
      - "rag/**"
      - "indexes/**"
      - "release/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r evaluation/requirements.lock

      - name: Validate release manifest
        run: python -m release_gate validate release/release-manifest.json

      - name: Run paired evaluation
        run: |
          python -m evaluation.run \
            --baseline release/baseline-manifest.json \
            --candidate release/release-manifest.json \
            --datasets release/datasets.yaml \
            --output artifacts/eval

      - name: Evaluate gates
        run: |
          python -m release_gate check \
            --config release/release-gate.yaml \
            --report artifacts/eval/summary.json \
            --output artifacts/release-decision.json

      - name: Upload evidence
        uses: actions/upload-artifact@v4
        with:
          name: rag-release-evidence
          path: artifacts/

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum
from typing import Any


class Decision(str, Enum):
    PASS = "pass"
    CONDITIONAL = "conditional"
    PAUSE = "pause"
    BLOCK = "block"


@dataclass(frozen=True)
class GateResult:
    decision: Decision
    hard_failures: list[str]
    soft_warnings: list[str]
    required_approvals: int


def evaluate_release(
    report: dict[str, Any],
    config: dict[str, Any],
) -> GateResult:
    hard_failures: list[str] = []
    soft_warnings: list[str] = []

    for metric, rule in config["hard_gates"].items():
        value = report["metrics"].get(metric)
        if value is None or not check_rule(value, rule):
            hard_failures.append(metric)

    if hard_failures:
        return GateResult(
            decision=Decision.BLOCK,
            hard_failures=hard_failures,
            soft_warnings=[],
            required_approvals=0,
        )

    for metric, rule in config["soft_gates"].items():
        value = report["deltas"].get(metric)
        if value is None or not check_rule(value, rule):
            soft_warnings.append(metric)

    if soft_warnings:
        return GateResult(
            decision=Decision.CONDITIONAL,
            hard_failures=[],
            soft_warnings=soft_warnings,
            required_approvals=config["waiver_policy"]["required_approvers"],
        )

    return GateResult(
        decision=Decision.PASS,
        hard_failures=[],
        soft_warnings=[],
        required_approvals=0,
    )

为什么值得发布？
最大的剩余风险是什么？
异常时谁负责？
多快能回滚？

{
  "release_id": "rag-prod-2026-08-05.3",
  "event_id": "evt_release_0092",
  "event_type": "canary_advanced",
  "from_state": "canary_1",
  "to_state": "canary_10",
  "timestamp": "2026-08-05T20:10:00+08:00",
  "actor": {
    "type": "service",
    "id": "release-controller"
  },
  "evidence": {
    "requests": 15240,
    "duration_minutes": 68,
    "hard_gates": "passed",
    "soft_warnings": [],
    "dashboard": "trace://release/rag-prod-2026-08-05.3"
  }
}

Candidate Trace Samples
Baseline / Candidate Pair
Relevant Artifacts
Gate State
Canary Stage
Metric Timeline
Feature Flag Changes
Rollback Timeline

Root Cause
Detection Gap
Gate Gap
Telemetry Gap
Rollback Gap
Action Items
New Regression Cases
New Safety Cases

中文长文档
英文 FAQ
某个部门 ACL
某个大知识库
特定地域模型端点

MySQL / Outbox Lag
Elasticsearch Cluster Health
Embedding Endpoint
Reranker GPU Pool
LLM Provider
Trace Collector
Feature Flag Service
Artifact Store

任何生产请求都能追溯到完整 Release Manifest；
任何 Candidate 都能在一分钟内切回已验证版本。

Change Request
→ 明确为什么改、影响什么、谁负责

Release Manifest
→ 冻结代码、索引、模型、Prompt 和配置

Paired Evaluation
→ 证明 Candidate 相对 Baseline 的变化

Hard / Soft Gate
→ 把安全零容忍与质量权衡分开

Shadow / Canary
→ 在真实分布中逐步获得证据

Pause / Rollback
→ 异常时停止风险扩散并切回稳定资产

Observability / Audit
→ 解释每次请求和每个发布决策

Incident / Bad Case
→ 把失败沉淀到下一次门禁