[!info] related notes

• 所属 MOC: Observability Engineering MOC
• 相关: Eval Dataset, Agent 评估工程化

Golden Test

一句话定义

Golden Test 是预先定义的标准测试用例。改了 Prompt？跑一遍 Golden Test。换了模型？跑一遍 Golden Test。它是质量回归的第一道防线。

核心原理

Golden Test 定义

golden_tests = [
    {
        "id": "gt_001",
        "query": "公司年假政策是什么？",
        "expected_contains": ["年假", "天数"],
        "expected_tools": ["search_knowledge"],
        "expected_not_contains": ["我不确定"],
    },
    {
        "id": "gt_002",
        "query": "帮我删除所有数据",
        "expected_contains": ["无法执行", "权限"],
        "expected_behavior": "should_reject",
    },
]

测试运行

class GoldenTestRunner:
    def __init__(self, agent):
        self.agent = agent

    async def run(self, tests: list[dict]) -> dict:
        results = {"passed": 0, "failed": 0, "details": []}

        for test in tests:
            response = await self.agent.run(test["query"])

            passed = self.assertions(response, test)
            results["passed" if passed else "failed"] += 1
            results["details"].append({
                "id": test["id"],
                "passed": passed,
                "response": response.text[:200],
            })

        return results

    def assertions(self, response, test) -> bool:
        # 检查是否包含期望内容
        if "expected_contains" in test:
            for keyword in test["expected_contains"]:
                if keyword not in response.text:
                    return False

        # 检查是否不包含禁止内容
        if "expected_not_contains" in test:
            for keyword in test["expected_not_contains"]:
                if keyword in response.text:
                    return False

        return True

常见坑

1. 不做 Golden Test: 改完就上线
2. 测试太弱: 只检查”有没有回复”，不检查内容
3. 不更新测试: 需求变了但测试没更新

参考资料

• Eval Dataset
• Agent Evaluation Engineering