原文出处：Techniques to improve reliability 原作者：OpenAI · 许可证：MIT License 中文译本由诸葛AI学院整理，仅供学习参考，版权归原作者与 OpenAI 所有。

提升大模型可靠性的技巧（上）

GPT-3 把一项任务做砸了，你该怎么办？

去找一个能给出更可靠回答的提示词（prompt）？
投入几千条样本，微调（fine-tune）一个定制模型？
认定模型干不了这个任务，转身放弃？

这里没有简单的答案，要看情况。不过，如果你的任务涉及逻辑推理或者足够复杂，不妨试试本文介绍的几种技巧，搭出更可靠、性能更强的提示词。

为什么 GPT-3 会在复杂任务上失败

要是让你算 13 乘 17，答案会立刻蹦进你脑子吗？我们大多数人不会。但这不代表人类不会做两位数乘法。给你几秒钟、一支笔一张纸，算出 13 x 17 = 130 + 70 + 21 = 221 并不费劲。

GPT-3 也一样。如果你给它的任务复杂到算出下一个令牌（token）的那点时间里做不完，它可能就会编造（confabulate）出一个错误的猜测。但跟人类似，这未必说明模型做不了这个任务。给它一些把道理想明白的时间和空间，它仍然可能给出可靠的回答。

举个例子，拿下面这道关于杂耍球的数学题去问 gpt-3.5-turbo-instruct，它答错了：

gpt-3.5-turbo-instruct Q: A juggler has 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there? A:

gpt-3.5-turbo-instruct There are 8 blue golf balls.

这是否说明 GPT-3 连简单数学题都不会做？不是。事实上，只要在提示里加一句 Let's think step by step，模型就能稳稳解出这道题：

gpt-3.5-turbo-instruct Q: A juggler has 16 balls. Half of the balls are golf balls and half of the golf balls are blue. How many blue golf balls are there? A: Let's think step by step.

gpt-3.5-turbo-instruct There are 16 balls in total. Half of the balls are golf balls. That means that there are 8 golf balls. Half of the golf balls are blue. That means that there are 4 blue golf balls.

单看一个例子，当然很难判断 Let's think step by step 这招是普遍有效，还是只是在这道题上碰巧撞对了。但它确实管用。在一个文字算术题基准上，这招把 GPT-3 的解题率大幅拉高，从毫无用处的 18% 到了像样的 79%！

模型能力取决于上下文

学习使用 GPT-3 的过程中，一个常见的概念性错误，是认为模型的能力在所有场景下都是固定的。比如，GPT-3 答错一道简单的逻辑题，就断定它不具备简单的逻辑能力。

但 Let's think step by step 这个例子说明，GPT-3 看似失败的回答，有时换一个更好的提示词就能纠正。好的提示词能帮模型把自己引向正确的输出。

如何提高复杂任务上的可靠性

本文剩下的部分介绍几种提高大语言模型（large language model）在复杂任务上可靠性的技巧。有些技巧只针对特定类型的问题，但更多技巧建立在通用原则上，能用在各种任务里，比如：

把指令说得更清楚
把复杂任务拆成更简单的子任务
组织指令结构，让模型不偏离任务
提示模型先解释、再回答
让模型为多个可能的答案给出论证，再做综合
生成大量输出，再让模型挑出最好的
微调定制模型，把性能做到最高

把复杂任务分解成更简单的任务

要给模型更多思考的时间和空间，一个办法就是分解（decomposition）：把任务拆成更简单的片段。

举个例子。我们让模型就一段文本做一道选择题，这里的文本是一局《妙探寻凶》（Clue）游戏。直接提问时，gpt-3.5-turbo-instruct 没能把线索 3 和线索 5 接到一起，答错了：

```gpt-3.5-turbo-instruct Use the following clues to answer the following multiple-choice question.

Clues: 1. Miss Scarlett was the only person in the lounge. 2. The person with the pipe was in the kitchen. 3. Colonel Mustard was the only person in the observatory. 4. Professor Plum was not in the library nor the billiard room. 5. The person with the candlestick was in the observatory.

Question: Was Colonel Mustard in the observatory with the candlestick? (a) Yes; Colonel Mustard was in the observatory with the candlestick (b) No; Colonel Mustard was not in the observatory with the candlestick (c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick

Solution: ```

gpt-3.5-turbo-instruct (c) Unknown; there is not enough information to determine whether Colonel Mustard was in the observatory with the candlestick

线索 3 和线索 5 其实已经确定了：Mustard 上校是天文台里唯一的人，天文台里那个人拿着烛台。可模型没能把这两条组合起来，推出正确答案 (a) Yes。

不过，不用直接问答案，我们可以把任务拆成三步：

第一步，逐条过一遍线索，判断每条线索是否可能相关
第二步，把相关的线索合起来，推理出问题的答案
第三步，写出最终答案：(a)、(b) 或 (c)

```gpt-3.5-turbo-instruct Use the following clues to answer the following multiple-choice question, using the following procedure: (1) First, go through the clues one by one and consider whether the clue is potentially relevant (2) Second, combine the relevant clues to reason out the answer to the question (3) Third, map the answer to one of the multiple choice answers: either (a), (b), or (c)

Solution: (1) First, go through the clues one by one and consider whether the clue is potentially relevant: ```

```gpt-3.5-turbo-instruct Clue 1: Miss Scarlett was the only person in the lounge. This clue is not relevant. Clue 2: The person with the pipe was in the kitchen. This clue is not relevant. Clue 3: Colonel Mustard was the only person in the observatory. This clue is relevant. Clue 4: Professor Plum was not in the library nor the billiard room. This clue is not relevant. Clue 5: The person with the candlestick was in the observatory. This clue is relevant.

(2) Second, combine the relevant clues to reason out the answer to the question: Clue 3 and Clue 5 both say that the person with the candlestick was in the observatory. Colonel Mustard was the only person in the observatory, so Colonel Mustard was the person with the candlestick.

(3) Third, write the final answer: (a) Yes; Colonel Mustard was in the observatory with the candlestick ```

给了模型更多思考的时间和空间，又沿着一条推理计划引导它，它就得出了正确答案 (a) Yes。

把复杂指令拆成小任务的另一个好处，是能帮助模型专注于当前的子任务。

比如，让 gpt-3.5-turbo-instruct 用原文的语言总结一段文本，模型可能滑回英文：

```gpt-3.5-turbo-instruct Summarize the text using the original language of the text. The summary should be one sentence long.

Text: """ La estadística (la forma femenina del término alemán Statistik, derivado a su vez del italiano statista, "hombre de Estado") es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad. La estadística es una ciencia formal deductiva, con un conocimiento propio, dinámico y en continuo desarrollo obtenido a través del método científico formal. En ocasiones, las ciencias fácticas necesitan utilizar técnicas estadísticas durante su proceso de investigación factual, con el fin de obtener nuevos conocimientos basados en la experimentación y en la observación. En estos casos, la aplicación de la estadística permite el análisis de datos provenientes de una muestra representativa, que busca explicar las correlaciones y dependencias de un fenómeno físico o natural, de ocurrencia en forma aleatoria o condicional. """

Summary: ```

gpt-3.5-turbo-instruct The text explains that statistics is a science that studies the variability, collection, organization, analysis, interpretation, and presentation of data, as well as the random process that generates them following the laws of probability.

但如果先让模型识别文本的语言，再做总结，输出就变得可靠了：

```gpt-3.5-turbo-instruct First, identify the language of the text. Second, summarize the text using the original language of the text. The summary should be one sentence long.

Language: ```

```gpt-3.5-turbo-instruct Spanish

La estadística es una ciencia que estudia la variabilidad, colección, organización, análisis, interpretación, y presentación de los datos, así como el proceso aleatorio que los genera siguiendo las leyes de la probabilidad. ```

让模型先解释、再回答

另一种提高回答可靠性的有力技巧，是提示模型逐步推理出答案，而不是直接跳到最终答案。'出声思考'的时候，模型命中正确答案的可能性会高得多。

零样本（zero-shot）

方法

按照 Takeshi Kojima 等人 2022 年发表的论文（Large Language Models are Zero-Shot Reasoners），让模型推理出答案最简单的办法，就是直接在答案前面加一句 Let's think step by step.。原文此处有一张图（论文图 2），演示了这个提示格式的一个实例。

结果

作者把这个简单技巧用在 MultiArith 数学数据集上，发现 Let's think step by step 让准确率翻到了四倍，从 18% 到 79%！

原文此处有一张表，列出了该方法在各数据集上的结果。

启示

尽管 Let's think step by step 在数学题上效果不错，但它并非对所有任务都灵。作者发现，它帮助最大的是多步算术题、符号推理题、策略问题这类推理任务；对简单的数学题或常识问题没有帮助，估计对许多其他非推理任务也不会有帮助。

原文此处有一张表，按任务类型列出了这招在哪里有效、在哪里无效。

更多内容请读完整论文。

如果你把这招用在自己的任务上，别怕动手定制指令。Let's think step by step 相当笼统，改用为你的场景收紧了格式的指令，你可能会发现效果更好。比如可以试试更结构化的变体：First, think step by step about why X might be true. Second, think step by step about why Y might be true. Third, think step by step about whether X or Y makes more sense.。你甚至可以给模型一个格式示例，帮它走在正轨上，例如：

```gpt-3.5-turbo-instruct Using the IRS guidance below, answer the following questions using this format: (1) For each criterion, determine whether it is met by the vehicle purchase - {Criterion} Let's think step by step. {explanation} {yes or no, or if the question does not apply then N/A}. (2) After considering each criterion in turn, phrase the final answer as "Because of {reasons}, the answer is likely {yes or no}."

IRS guidance: """ You may be eligible for a federal tax credit under Section 30D if you purchased a car or truck that meets the following criteria: - Does the vehicle have at least four wheels? - Does the vehicle weigh less than 14,000 pounds? - Does the vehicle draw energy from a battery with at least 4 kilowatt hours that may be recharged from an external source? - Was the vehicle purchased in a year before 2022? - If so, has the manufacturer sold less than 200,000 qualifying vehicles? (Tesla and GM have sold more than 200,000 qualifying vehicles.) - Was the vehicle purchased in a year after 2022? - If so, is the vehicle present in the following list of North American-assembled vehicles? (The only electric vehicles assembled in North America are the Audi Q5, BMW 330e, BMW X5, Chevrolet Bolt EUV, Chevrolet Bolt EV, Chrysler Pacifica PHEV, Ford Escape PHEV, Ford F Series, Ford Mustang MACH E, Ford Transit Van, GMC Hummer Pickup, GMC Hummer SUV, Jeep Grand Cherokee PHEV, Jeep Wrangler PHEV, Lincoln Aviator PHEV, Lincoln Corsair Plug-in, Lucid Air, Nissan Leaf, Rivian EDV, Rivian R1S, Rivian R1T, Tesla Model 3, Tesla Model S, Tesla Model X, Tesla Model Y, Volvo S60, BMW 330e, Bolt EV, Cadillac Lyriq, Mercedes EQS SUV, and Nissan Leaf.) """

Question: Can I claim a federal tax credit for my Toyota Prius Prime bought in 2021?

Solution:

(1) For each criterion, determine whether it is met by the vehicle purchase - Does the vehicle have at least four wheels? Let's think step by step. ```

```gpt-3.5-turbo-instruct The Toyota Prius Prime has four wheels, so the answer is yes. - Does the vehicle weigh less than 14,000 pounds? Let's think step by step. The Toyota Prius Prime weighs less than 14,000 pounds, so the answer is yes. - Does the vehicle draw energy from a battery with at least 4 kilowatt hours that may be recharged from an external source? Let's think step by step. The Toyota Prius Prime has a battery with at least 4 kilowatt hours that may be recharged from an external source, so the answer is yes. - Was the vehicle purchased in a year before 2022? Let's think step by step. The Toyota Prius Prime was purchased in 2021, which is before 2022, so the answer is yes. - Was the vehicle purchased in a year after 2022? N/A - If so, is the vehicle present in the following list of North American-assembled vehicles? N/A

(2) After considering each criterion in turn, phrase the final answer as "Because of {reasons}, the answer is likely {yes or no}."

Because the Toyota Prius Prime meets all of the criteria for a federal tax credit, the answer is likely yes. ```

少样本（few-shot）示例

方法

提示模型推理出答案的做法有很多种。一种是给几个示例做示范（'few-shot'，少样本），Google 的 Jason Wei、Denny Zhou 等人研究过这条路线。下面是一个少样本思维链（chain of thought）提示的例子：

原文此处有一张图，展示的正是这样一个少样本思维链提示（出自论文 Chain of Thought Prompting Elicits Reasoning in Large Language Models）。

更多由人工标注者撰写的推理链示范：

原文此处有一张图，列出了这些例子。

（注意：梨到底会不会浮在水面上，这一点被人质疑过）

结果

在小学水平的数学题测试里，作者发现思维链提示让解题率翻了三倍，从 18% 到 57%。

原文此处有一张图（论文图 5），对比了各数据集上的表现。

除了数学题，思维链提示还拉高了体育理解、掷硬币追踪、末字母拼接这几类问题的表现。多数情况下，不需要多少示例就能吃到几乎全部收益（8 个以内）。

原文此处有一张图（论文图 11），列出了这些任务上的具体结果。

更多内容请读完整论文。

启示

和 Let's think step by step 这招相比，基于少样本示例的做法有一个好处：你能更容易地指定模型在落到最终答案之前，该用什么格式、什么长度、什么风格来推理。当模型一开始推理的方向或深度不对头时，这一点特别有用。

微调（fine-tuned）方案

方法

一般来说，想在任务上榨出最高性能，你需要微调一个定制模型。但拿解释来微调模型，可能要成千上万条带解释的示例，写起来成本很高。

2022 年，Eric Zelikman、Yuhuai Wu 等人发表了一套聪明的流程：用一个少样本提示去生成一批解释数据集，拿它来微调模型。思路是：用少样本提示生成候选解释，只保留那些能推出正确答案的解释；然后，为了给一部分答错的答案补充解释，把正确答案写进题目里，再重试同一个少样本提示。作者把这套做法称为 STaR（Self-taught Reasoner，自学成才的推理者）：

原文此处有一张图，演示了 STaR 的完整流程。

用这招，你能把微调的好处和思维链提示的好处合在一起，而不必亲手写上千条示例解释。

结果

作者把这套技巧用在一个常识问答（Common Sense Q&A）数据集上，发现 STaR 同时强过单独使用思维链提示（73% 对 37%）和单独微调（73% 对 60%）：

原文此处有一张表，列出了这组对比结果。

更多内容请读完整论文。

启示

用少样本提示去扩充或改造微调数据集，这个思路不限于写解释这一件事。比如，你手头如果有大量想用来训练的非结构化文本，也许可以找到机会，用提示从这些文本里抽出一个结构化数据集，再拿这个结构化数据集去微调定制模型。

（下篇继续）