Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions

Trivedi H., Balasubramanian N., Khot T., Sabharwal A. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. ACL, 2023.

概

CoT (Chain of Thought) + 检索.

IRCoT

• 对于如上的问题, "In what country was Lost Gravity manufactured?" 单独问 LLM 或者单独在维基百科上搜索, 都很难得到答案. 但是通过如下步骤或许可以得到最终的答案:

  1. 首先将该问题在维基百科上搜索, 得到关于 《Lost Gravity》 的一些信息.
  2. LLM 能够从该信息中抓取到 《Lost Gravity》 的制作公司为 Mack Rides.
  3. 接着在维基百科中搜索 "The Lost Gradvity was manufactured by Mack Rides", 我们会得到一些关于 Mack Rides 的信息.
  4. 基于该信息我们能够得到 (通过 LLM 抓取) "Mack Rides" 是一家德国公司的信息.
  5. 最终我们能够得到答案为: Germany.

• 上面的例子就是告诉我们, 很多问题是需要检索 + 提取信息交替进行最后才能得到答案的.

• 所以 IRCoT 的步骤就是迭代如下的过程:

  1. 已经有了外部的信息和思维链如下:
     

  2. 将该 prompt 喂入 LLM 得到输出, 并保留第一行, 若输出中带有 "answer is" 则停止 (或者迭代次数达到上限的时候也停止).

  3. 将这第一行通过检索器检索出一些额外的信息, 再次进入 1.

代码

[official]

本栏目推荐文章
• [Design-Data-Intensive-Application] Data Structrure that Power Database
• [Design-Data-Intensive-Application] Scalability
• [Design-Data-Intensive-Application]- Reliability
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• 语言模型的预训练[6]：思维链(Chain-of-thought，CoT)定义原理详解、Zero-shot CoT、Few-shot CoT 以及在LLM上应用
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Chain-of-Thought Prompting in Large Language Models 论文学习
• 大模型思维链（Chain-of-Thought）技术原理

Knowledge-Intensive Chain-of-Thought Interleaving Multi-Step Intensiveknowledge-intensive chain-of-thought interleaving knowledge-intensive retrieval-augmented knowledge-intensive generation multi-step interleaving chain-of-thought intensive design-data-intensive-application application design-data-intensive-application chain-of-thought prompting language thought