利用人类反馈训练语言模型遵循指令
Training language models to follow instructions with human feedback
利用人类示范与偏好反馈改进语言模型遵循指令的能力。
THE RESEARCH ATLAS
宪法式 AI：从 AI 反馈中学习无害性
以明确原则引导模型自我批评、修订及生成偏好反馈，用于对齐训练。
Constitutional AI: Harmlessness from AI Feedback
为什么值得读
理解监督信号的来源与被训练模型之间的区别。
引用
@misc{bai2022constitutional,
title = {Constitutional AI: Harmlessness from AI Feedback},
author = {Yuntao Bai and Saurav Kadavath and Sandipan Kundu and Amanda Askell and Jackson Kernion and Andy Jones and Anna Chen and Anna Goldie and Azalia Mirhoseini and Cameron McKinnon and Carol Chen and Catherine Olsson and Christopher Olah and Danny Hernandez and Dawn Drain and Deep Ganguli and Dustin Li and Eli Tran-Johnson and Ethan Perez and Jamie Kerr and Jared Mueller and Jeffrey Ladish and Joshua Landau and Kamal Ndousse and Kamile Lukosuite and Liane Lovitt and Michael Sellitto and Nelson Elhage and Nicholas Schiefer and Noemi Mercado and Nova DasSarma and Robert Lasenby and Robin Larson and Sam Ringer and Scott Johnston and Shauna Kravec and Sheer El Showk and Stanislav Fort and Tamera Lanham and Timothy Telleen-Lawton and Tom Conerly and Tom Henighan and Tristan Hume and Samuel R. Bowman and Zac Hatfield-Dodds and Ben Mann and Dario Amodei and Nicholas Joseph and Sam McCandlish and Tom Brown and Jared Kaplan},
year = {2022},
eprint = {2212.08073},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2212.08073}
}论文与代码版权归原作者；引用及许可请以原始来源为准。
相关研究
直接偏好优化：你的语言模型其实也是奖励模型
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
通过偏好优化目标直接训练语言模型，省去单独训练奖励模型的阶段。
DeepSeek-R1：通过强化学习激发大语言模型的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
利用强化学习发展推理行为，并研究可用模型训练及蒸馏流程。