LoRA: Low-Rank Adaptation of Large Language Models
Low-rank trainable updates adapt a frozen pretrained model without updating all of its parameters.
THE RESEARCH ATLAS
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
A preference-based objective optimizes a language model directly, avoiding a separate reward-model training stage.
Institution and location sources
Location refers to credited institutions or the listed organization, not author citizenship or an exhaustive collaboration list.
Why read this
Compare the objective and assumptions with an RLHF pipeline.
Citation
@misc{rafailov2023direct,
title = {Direct Preference Optimization: Your Language Model is Secretly a Reward Model},
author = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Stefano Ermon and Christopher D. Manning and Chelsea Finn},
year = {2023},
eprint = {2305.18290},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2305.18290}
}Rights remain with the original authors. Refer to the original source for citations and licensing.
Related research
Training language models to follow instructions with human feedback
Human demonstrations and preference feedback are used to improve how a language model follows instructions.
Training Compute-Optimal Large Language Models
The study examines how to allocate a training-compute budget between model size and training data.