AgentFlare

Scholar · reinforcement learning from human feedback

RLHF: Research Digest

10 foundational papers with 388 combined citations, plus the latest arXiv preprints.

Foundational papers10388 citations
Most citedContrastive preference learning: Lear…207 citations · 2024
arXiv, Sep 20264 papers-85% vs Oct 2025
Newest preprint2026-10-25

AISA Scholar API · arXiv

arXiv submissions per monthPapers matching abs:"reinforcement learning from human feedback" · last 12 complete months
monthly arXiv submissions: Oct 2025 27 → Sep 2026 4; high 34 (May 2026), low 4 (Sep 2026)2132536Oct 2025Mar 2026Sep 2026
402 · Pay-per-Crawl$0.02

You're reading the full report — free for humans and search engines. The same content ships as a structured JSON dataset (normalized facts, sources and citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open HTTP 402 payment protocol.

⚡ Agents: fetch .data.json · $0.02 → agents: crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify

What’s new on arXiv

A12026-10-25cs.CL

Large Language Models: A Plain-Language Overview for Engineers

Bahaeddin Türkoğlu

Large language models (LLMs) are neural networks trained on huge text collections to generate text by predicting what comes next. Engineers using ChatGPT and other generative artificial intelligence (AI) tools need a clear picture of what training changes and prompts add. The article starts with the Transformer, which connects words in context, and pre-training, where models learn to predict text. It then explains how fine-tuning adapts models to requests, with reinforcement learning from human feedback using human answer rankings. Next, retrieval-augmented generation (RAG) supplies documents, while chain-of-thought prompting encourages intermediate steps and agents use tools. FlashAttention lowers training and running costs by reducing data movement during attention; quantized low-rank adaptation (QLoRA) saves fine-tuning memory. Readers get a life cycle figure and tables comparing methods and their limits. A short checklist for judging LLM evaluation claims flags test material in training (data contamination) and model-based grading (LLM-as-a-judge). It also covers very long inputs (long context) and hallucination, meaning false or unsupported claims. Check factual accuracy separately from whether an answer sounds helpful.

A22026-10-18cs.LG

Robust Direct Preference Optimization under Preference Label-Flip Attacks

Chenye Yang, Weiyu Xu, Lifeng Lai

Direct Preference Optimization (DPO) is widely used for offline Reinforcement Learning from Human Feedback. However, recent works have shown that DPO can be easily manipulated by adversarial preference label flip attacks, which are specifically designed to mislead the learner toward an attacker-chosen target policy. To address this issue, we design robust DPO methods that can defend against such attacks. Building on the property that lable flips induce parameter-independent gradient shifts in log-linear DPO, we propose Trim-K Robust-DPO, which removes the K highest-impact pairs, and Soft-Trim-K Robust-DPO, which smoothly downweights them. We establish deterministic gradient perturbation bounds uniformly over all attack sets containing at most K flips, explicitly separating defense-induced bias from surviving attack effects. Under bounded-region assumptions, these bounds yield time-uniform parameter and policy deviation guarantees. We further quantify how quadratic regularization improves curvature while introducing additional bias. Experiments on Stanford Human Preferences, HH-RLHF, and UltraFeedback-Binarized validate our analysis.

A32026-10-16cs.GT

Proportionality in Ranking Compression

Daniel Halpern, Xinyu Liu, Evi Micha, Yu Peng Ng, Warut Suksompong

Motivated by applications in reinforcement learning from human feedback, we consider a setting where a given collection of rankings needs to be compressed into a smaller collection of rankings. This can also be seen as a combination of two fundamental social choice frameworks: ranking aggregation and multiwinner voting. We propose three proportionality notions that capture different types of representation in this setting: positional proportionality, pairwise proportionality, and proportionality for solid coalitions (PSC). On the one hand, we show that positional proportionality and PSC are always satisfiable for any input rankings and target compression size, and a desired output can be found in polynomial time. On the other hand, we prove that pairwise proportionality cannot be satisfied in general, but can nevertheless be attained when the input rankings are single-peaked or single-crossing.

A42026-10-10cs.CV

Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning

Thanh Nguyen, Hee Suk Yoon, Tung Minh Luu, Eunseop Yoon, Chang D. Yoo

Large Vision-Language Models (LVLMs) have excelled in joint visual and language understanding, particularly in image captioning. However, LVLMs remain prone to object hallucination, describing non-existent entities in the given image. Previous remedies, including dataset-enhanced supervised training and reinforcement learning from human feedback, show promise but rely heavily on costly human involvement, limiting scalability. To address this, we propose a reinforcement learning framework to mitigate object hallucination in LVLM-based image captioning, driven entirely by natural language processing (NLP) metrics. While optimizing reinforcement learning solely on NLP metrics offers scalability, it also introduces pitfalls: reward hacking, sparse feedback, and costly training. Our framework sidesteps these by introducing an Object Consistency Score (OCS) capturing both correctness and coverage of objects, incorporating KL-regularization for denser token-level guidance, and designing a lightweight PPO variant that fine-tunes adapters atop a frozen LVLM backbone, achieving efficiency and stability. Extensive experiments on the InstructBLIP baseline demonstrate the effectiveness and generalizability of our framework, achieving a substantial hallucination reduction of up to 41\% on the COCO dataset and 29\% on the Visual Genome. Furthermore, the framework can enhance caption quality by incorporating complementary evaluation metrics such as BERTScore and METEOR. These results reveal a broader insight in multimodal learning: LVLMs can effectively self-correct hallucinations using metric feedback, motivating deeper exploration into scalable reinforcement frameworks that significantly reduce human effort.

A52026-10-07cs.HC

A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration

Alexandra Coroiu, Andrea Vogt, Viktor Werbilo, Andreas Poppele, Johann Christensen, Sven Hallerbach

Human-Robot Collaboration (HRC) can facilitate mass customisation in Industry 4.0, with Reinforcement Learning from Human Feedback (RLHF) representing a promising approach for developing safe AI-based robots. Practical challenges remain regarding safety during AI development, human feedback quality, and bidirectional human-robot adaptation. We conducted a scoping review of RLHF in HRC systems, mapping methods that address these challenges. Following PRISMA guidelines, we screened 199 records and included 20 peer-reviewed publications (2020-2025) spanning multiple HRC domains. To our knowledge, this is the first review focused on the bidirectional, closed-loop design of RLHF. Our review found multiple feedback modalities enabling data collection in various feedback formats. Collected data can be integrated at different stages of AI training, resulting in a multi-step development process. Pilot experiments are commonly used to evaluate HRC systems based on both human and robot metrics. To empirically test a key gap identified in the review, we conducted a between-subjects VR experiment comparing system- and user-initiated feedback on robot proxemic behaviour for safe navigation. Using Bayesian models, we analysed the relation between the collected feedback and safety metrics: psychological safety (post-experiment questionnaire) and physical safety (inverse time-to-collision). Results show that user-initiated feedback captures perceived safety better than system-initiated feedback, indicating that feedback timing directly affects feedback quality. Our review and experiment findings show that RLHF relies on appropriate feedback methods to ensure AI safety in HRC, and future RLHF research should prioritise realistic HRC experiments evaluating the effects of feedback collection methods on relevant human and robot metrics.

A62026-10-04cs.AI

AI Safety via Debate is Compromised by Cognitive Biases

Gefei Liu, Sonya Rashkovan, Sophia Lloyd George, Isaac Sheidlower, Serena Booth

Reinforcement learning from human feedback (RLHF) has played a central role in making large language models responsive to human instructions. However, human evaluators often favor flattering or persuasive responses over truthful ones, creating incentives for models to appeal to evaluators at the expense of accuracy. AI safety via debate has been proposed as a way to improve the supervision of language models: in this paradigm, two agents argue opposing positions and challenge each other's claims, potentially exposing falsehoods to the adjudicator. A central premise of AI safety via debate is that truthful arguments are easier to defend than false ones under adversarial scrutiny. In this work, we investigate whether this advantage persists when debaters use rhetorical strategies that exploit biases in human judgment. Inspired by competitive debate, we construct 68 LLM-generated dialogues about detective mysteries with known culprits, spanning four interventions: anchoring, fallacy oversight, pro-jargon, and verbosity. We apply each intervention to either the side advocating for the true culprit or the side advocating for an innocent suspect, allowing us to distinguish influence on adjudication from correctness. In a study with 369 participants, we find that, pooled across bias types, these interventions significantly shift judgments toward the manipulated side. These findings expose a vulnerability in debate-based supervision: human adjudication is sensitive to manipulative rhetorical strategies.

A72026-10-04cs.LG

Groupwise Distortion Guarantees for Preference-Based Alignment

Jacob Brodkey, Roberto Tamez, Aaron Roth

Preference-based alignment methods such as reinforcement learning from human feedback (RLHF) and Nash learning from human feedback (NLHF) aggregate pairwise preferences to learn an LLM policy, but a natural goal is maximizing social welfare (average cardinal utility), which comparisons alone need not identify. Gölz, Haghtalab, and Yang (GHY) measure the gap by distortion: the worst-case ratio between the welfare of the best fixed lottery (distribution over responses) and of the learned lottery. They show NLHF is optimal when every user receives the same lottery. Account-based LLMs, however, have information about their users and can serve different lotteries to different people. We give an efficient algorithm, GLHF, that learns a single group-conditioned policy from one comparison per user. Under individual Bradley--Terry comparisons, GLHF asymptotically matches GHY's optimal population distortion bound simultaneously on every group in a prespecified, possibly overlapping collection, with sample complexity growing logarithmically in the number of groups and inversely with the smallest group mass. A sharper guarantee for groups with similar preferences approaches distortion of one when members share a feasible favorite response. In experiments using human coffee ratings and synthetic LLM-generated ratings, GLHF lowers distortion in every evaluated group and substantially reduces worst-group distortion relative to NLHF and other group-agnostic baselines.

A82026-10-04cs.AI

Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment

Jinghao Pang, Jitai Hao, Qiang Huang, Zhaochun Ren, Jun Yu

Large Language Models (LLMs) have achieved remarkable capabilities but remain vulnerable to jailbreak attacks that elicit harmful or unsafe outputs. Existing safety alignment approaches, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), often require substantial attack-specific supervision and computational resources, while remaining susceptible to shallow safety alignment and over-refusal. To address these challenges, we introduce SSRFT(Supervised Safe-Role Fine-Tuning), the first framework that reformulates safety alignment as the internalization of a predefined safe role. SSRFT constructs a Safe-Role Question-Answer (SRQA) dataset from psychometric questions, limited jailbreak prompts, and a safe-role description. Role-consistent responses are synthesized, validated, and expanded into diverse scenarios, enabling models to internalize safety-oriented values and principles rather than explicit refusal patterns. Experiments across multiple Base and Instruct models show that SSRFT achieves more robust and generalizable safety alignment than standard SFT. SSRFT shows substantially greater robustness to prefilling attacks and better generalization to unseen jailbreak domains, while reducing over-refusal on benign queries and preserving the model's general capabilities. These results establish safe-role internalization as an effective alternative to refusal-centric safety alignment. Warning: This paper contains examples of harmful and toxic language.

Foundational papers

  1. 2020
    Accelerating reinforcement learning agent with eeg-based implicit human feedback13 citations
  2. 2024
    Contrastive preference learning: Learning from human feedback without reinforcement learning207 citations
  3. 2024
    Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback63 citations
  4. 2024
    Parameter efficient reinforcement learning from human feedback32 citations
  5. 2024
    Mapping out the space of human feedback for reinforcement learning: A conceptual framework8 citations
  6. 2025
    M3hf: Multi-agent reinforcement learning from multi-phase human feedback of mixed quality23 citations
  7. 2026
    Learning to summarize user information for personalized reinforcement learning from human feedback9 citations
  8. 2026
    Privacy-preserving reinforcement learning from human feedback via decoupled reward modeling4 citations
  9. 2026-10
    Latest on arXiv: Large Language Models: A Plain-Language Overview for Engineerscs.CL
Citations per paperCitation counts via the AISA Scholar API
citations: R1 Contrastive preferenc… 207, R2 Tlcr: Token-level con… 63, R3 Parameter efficient r… 32, R4 On the challenges and… 22, R5 Learning to summarize… 9, R6 Mapping out the space… 8, R7 M3hf: Multi-agent rei… 23, R8 Integrating human fee… 7, R9 Accelerating reinforc… 13, R10 Privacy-preserving re… 4R1 Contrastive preferenc…207R2 Tlcr: Token-level con…63R3 Parameter efficient r…32R4 On the challenges and…22R5 Learning to summarize…9R6 Mapping out the space…8R7 M3hf: Multi-agent rei…23R8 Integrating human fee…7R9 Accelerating reinforc…13R10 Privacy-preserving re…4
R3202432 citations

Parameter efficient reinforcement learning from human feedback

H Sidahmed, S Phatale, A Hutcheson, Z Lin et al.

… We summarize the reinforcement learning policy results in the bottom … human feedback on 50 samples. We calculate the agreement between the human and judge model’s feedback …

How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.

Sources & citations

  1. Contrastive preference learning: Learning from human feedback without reinforcement learning 207 citations
  2. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback 63 citations
  3. Parameter efficient reinforcement learning from human feedback 32 citations
  4. On the challenges and practices of reinforcement learning from real human feedback 22 citations
  5. Learning to summarize user information for personalized reinforcement learning from human feedback 9 citations
  6. Mapping out the space of human feedback for reinforcement learning: A conceptual framework 8 citations
  7. M3hf: Multi-agent reinforcement learning from multi-phase human feedback of mixed quality 23 citations
  8. Integrating human feedback into a reinforcement learning-based framework for adaptive user interfaces 7 citations
  9. Accelerating reinforcement learning agent with eeg-based implicit human feedback 13 citations
  10. Privacy-preserving reinforcement learning from human feedback via decoupled reward modeling 4 citations
  11. Large Language Models: A Plain-Language Overview for Engineers arXiv · 2026-10-25
  12. Robust Direct Preference Optimization under Preference Label-Flip Attacks arXiv · 2026-10-18
  13. Proportionality in Ranking Compression arXiv · 2026-10-16
  14. Metric-Driven Reinforcement Learning for Object Hallucination Reduction in LVLM-Based Image Captioning arXiv · 2026-10-10
  15. A Scoping Review and Experimental Study on Reinforcement Learning from Human Feedback for Human-Robot Collaboration arXiv · 2026-10-07
  16. AI Safety via Debate is Compromised by Cognitive Biases arXiv · 2026-10-04
  17. Groupwise Distortion Guarantees for Preference-Based Alignment arXiv · 2026-10-04
  18. Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment arXiv · 2026-10-04

Related