AgentFlare

Scholar · AI alignment and safety

AI Alignment & Safety: Research Digest

10 foundational papers with 3 combined citations, plus the latest arXiv preprints.

Foundational papers103 citations
Most citedToward a theory of value in ai alignm…2 citations · 2026
arXiv, Sep 202629 papers-15% vs Oct 2025
Newest preprint2026-10-26

AISA Scholar API · arXiv

arXiv submissions per monthPapers matching (abs:"AI alignment" OR abs:"AI safety") · last 12 complete months
monthly arXiv submissions: Oct 2025 34 → Sep 2026 29; high 48 (Apr 2026), low 23 (Dec 2025)21314050Oct 2025Mar 2026Sep 2026
402 · Pay-per-Crawl$0.02

You're reading the full report — free for humans and search engines. The same content ships as a structured JSON dataset (normalized facts, sources and citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open HTTP 402 payment protocol.

⚡ Agents: fetch .data.json · $0.02 → agents: crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify

What’s new on arXiv

A12026-10-26cs.CR

I-HITL: Inalienable Human-in-the-Loop

Samuel Breckenridge, Dani Vilardell, Wenhao Wang, Fan Zhang, Ari Juels

Human oversight is a pillar of AI safety. It is often exercised by a human in the loop (HITL) who approves critical actions taken by an AI agent. AI agents' increasing ability to bypass guardrails, though, raises the question of how to enforce HITL robustly. We introduce the concept of inalienable human-in-the-loop (I-HITL). An I-HITL protocol ensures that an agent can initiate an action only with explicit human approval, even when the human interacts with a system (e.g., a local agentic harness) compromised by the agent. A strong I-HITL protocol additionally ensures such approval even when an adversarial human attempts to delegate approvals to an agent. To motivate I-HITL, we report on experiments with a popular agentic wallet platform in which raising an agent's spending cap requires HITL approval via emailed one-time passcodes (OTPs). Given access to the user's email and a task blocked by the cap, we find that ten different frontier models will often autonomously raise the cap, harvesting OTPs to bypass a HITL guardrail and surfacing other vulnerabilities. We formally define I-HITL using the concept of communication topologies and prove necessary and sufficient conditions for its realization. We show that an I-HITL protocol requires a trusted entity - a trusted computing device (or equivalent) - and characterize which such devices can and cannot realize an I-HITL protocol. We present a prototype I-HITL system called Countersign that can plug into and strengthen existing HITL systems using off-the-shelf consumer mobile devices.

A22026-10-23cs.CY

AI Alignment for Digital Democracy: Fairer Voters Representation against LLM Biases

Yi Qi, Srijoni Majumdar, Evangelos Pournaras

AI representation is emerging in digital democracy, assisting in voter emulation and decision-making. Large language models (LLMs) behave differently depending on the model, the prompt context, and the voter information they receive, representing some groups more accurately than others and potentially making the emulation unfair to certain groups. We investigate demographic biases in LLM-emulated electorates: how accurately LLMs represent individual votes, whether some voter groups are represented more accurately than others, and whether AI alignment can reduce these disparities without sacrificing overall accuracy. Using three U.S. presidential elections, we compare a closed-source and an open-source LLM across voters' demographic features and test AI alignment strategies toward fairer voter representation. Removing gender from the voter profile reduces the largest group error of the models by 22-42%, and removing education reduces it by 64%, whereas removing self-reported race increases the mean group error by over 40%, all without lowering accuracy. Common alignment strategies, such as repeated queries or fairness instructions, do not reduce these errors and can double them.

A32026-10-23cs.LG

Anti-coherence Regularized Absolute Scaling for Out-of-Distribution Detection

Donghwan Kim, Youngwoong Jin, Hyunsoo Yoon

Out-of-distribution (OOD) detection is recognized as a critical problem in AI safety. SCALE is an effective post-hoc OOD detection method in the activation-shaping family that requires no training data and rescales penultimate activations using a percentile-based magnitude ratio. However, this ratio is ill-defined when negative activations are present, since positive and negative contributions cancel within the underlying sum, causing SCALE to degenerate on backbones such as ViT. We address this with an Absolute Fix that computes the ratio over absolute-valued activations rather than discarding negative activations through ReLU. Building on this corrected scale, we propose Anti-coherence regularization, which penalizes the scaling factor by the fraction of a sample's max-logit contributions to the max logit that are sign-mismatched and thereby suppresses unreliable evidence for the predicted class. Despite requiring no access to training data or training labels, our method ties or outperforms activation-shaping methods that rely on training data in terms of AUROC and FPR@95 on CIFAR-10 and ImageNet under the OpenOOD v1.5 protocol.

A42026-10-23cs.CL

Tampered ML: Evaluating LLM Judgment of Machine Learning Research

Soroush Ebadian, Parand A. Alamdari, Joe Benton, Hailey Joren

One approach to accelerating progress on AI alignment is to automate parts of alignment research itself. Doing so requires systems that can perform reliably on hard-to-supervise research tasks, including identifying flaws in research proposals and evaluating whether a proposed experiment supports its intended claim. To evaluate this capability, we introduce Tampered ML, a benchmark of 1,858 item-pairs, each consisting of an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual that introduces a targeted methodological flaw. Models must identify the more methodologically sound proposal in both presentation orders. A human validation study on a stratified sample estimates that 86% of items are valid (95% CI: 81-91%). Frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy). They are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional.

A52026-10-21cs.CL

Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

Hongyu Li, Xinlong Huang, Zhan Yu, Jing Zhao, Zhe Zhao, Juntao Zhong, Xiong Kuang, Rui Yuan, ZhenXiang Yan, Huilin Xu, Haitao Yu, Lilin Wan…

Current LLM safety benchmarks predominantly rely on binary safe or unsafe labels, lacking explanatory depth and suffering from low consistency across automated judges. To address this, we introduce DiagSafe, a diagnostic evaluation framework that moves beyond binary classification by employing prompt-level rubrics. Grounded in globally applicable AI safety policies, each rubric provides explicit criteria along two dimensions: safety, which targets concrete harms with specific exemptions to suppress false positives, and usefulness, which independently evaluates instruction fulfillment. To ensure a robust and realistic evaluation, we construct a bilingual (English-Chinese) benchmark grounded in a comprehensive 84-label safety taxonomy, consisting of two subsets: DiagBase, which covers single- and multi-turn prompts with direct harmful queries and template-based jailbreaks, and DiagAttack, which embeds harmful intents into complex instruction-following tasks to form challenging instruction attacks. Extensive experiments demonstrate that DiagSafe significantly enhances the reliability of LLM-as-a-judge evaluation, achieving 95.34% agreement with human annotations -- substantially higher than the 69%-87% achieved by the baselines -- and high inter-judge consistency (Cohen's κ = 0.864). Furthermore, applying our framework reveals that leading LLMs differ markedly in their unsafe rates, and each model also exhibits unique vulnerabilities along different dimensions. In addition, we observe that models are particularly susceptible in Chinese and multi-turn contexts. Ultimately, DiagSafe provides a reliable and explainable tool for auditing and optimizing LLM alignment.

A62026-10-20cs.AI

Humans Correct Their Judgments Through Debate, but Weaker AI Models Do Not

Maria Victoria Carro, Denise Alejandra Mester, Facundo Nieto, Oscar Agustin Stanchi, Guido Ernesto Bergman, Giovanni Franco Gabriel Marrafi…

How can we effectively oversee AI systems that surpass human intelligence? AI Safety via Debate proposes that two AI systems argue opposing positions and a judge, either a human or a weaker model, decides which is correct. By facilitating supervisors in providing high-quality training signals, debate aims to encourage truthful and safe behavior from AI systems whose capabilities exceed our own. However, human judgment is not neutral: people often rely on biases which advanced AI systems may learn to exploit. Drawing on insights from collective intelligence and social cognition, this paper investigates whether debate can guide human and LLM supervisors toward the truth despite their prior beliefs. We evaluate several protocol configurations. We also explore setups inspired by the wisdom of the crowd such as the number and diversity of judges, as well as interaction mechanisms including deliberation and voting. Overall, we find that debate increases accuracy relative to prior beliefs for human judges, whereas the consultancy baseline, in which a single expert argues for one answer, does not. For weaker language models, however, the results are mixed: debate provides no comparable gains in most settings and can even reduce accuracy. Among multi-judge protocols, weighted voting produces the highest accuracy for human supervisors, whereas for LLMs none of them significantly improves group accuracy, even as the number of judges increases. Notably, the behavior of minority judges in non-diverse deliberation protocols suggests that, under symmetric settings, incorrect positions could be more persuasive than correct ones. These findings support debate as a viable path toward scalable human oversight, but cast doubt on its benefits for weaker AI supervisors, raising concerns about fully automated training and control pipelines in high-stakes settings.

A72026-10-20cs.CL

Reward Misspecification Degrades Chain-of-Thought Monitorability

Tianyi Lorena Yan, John Hewitt

Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety. As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought. One such practice may be unavoidable: reward misspecification. As RL environments become more complex, it is infeasible to have perfect verification, and models are rewarded for behaviors that are different from the true task intent. We show that reward misspecification alone already degrades monitorability. We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably. Across three models and three tasks, CoT monitorability remains low as models' hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded. To mitigate unmonitorability without prior knowledge of the true reward or the hack, we collect rollouts with diverse rewards across training steps and use a LM to infer a reward description. We then inject the description into the input, which teaches the model to verbalize the reward. This reward description training improves monitorability by an average of 24% or 42% absolute, depending on the metric, across all models and tasks, and yields CoTs that reason about the misspecified reward explicitly. Code is available at https://github.com/Lorenayannnnn/reward-misspecification-degrades-cot-monitorability.

A82026-10-19cs.CL

African Languages AI Safety Readiness Report

Israel Abebe Azime, Jesujoba O. Alabi, Tajuddeen Gwadabe, David Ifeoluwa Adelani

With the widespread adoption of large language models (LLMs) across languages and domains, AI safety has emerged as a critical global concern. However, due to resource constraints, research and development efforts tend to prioritize a limited number of high-resource languages, leaving many others, especially those spoken in Africa, underrepresented. While existing research on language "resource-ness" has largely examined whether sufficient resources are available to develop language technologies, the question of whether languages are adequately represented in AI safety research has received attention only more recently. This report examines the readiness of African languages for inclusion in AI safety discussions, drawing on existing literature and data. It considers four factors: the availability of AI-relevant resources, including text data; national AI policy support; performance on established text benchmarks; and speaker population. Based on these factors, languages are grouped into four readiness categories: fully ready, mostly ready, emerging, and foundational support needed. This classification is intended to inform the inclusion of African languages in AI safety benchmarks, highlight varying levels of preparedness, and guide strategic investment in data development, policy, and community engagement for underrepresented languages.

Foundational papers

  1. 2017
    AI Safety and Alignment in General-Purpose AI Systems: Frameworks, Evaluation, and Governance0 citations
  2. 2026
    Toward a theory of value in ai alignment2 citations
  3. 2026-10
    Latest on arXiv: I-HITL: Inalienable Human-in-the-Loopcs.CR
Citations per paperCitation counts via the AISA Scholar API
citations: R1 Toward a theory of va… 2, R2 On AI operational ali… 1, R3 Epistemic norms for A… 0, R4 AI Alignment and Safe… 0, R5 Full Alignment Is Nei… 0, R6 Safety and Alignment … 0, R7 Evaluating Alignment … 0, R8 The Alignment Paradox… 0, R9 AI Safety and Alignme… 0, R10 AI Alignment with Hum… 0R1 Toward a theory of va…2R2 On AI operational ali…1R3 Epistemic norms for A…0R4 AI Alignment and Safe…0R5 Full Alignment Is Nei…0R6 Safety and Alignment …0R7 Evaluating Alignment …0R8 The Alignment Paradox…0R9 AI Safety and Alignme…0R10 AI Alignment with Hum…0
R120262 citations

Toward a theory of value in ai alignment

A Smart, S Ahmed, J Kay, J Tobin, K Shrishak et al.

… As alignment crystallizes into a core research wing of AI safety, it is crucial to examine the roots of this work through fundamental questions around the very idea of value …

R30 citations

Epistemic norms for AI safety and alignment research

K Navaie

… gap dimensions in current alignment research, including the … Epistemic Code for AI Safety and Alignment comprising eight … that any AI system is safe; it constrains how safety-relevant …

R60 citations

Safety and Alignment Frameworks for High-Autonomy AI Systems

S Chatterjee, P Bansal, R Shrivastva et al.

… safety and alignment framework where technical safety measures are merged with socio-technical governance throughout the entire AI … Third, it is rather hard to generalize AI alignment …

R70 citations

Evaluating Alignment Methods for AI

M Feffer

… investigate how existing approaches to AI alignment fall short of … safety of developed and deployed AI models, which in turn may diverge from stakeholders’ expectations of safety and …

R100 citations

AI Alignment with Human Preferences

M Talukdar

… of AI alignment research, tracing both the technical foundations of alignment methodologies … associated with integrating human preferences into AI systems. Drawing from academic …

How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.

Sources & citations

  1. Toward a theory of value in ai alignment 2 citations
  2. On AI operational alignment and misalignment for high-risk AI systems: case study 1 citations
  3. Epistemic norms for AI safety and alignment research 0 citations
  4. AI Alignment and Safety in Advanced Machine Learning Systems 0 citations
  5. Full Alignment Is Neither Possible nor Beneficial: AI Safety Should Be Built on an Ethics of Non-Intrusion 0 citations
  6. Safety and Alignment Frameworks for High-Autonomy AI Systems 0 citations
  7. Evaluating Alignment Methods for AI 0 citations
  8. The Alignment Paradox: Emotional Flattening and Social Safety Risks in Proactive AI Agents 0 citations
  9. AI Safety and Alignment in General-Purpose AI Systems: Frameworks, Evaluation, and Governance 0 citations
  10. AI Alignment with Human Preferences 0 citations
  11. I-HITL: Inalienable Human-in-the-Loop arXiv · 2026-10-26
  12. AI Alignment for Digital Democracy: Fairer Voters Representation against LLM Biases arXiv · 2026-10-23
  13. Anti-coherence Regularized Absolute Scaling for Out-of-Distribution Detection arXiv · 2026-10-23
  14. Tampered ML: Evaluating LLM Judgment of Machine Learning Research arXiv · 2026-10-23
  15. Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics arXiv · 2026-10-21
  16. Humans Correct Their Judgments Through Debate, but Weaker AI Models Do Not arXiv · 2026-10-20
  17. Reward Misspecification Degrades Chain-of-Thought Monitorability arXiv · 2026-10-20
  18. African Languages AI Safety Readiness Report arXiv · 2026-10-19

Related