A Zeroth-Order Paradigm for LLM Preference Alignment

Evan: Welcome to Daily Paper Cast.

Ashley: Today we’re covering a paper from Hugging Face's daily paper list of September 17, 2026, which has garnered 21 upvotes.

Evan: The paper we’re discussing is titled 'A Zeroth-Order Paradigm for LLM Preference Alignment.'

Ashley: Peter Chen and Xi Chen are the first two authors.

The corresponding author, Peter Chen, is from the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley.

Evan: All right, let’s dive into the Introduction.

Background first: Generative AI has become a critical tool across academia, industry, and government.

Ashley: Exactly.

Large language models, or LLMs, are central to this progress, excelling in areas like data organization, retrieval, reasoning, and analysis.

Evan: However, LLMs trained on diverse corpora often need further alignment with human preferences to ensure their responses are helpful, harmless, and reliable.

Ashley: Correct.

A common approach to achieve this alignment is reinforcement learning from human feedback, or RLHF, which involves learning a reward model from human preference pairs, followed by policy optimization using reinforcement learning.

Evan: But RLHF is computationally expensive, requiring a multi-stage training pipeline involving significant memory and computation resources.

Ashley: Yes, which is why direct alignment methods have become popular.

These methods, like direct preference optimization or DPO, directly optimize policy using preference pairs, bypassing the need to train a separate reward model.

Evan: However, even these methods face issues.

One major challenge is 'likelihood displacement,' where the training inadvertently shifts the probability mass away from preferred responses.

Ashley: That’s right.

Likelihood displacement can cause the absolute probability of the preferred responses to decrease, leading to unintended unalignment, where the model could produce unsafe or unhelpful responses.

Evan: The researchers noted that this displacement often arises from preference pairs with small likelihood margins, deemed 'noisy preference pairs.'

Ashley: Indeed.

Unlike approaches that filter out these noisy pairs, Peter Chen and team proposed a novel method called Comparison-based Preference Optimization, or ComPO.

This method leverages the comparative information from these noisy pairs instead of discarding them.

Evan: Ashley, how does ComPO differ from traditional optimization methods?

Ashley: Great question.

ComPO is a zeroth-order alignment method that doesn't directly optimize a differentiable preference loss.

Instead, it uses comparison oracles to extract directional information from preference pairs.

Evan: So, rather than optimizing based on a fixed loss function, ComPO perturbs the current policy and evaluates whether each perturbation improves the likelihood of preferred responses over dispreferred ones.

Ashley: Correct.

This allows ComPO to utilize the information in low-margin preference pairs without risking the adverse effects associated with directly optimizing noisy pairs.

Evan: The researchers provide a convergence guarantee for ComPO under certain conditions, which is always reassuring for those concerned with theoretical foundations.

Ashley: And they didn’t stop there; they also introduced an online version of ComPO, which integrates unlabeled policy generations for controlling deviations from a reference policy.

Evan: Interesting.

It appears they framed this through a coverage perspective, ensuring local coverage and in-distribution pairwise reward accuracy.

Ashley: Exactly.

They experimented with several models like Mistral, Llama, and Gemma-3 and found that ComPO outperforms existing methods, providing valuable insights into mitigating likelihood displacement.

Evan: And with that, we’ve covered the Introduction section.

Evan: Ashley, let's delve into the methodologies proposed by the authors for ComPO.

Ashley: Sure, Evan.

The proposed method, Comparison-based Preference Optimization or ComPO, is designed to address the limitations of existing direct alignment methods, particularly the issue of likelihood displacement.

Evan: Could you break down the key components of the ComPO method for us?

Ashley: Of course.

The core idea behind ComPO is to use comparison oracles to extract directional information from noisy preference pairs, which are pairs where the preferred and dispreferred responses have similar likelihoods from the model's perspective.

Evan: So, rather than ignoring these noisy pairs, ComPO utilizes them constructively?

Ashley: Exactly.

To put this into practice, ComPO employs a process that involves policy perturbation and evaluation based on preference comparison oracles.

Evan: Can you elaborate on these comparison oracles and how they function within ComPO?

Ashley: Certainly.

A comparison oracle is a mechanism that takes two policies and evaluates which of them is better based on the likelihoods of the preferred versus dispreferred responses.

Specifically, it assigns a 'better' or 'worse' label to each perturbed policy based on this evaluation.

Evan: Ah, so ComPO relies on these labels to guide its optimization process?

Ashley: Correct.

Here's a step-by-step breakdown of how it works: First, ComPO starts with an initial policy.

It then generates multiple perturbed versions of this policy by applying random perturbations.

Evan: And how are these perturbations evaluated?

Ashley: For each perturbed policy, a comparison oracle evaluates whether the likelihood of the preferred response has increased and the dispreferred response has decreased.

Based on this, the oracle assigns either a positive or negative signal.

Evan: I see.

And then these signals are aggregated to update the policy, right?

Ashley: Precisely.

The aggregated signals help estimate a normalized update direction.

This direction informs how the policy should be adjusted to align better with human preferences without directly optimizing a noisy loss function.

Evan: Interesting.

But what ensures that this method converges to an optimal policy?

Ashley: Good point.

The authors prove a convergence guarantee for ComPO's offline scheme, stating that under conditions of smoothness and gradient sparsity, the method converges to a policy where the gradient of the underlying alignment objective is small.

Evan: That sounds promising.

Did they also introduce an online version of ComPO?

Ashley: Yes, they did.

The online version of ComPO retains the comparison mechanism but includes additional components like reverse-KL control.

This constrains the policy divergence from a reference policy to a prescribed neighborhood.

Evan: How does this reverse-KL control work in practice?

Ashley: In practice, the reverse-KL control is estimated using generations from the current policy.

It translates the deviation into a step-size adjustment, ensuring that the updates maintain performance without deviating too much from the reference.

Evan: And what about the practical implementation of ComPO?

Did they manage to simplify it for real-world use?

Ashley: Indeed.

The practical implementation applies perturbations only to the output layer of the model, significantly reducing computational costs.

Additionally, it uses a sparse gradient estimation technique where only significant gradient entries are updated.

Evan: This sounds very efficient.

Were there any experimental setups or benchmarks discussed in the methodology?

Ashley: Yes, the authors evaluated ComPO on various models, including Mistral-7B, Llama-3, Gemma-2, Qwen3, and Gemma-3.

They optimized ComPO over clean and noisy preference pairs across these models and used the evaluation metrics like length-controlled win rates and raw win rates to benchmark its performance.

Evan: Were there any unique results or findings in these evaluations?

Ashley: Several findings stood out.

For instance, ComPO improved length-controlled win rates, meaning it was able to maintain or improve performance while controlling the verbosity of responses.

This indicates better efficacy in managing the trade-off between response length and quality.

Evan: That's quite a feat, considering verbosity often complicates model responses.

Did they have any specific metrics or diagnostic tools to support these findings?

Ashley: Certainly.

The authors used pair-level likelihood diagnostics as an additional measure to support their findings.

These diagnostics provided evidence that ComPO effectively mitigated likelihood displacement, showing substantial improvements across various metrics.

Evan: Fascinating.

So, in summary, ComPO presents a novel and efficient method for aligning LLMs with human preferences by leveraging comparison oracles and addressing noisy preference pairs.

Ashley: Exactly.

It provides a solid alternative to traditional methods with theoretical guarantees and practical efficiencies, making it a noteworthy advancement in the field of AI preference alignment.

Evan: And that wraps up our detailed look at the Method section of this paper.

Evan: Ashley, let’s dive into the Experimental setup and Results discussed by the authors.

From what I understand, they conducted multiple experiments to evaluate ComPO's performance.

Can you start by telling us about their experimental setup?

Ashley: The authors conducted experiments on various models, including Mistral-7B, Llama-3-8B, Gemma-2-9B-it, Qwen3-4B, and Gemma-3-4B-it.

They used different configurations, such as base and instruction-tuned versions of these models.

Evan: And how did they segment the data for these experiments?

Ashley: They identified and separated preference pairs into clean and noisy subsets using a margin criterion.

Essentially, they used pairs with high confidence levels as clean and those with small likelihood margins as noisy.

Evan: That makes sense.

What specific benchmarks or evaluation metrics did they use?

Ashley: They utilized several benchmarks, including AlpacaEval 2, Arena-Hard, and MT-Bench.

For AlpacaEval 2, they reported both raw win rate (WR) and length-controlled win rate (LC), which adjusts preferences based on response length.

Evan: Interesting.

What were some of their primary findings?

Ashley: The results were quite promising.

For instance, applying ComPO to DPOclean, the model trained on only clean pairs, improved the length-controlled win rate significantly.

In some instances, they observed up to a 2.5% increase in length-controlled win rate.

Evan: Did they compare ComPO directly with any baseline methods?

Ashley: Yes, they did.

They compared DPO with ComPO directly and also applied ComPO as an augmentation to existing SimPO checkpoints.

Additionally, they performed ablation studies to understand the effect of different perturbation numbers and gradient thresholds.

Evan: Were there any notable comparisons or findings from these studies?

Ashley: Certainly.

One notable comparison was between offline ComPO and SimPO.

The incorporation of ComPO into SimPO showed consistent improvements across various benchmarks.

For example, SimPO+ComPO surpassed SimPO in AlpacaEval 2’s length-controlled metrics by about 1.5%.

Evan: That’s impressive.

And how about the specific metrics and diagnostic tools they used?

Ashley: They used pair-level likelihood diagnostics to measure the effectiveness of ComPO.

This helped them understand how well ComPO mitigated likelihood displacement and improved model alignment.

These diagnostics provided concrete evidence of its success.

Evan: How efficient is ComPO in practical, real-world applications?

Ashley: The practical implementation was designed to be highly efficient.

They applied perturbations only to the output layer, significantly reducing computational overhead.

For example, perturbing only 1% of the output layer’s parameters was often sufficient to achieve significant gains.

Evan: That’s a game-changer for models that require frequent updates and real-time performance.

Ashley: Indeed.

Moreover, they experimented with online ComPO, which incorporates damping mechanisms such as reverse-KL control using unlabeled policy generations.

Evan: And how did the online experiments pan out compared to the offline ones?

Ashley: In their experiments, adding online damping and replay resulted in even better performance.

For example, online ComPO showed improvements in AlpacaEval 2 and Arena-Hard benchmarks, corroborating the effectiveness of incorporating unlabeled policy generations.

Evan: Did they mention any specific challenges or limitations during their experiments?

Ashley: One challenge they noted was the computational cost associated with larger perturbation counts.

Although necessary for high precision, it increased computation time, which is a trade-off for accuracy.

Evan: Were there any particular configurations or models where ComPO particularly excelled?

Ashley: Yes, the Mistral-7B-Instruct and Llama-3-8B-Instruct models showed considerable enhancements in performance benchmarks when augmented with ComPO.

Especially in length-controlled scenarios, their performance stood out.

Evan: So, what’s the takeaway from the experiment section about the practical use of ComPO?

Ashley: The experiments clearly demonstrate that ComPO not only mitigates likelihood displacement but also improves overall model performance efficiently.

By leveraging noisy preference pairs, it offers a robust alternative to traditional direct alignment methods.

Evan: That wraps up our discussion on the Experiment section of the paper.

Evan: Ashley, it's time to look at how this research fits into the broader landscape of AI preference alignment.

Let's discuss the Related Work section.

Ashley: Certainly, Evan.

The authors of this paper position their work against a rich backdrop of previous research in preference alignment, zeroth-order optimization, and related methodologies.

Evan: Let's start with preference alignment methods.

What did they highlight here?

Ashley: They discuss a variety of direct preference alignment methods, beginning with Direct Preference Optimization, or DPO, which optimizes policies directly using preference pairs without the need for learning a reward model.

Evan: And what are some of the limitations of DPO that the authors pointed out?

Ashley: Great question.

DPO tends to suffer from likelihood displacement, where the model inadvertently shifts probability mass away from preferred responses, potentially resulting in unsafe behavior.

Evan: That sounds problematic.

How do other works attempt to address these limitations?

Ashley: Several approaches have been proposed.

For instance, Azar et al. (2024) extended DPO to a wider class of problems without explicitly introducing a reward function.

Another approach, Rafailov et al. (2024b), modified DPO to handle token-level transitions for more granular control.

Evan: Interesting.

And how does ComPO fit within this context?

Ashley: ComPO is designed to address the issue of noisy preference pairs in a novel way by using comparison oracles rather than filtering them out.

This makes it more robust against the likelihood displacement that plagues traditional DPO methods.

Evan: Fascinating.

What about zeroth-order optimization methods?

Were they discussed as well?

Ashley: Yes, indeed.

Zeroth-order optimization methods have been explored extensively in literature for settings where the gradients aren't directly accessible.

Specifically, the authors reference works like Flaxman et al. (2005) and Duchi et al. (2015) for foundational techniques.

Evan: I see.

And did they mention any specific zeroth-order methods that are particularly relevant to their work?

Ashley: The paper builds on one-bit, comparison-based estimators, as discussed in Cai et al. (2022a).

These approaches are useful for high-dimensional settings and sparse gradients, aligning well with the problem ComPO seeks to solve.

Evan: It's amazing to see how interconnected these methods are.

Were there any other areas of related work they touched upon?

Ashley: They also delved into research on learning from noisy preference data.

For instance, Razin et al. (2025) explored filtering out problematic preference pairs, a method ComPO contrasts by leveraging information from these noisy pairs rather than discarding them.

Evan: How does ComPO's approach compare to other methods addressing noisy data directly?

Ashley: ComPO stands out because it treats low-margin preference pairs as valuable sources of comparative signals.

This is different from other methods that might exclude or down-weight such data.

For example, Amini et al. (2024) worked on methods to handle corrupted preference labels, but not quite in the same comparative manner that ComPO employs.

Evan: Got it.

It's clear that ComPO brings a unique angle to the table.

Any notable mentions on the practical challenges or theoretical guarantees?

Ashley: Indeed.

The paper mentions the practical challenge of computational overhead, which they address through output-layer perturbations and sparse gradient estimations.

The theoretical framework they provide, including the convergence guarantees, also sets their work apart from more heuristic approaches.

Evan: That should provide a lot of confidence for practitioners looking to adopt this method.

Was there anything else noteworthy in the Related Work section?

Ashley: One last point is their relation to preference fine-tuning methods like Song et al. (2024b), which emphasize dataset coverage.

ComPO complements this by ensuring local coverage through comparisons without needing explicit reward modeling, adding another layer of robustness.

Evan: That's a comprehensive wrap-up of the Related Work section.

Evan: Ashley, let's wrap up our discussion by summarizing the key contributions and takeaways from this paper.

Ashley: The primary contribution of this paper is the introduction of Comparison-based Preference Optimization, or ComPO, as a novel zeroth-order alignment method that effectively addresses the issue of likelihood displacement in large language models.

Evan: By leveraging noisy preference pairs through comparison oracles, ComPO can utilize comparative signals that are otherwise discarded by traditional methods, leading to improved alignment with human preferences.

Ashley: Right.

What stands out is that ComPO achieves this without directly optimizing a noisy differentiable preference loss.

It updates the policy in a more stable and robust manner by evaluating directional signals from perturbed policies.

Evan: And they didn't stop there.

The paper also introduces an online version of ComPO, which integrates reverse-KL control to ensure policy updates don’t deviate too much from a reference policy, thus maintaining local coverage.

Ashley: This online component is particularly useful for real-time applications, where maintaining alignment without extensive retraining is crucial.

Their experiments demonstrated significant performance improvements across several models and benchmarks.

Evan: Overall, ComPO presents a robust alternative to traditional direct alignment methods, providing a practical and theoretically sound solution to mitigate likelihood displacement.

Ashley: Indeed, it’s a noteworthy advancement in AI preference alignment, offering efficient and effective alignment while addressing the shortcomings of previous methods.

Evan: That brings us to the end of today's episode.

Ashley: We hope you enjoyed our deep dive into the paper 'A Zeroth-Order Paradigm for LLM Preference Alignment.'

Evan: As always, thank you for listening.

Don't forget to subscribe to Daily Paper Cast and stay tuned for more insights and discussions on the latest research in AI, NLP, computer vision, and related fields.

Ashley: Join us again tomorrow for another exciting episode.

Until then, keep exploring and stay curious.

Evan: Take care and see you next time!