Does Reinforcement Learning Improve Reasoning Beyond the Base Model?

RLVR and reasoning models

Models such as OpenAI’s o1 and DeepSeek-R1, which shook the stock market, were trained using Reinforcement Learning with Verifiable Rewards (RLVR). The basic idea is to reward a model when it performs a long chain of thought and solves a difficult problem correctly.

The question is whether this process truly improves reasoning ability beyond that of the base model. A paper submitted on 18 April 2025, “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” by Yang Yue and colleagues, argues that it does not.

What pass@k reveals

Because a language model generates an answer by continuing tokens, reaching the correct answer can be viewed as one possible outcome among many. To test whether RLVR enables models to do things they previously could not do, we can increase the number of samples drawn from the model.

At small values of k, the RL-trained model is more likely to produce a correct answer. However, when k becomes large—for example, 256 samples—the base model can instead achieve a higher accuracy. This appears at the far right of the paper’s graph.

In other words, as k increases, the base model catches up. RL appears to guide the model toward a sample space in which correct answers are more likely, rather than enabling it to solve problems it could not solve before.

It can even be counterproductive. Narrowing the model’s output distribution to increase the probability of a correct answer may sometimes hurt performance.

• pass@k is the probability that at least one of k runs produces a correct answer.
• The paper reports this pattern consistently across multiple problems and multiple models.

Distillation appears different

What, then, can help a model solve problems it previously could not solve? The paper suggests that distillation can be effective: a larger model supplies correct answers that are then used to train a smaller model.

In Figure 6, the black line represents Qwen-7B trained through knowledge distillation with DeepSeek-R1 as the teacher model. The results suggest that RL and instruction fine-tuning have limitations that distillation can overcome.

My view

Since the shock caused by OpenAI’s o-series and DeepSeek late last year, there has been an optimistic expectation that repeatedly applying reinforcement learning to models would make them increasingly intelligent and eventually lead to AGI. These findings could be seen as a challenge to that hope.

The authors argue that we need methods that do more than raise the probability that a model expresses capabilities it already has. We need approaches that elicit novel reasoning patterns.

I am curious about how agentic RL will develop. If it incorporates simulations or real-world data, it may encounter things that conventional pre-training and post-training do not, and that could potentially produce novel reasoning patterns.

The approach of increasing pass@k is also interesting. It may be useful for identifying an upper bound on a model’s latent reasoning capability.