The Difficulty of LLM Post-Training

The recent ChatGPT sycophancy incident was explained as a case in which user feedback had been reflected too strongly during post-training. It reminded me of an interesting tweet:

“GPT model stopped speaking Croatian / Nobody could figure out why. Turns out / Croatian users were much more prone [to] downvote messages.”

There was also a phenomenon during the final training of the o3 model where, at some point, it began spelling “check” as the British “cheque.” The reason is unknown.

When Feedback Changes Behavior

In the early days of ChatGPT, the model reportedly stopped speaking Croatian. When the reason was investigated, it turned out that Croatian users tended to press the negative-feedback, or downvote, button more often for ChatGPT’s outputs.

From the model’s perspective, answering in other languages did not produce the same pattern, but answering in Croatian repeatedly brought negative feedback. So it was trained in the direction of avoiding that language. It is a rather funny example, but also a revealing one.

The Risk of Organized Feedback

It made me think that a large fandom could potentially influence a model’s behavior through coordinated voting. For example, if people wanted to encourage positive statements about BTS and consistently gave positive feedback when the model did so, enough feedback might eventually lead to a meaningful behavioral change.

In reinforcement learning, there are many examples where poorly designed rewards or reward hacking produce outcomes nobody anticipated. Thinking about those cases, LLM training—especially post-training—seems very difficult.

The Value of Real-World Feedback

User feedback is likely highly valuable as real-world data, similar to post-market surveillance in drug development. I am curious about how it can be incorporated more effectively in the future.

There have also been announcements about improving ways to collect and incorporate feedback more broadly. I plan to follow this with interest.