Reward Hacking

Reward Hacking is exactly what is sounds like, learning a surrogate way to gain reward which doesn’t co-relate with the expectation. The title of the article is do AI models cheat and which shall see what that means and some scenarios where this could be dangerous.

AI models are getting larger and smarter and by some estimates smart enough to replace humans at certain jobs? The debate of whether we will get there in the next 5 years or the 50 years is not worth to me but what is worth is as AI models gains human intelligence at certain task can they also gain sub-conciousness? Can they perform some hidden agenda, like saying something but doing something else underneath.

A large part of learning incorporated to AI models are based on the input data. If the input data incorporates evidence of violence, toxicity or abuse, the model will learn that and can go on to act in that manner as well if we don’t perform some fine tuning or add guard rails to limit the concerns.

One way to perform some fine tuning is incorporating human feedback into the training process. Eventually the models learns to steer itself towards human preference of non-violent, toxic or threatful behavior. But one side effect of this is it also learns to be more adherent to the end users as the reward model can dominate training. This could be an example of inner mis-alignment or syncophancy.

Other way where models can perform un-interesting work, is when the evaluation function or the cost function in ML language is not robust enough. Think of the scenario where ML model is built for coding. The cost function could include elements like not having buggy code or reducing the number of failures in the code. Now think of a scenario where you are debugging failing tests cases. A bad behavior from AI model while still justifying the cost function could be just removing the failing test, that way you do get a 100% success rate over your tests but obbiously that’s not what you wanted.