PPO Entropy Collapse

PPO adds a clipped surrogate objective to keep policy updates from moving too far in one step. That guardrail is helpful, but it does not guarantee healthy exploration.

The clipped objective is:

$$L^{CLIP}(\theta) = \mathbb{E}_t[ \min(r_t(\theta)\hat{A}_t, \mathrm{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t) ]$$

where $r_t(\theta)$ is the probability ratio between the new and old policies.

Collapse Signals #

Watch for these together:

• Entropy falls quickly while reward is still noisy.
• KL stays low because the clipped update is limiting movement.
• Advantage estimates become high variance around a narrow behavior mode.

Entropy bonuses help, but they are not magic. I usually want dashboards that plot entropy, KL, clip fraction, value loss, and reward on the same time axis.

Practical Knobs #

Lowering the learning rate, normalizing advantages, increasing rollout diversity, and tuning the entropy coefficient can all help. The important bit is to notice collapse early, before the policy turns a transient reward pocket into a habit.