Notes to build mental models for how things work.
Why a mismatched inference engine pushes the trainer toward itself, why ratios cannot fix it cleanly, and the one-line change that does: subtract the engine’s distribution instead of the trainer’s.
Every RL-for-LLM weighting rule, from REINFORCE to DPPO, measured on a toy where the true gradient is exact: bias, variance, the cap, and why a trust region is needed.
