Temperature and drift

Two identical prompts, two different answers, and the dial that explains it is temperature. It's one of the few knobs you actually get, and understanding it tells you when to expect consistency and when to expect a model to wander.

Temperature controls how sharp the model's probability distribution is when it picks the next token. At zero, it's deterministic: it always takes the single highest-probability token, so the same prompt gives you the same answer every time. At one, you get the raw distribution, real randomness. Push above one and you flatten the distribution toward chaos, where low-probability tokens start getting picked and the output gets weird.

Why does a little randomness turn into a lot of drift?

Because temperature compounds across tokens. This is the part people miss. A five-hundred-token answer is five hundred dice rolls, not one. Even a small amount of randomness per token accumulates, and by the end of a long generation the output can be a long way from where a lower-temperature run would have landed. The wander isn't in any single token. It's in the stacking.

The way I hold it is in trading terms. Temperature zero is a stablecoin, pegged, predictable, boring on purpose. Temperature one is a mid-cap, real movement, still tradable. Temperature one and a half is a shitcoin, and you're along for whatever ride it decides to take. None of those is wrong. They're for different jobs.

When do you want which?

Anything that has to be consistent or exact wants low temperature. Structured output, anything touching money or format, anything you'll parse downstream, keep it cold so it doesn't roll a bad die on token four hundred. Anything creative, where you want range and surprise, can take the heat. And remember the compounding: the longer the output, the more a warm temperature will drift, so long generations are exactly where low temperature earns its keep.

The takeaway: temperature is a randomness dial, and its effect stacks with length. Cold for exact and long, warm for creative and short.