Everybody has an optimal level of stimulation. Most people have a very strong preference for achieving this level of stimulation. [...]

Some programming tasks are quite taxing on my mental abilities and so I have to do them in the most peaceful of settings. At the other extreme are programming tasks that are only taxing of my patience and thus a beer can be helpful.

I think this is incomplete.

A point Marvin Minsky has made is that investigation into "intelligence" is a spurious chase as long as you think of it in the singular. The contention is that what we perceive from the outside as "intelligence" is really a huge bucket of completely unrelated mechanisms for totally different tasks, and what we perceive is an emergent property of all of them acting in semi-random configurations on the tidbits we feed them.

Now, if you disagree with that in broad, there's no point reading further. But if you go with it, I think it expands the understanding of the issue here. Your "mental abilities" are no more singular than "intelligence" is. Nothing is ever taxing on your mental abilities; it's taxing on some mental ability, or realistically, on some combination of your mental abilities. The rest of them are just sort of floating around twiddling their thumbs.

That's where distraction comes from; those pieces that aren't being used are still turning over, waiting for something to grab onto. If I don't need my internet connection, I can unplug my router. If I don't need music, I can unplug speakers. Don't need to know the time, unplug the clock. But I can't unplug pieces of my mind. The mind commands the body and it obeys; the mind orders itself and meets resistance.

There are things I've been unable to make progress on without music playing. Sometimes, very loudly. That's not because the music is helping me think; it's because the music is helping me not think, with those mechanisms that aren't involved in what I'm trying to do, but are still searching for input somewhere. I keep them busy so they don't go seeking out stimulation elsewhere and drag me away. It's actually the same problem you solve when you work in silence; eliminating distractions. It's just that in that case, you're eliminating external distractions, while in the other, you're harnessing external distractions to eliminate internal distractions.

I'm a twitchy sort; I have to have things on my desk to fiddle with. There's a pen on my keyboard I get to clicking; a folding knife I'll open and close and twirl. I've had a mini-Slinky that got flung around. I've even got a Ball of Whacks. It claims to be a "creativity tool" and a "creative stimulant". But that's crap. Fiddling with it doesn't lead me to new ideas; it keeps my hands and part of my mind busy, so the rest of me can ignore those bits of me and think.

Heck, it's the same thing you do with kids; you give 'em something harmless (well, as harmless as anything is in the hands of a small destruction machine :) to do to keep them out of your way while you try to do something else. All your mental tools are like little children in that way; if you don't feed them, they'll scream and yell until you do, so if you want to be able to get anything done they need to be either involved in it (and some problem do involve nearly everything you can bring to bear), or otherwise appeased. And indeed, chemically-induced altered states can function the same way as these other distractions, at least for some people.

Well, except most jurisdictions frown on applying them to your kids...