Have you ever gotten completely unfollowable instructions from someone about how to get to a place they know? I would suspect that lack of attention to detail while describing an algorithm would apply both to giving directions to a location and to writing a program. Hence, those people would probably also be bad programmers.

To be a good programmer, you have to consciously and reflexively know how to solve a problem (or at least the meta-problem), know how to reduce it to things that the language can represent, and then construct a representation of the solution steps in that language. Most non-programmers I know cannot do one or more of those steps with any precision.

It's amazing how many of us programmers take those skills for granted. Yeah, we think "well of course, everyone can do what I do easily!" But I have good counterexamples: I cannot remember pictures, nor can I construct good art. Yeah, it seems most people can do both of those, but then those people also take those skills for granted. So I'm not so naive as to presume that the skills I take for granted are also universal.