These are all good points, thanks. If it were up to me, I wouldn't bother with a written test, but instead would be comfortable with a whiteboard-based discussion, as various people have suggested. Unfortunately, I am but a grunt, and the CTO wants to give a test. Thus: we'll give a test.

And yes, all the skills you mention (learning, working with others, thorough, disciplined, etc) are indeed very important. The test is by no means the only (or even the hightest weighted) part of our evaluation process. There are other parts of the interview and screening that try to focus on that.

A couple weeks ago, we brought in a candidate who looked good on paper, sounded fine on the phone, and sent in a very impressive-looking code sample. But when he got here, he couldn't explain the code he'd sent very well, he was pretty bad at "what does this code do", and he made some of the people he met feel unconfortable. So no test was required, since we wouldn't have hired him regardless of how well he performed.