Some opinions:

Should I start building test/benchmark code into my future scripts for debugging purposes..?

As a general rule, no. It makes your code messy, and you'll do a lot of work that you may not use at all.

What do other monks do for benchmarking/testing their scripts?

print statements (sometimes in combination with YAML - never with Data::Dumper). make test. I seldomly benchmark programs I write, but I do benchmark theoretical cases.

When do you reach the point of "OK, this has too many tests"?

With about 150,000 tests in Regexp::Common, I say "never".

What other metrics do you measure (I.e. speed, memory)?

Several others, of which the most important is I/O. With modern hardware, I/O is often a bigger bottleneck than CPU speed or memory. But of course, it depends on the problem and what else is going on on the box.

Abigail