Some shortcuts:

• Carefuly construct your list of dividers: there is no need to check division by a number higher than 1/2 of the largest formed number. This means that you first construct your list of formed numbers and then your list of dividers.
• If one of the formed numbers is prime, all other formed numbers must be multiples of that prime number. Of course checking if a number is prime is very time-consuming, but if you check for a 3x3 matrix, the largest prime number should be less than 999/5 (since the smallest formed number is prime, the sixth number must be 5 x the smallest number). All prime numbers above that can be skipped. So by keeping a list of "forbidden prime numbers" you can avoid these.
• You can also skip all multiples of 10 as common divisors, since all multiples of 10 end in (at least) one zero and this must mean that the formed number of the last column must start with a zero, which is not allowed.

CountZero

"If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law