If we had to live with only having string compare on scalar values, it could be worked around by nesting given with in the default clause of another given:

given (+ $x) {
    when (+ $n) { ... }
    when (+ $m) { ... }
    default { given ($x) { ... } }
[download]

But, having dug in to Perl Guts more than I should have, I think that disallowing tied (and probably other magic) variables on the RHS, a built-in smart match would be able to reliably determine if the RHS is a number.