I think you can say

BEGIN {
package Y;
$VERSION = 0.1;
1;
package X;
if ( $Y::VERSION < 1 ) {
  # How to enable use bytes for the whole modue X?
  # with condition $Y::VERSION in mind?

  ## Not use()'d anymore.
  # use bytes;
  require bytes;
}
[download]

And this will do what you mean.

Note: this is untested code, as always.