A solution using a Guttman Rosler Transform which might be more efficient if there are a large number of rectangles.

use strict;
use warnings;

use feature qw{ say };

my @x     = ( 3, 2, 4, 0, 3, 0, 4, 0, 3, 2, 0, 4 );
my @y     = ( 0, 0, 0, 1, 1, 0, 3, 3, 3, 3, 2, 2 );

my @rects =
   map { [ split m{:}, substr $_, 8 ] }
   sort
   map { my $packed
         = ~ ( pack q{N}, $x[ $_ ] )
         . pack q{Na*}, $y[ $_ ], join( q{:}, $x[ $_ ], $y[ $_ ] )
      }
   0 .. $#x;

my $n = 0;
say qq{@{[ sprintf q{%2d}, $n ++ ] }: @$_} for @rects;
[download]

The output.

 0: 4 0
 1: 4 2
 2: 4 3
 3: 3 0
 4: 3 1
 5: 3 3
 6: 2 0
 7: 2 3
 8: 0 0
 9: 0 1
10: 0 2
11: 0 3
[download]

I hope this is of interest.

Update: Simplified decoration/de-decoration removing need for join and split.

use strict;
use warnings;

use feature qw{ say };

my @x     = ( 3, 2, 4, 0, 3, 0, 4, 0, 3, 2, 0, 4 );
my @y     = ( 0, 0, 0, 1, 1, 0, 3, 3, 3, 3, 2, 2 );

my @rects =
   map { [ unpack( q{N}, ~ $_ ), unpack( q{x4N}, $_ ) ] }
   sort
   map { my $packed
         = ~ ( pack q{N}, $x[ $_ ] )
         . pack q{N}, $y[ $_ ]
      }
   0 .. $#x;

my $n = 0;
say qq{@{[ sprintf q{%2d}, $n ++ ] }: @$_} for @rects;
[download]