I wonder how hard would it be to create a lexical version of the lib pragma.

Short answer: very hard, since restoring @INC from a lexical must be done at compile time. But without a BEGIN block, this lexical is set at runtime. So Padwalker maybe, or some wizardry which works on the code which is currently being compiled - if such a beast exists at all.

I would go the route of the if module:

use lib::lexical MODULE, PATHSPEC;
[download]

or even

use lib::lexical modules => MODULESPEC, path => PATHSPEC;
[download]

without modifying @INC at all in the calling code, and leave that to the module loader lib::lexical, which localizes @INC. So, for the first usage, something like this:

package lib::lexical;
our @SAVEINC = @INC;
our @LOCALINC = @INC;

sub work {
  my $flag = shift();
  my $method = $flag ? 'import' : 'unimport';
  my $p = shift;                # PACKAGE
  if (! $p and ! $flag) { # plain 'no lib::lexical;'
    @LOCALINC = @SAVEINC unless $p; # restore to original @INC
    return;
  }
  if ($flag) { # prepend @paths to @INC
    unshift @LOCALINC, @_;
  } else {     # weed out @paths from @INC
    my %i;
    @i{@_} = (1) x @_;
    @LOCALINC = grep { ! $i{$_} } @LOCALINC;
  }
  local @INC = @LOCALINC;
  warn $_,$/ for @INC;
  (my $file = "$p.pm") =~ s!::!/!g;
  require $file;
  my $m = $p->can($method);
  goto &$m if $m;
}

sub import   { shift; unshift @_, 1; goto &work }
sub unimport { shift; unshift @_, 0; goto &work }

1;
__END__
[download]

Wait, what - this is (roughly) just what the lib module does? Yes of course! But I'd like to have

no lib;
[download]

just restore the original @INC, so lib::unimport should be:

sub unimport {
    shift;
    @INC = @ORIG_INC and return unless @_; # patch for the lazy <--- H
+ERE
    my %names;
    foreach my $path (@_) {
        my($arch_auto_dir, $arch_dir, $version_dir, $version_arch_dir)
            = _get_dirs($path);
        ++$names{$path};
        ++$names{$arch_dir}         if -d $arch_auto_dir;
        ++$names{$version_dir}      if -d $version_dir;
        ++$names{$version_arch_dir} if -d $version_arch_dir;
    }   
    
    # Remove ALL instances of each named directory.
    @INC = grep { !exists $names{$_} } @INC;
    return;
}
[download]

And then, the lib module modifies @INC globally, whereas the above module just uses a modified @INC for one use call (and the use calls which this call triggers)/.