I agree with that approach. Specifying both LIB and PREFIX works very well for me. I use install_base for Module::Build and anything that insists on it.

All modules are now installed in the given directory instead of architecture-dependant ones and such.

That's not strictly true. Pure perl modules will be installed generically to the lib directory. Modules containing XS (or Inline::C, swig, etc.) are still installed into architecture dependent paths, but this is what you want. Indeed, you can share the lib tree across platforms - I do this between Solaris and Windows, and it works too!