Better to declare them in httpd.conf or a startup.pl type script.
# httpd.conf
PerlModule Date::Manip
# or startup.pl
use Date::Manip ();
This allows them to be cached on server boot and shared between child processes. The () after the module in startup.pl is very important as it means the module won't polute your global namespace (exports nothing).
This gives you the performance advantage of having the module already compiled and cached, while keeping your memory requirements (per child) down. Make sure to still use the module in your hanlder/script though; while not always nesseccary it gets pretty confusing if you don't.
If you're running Apache::Registry or PerlRun the rules are slightly different. Check the guide for more info.