Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Best practices for local libraries

by Barrabas (Novice)
on Dec 07, 2019 at 04:11 UTC ( #11109789=perlquestion: print w/replies, xml ) Need Help??

Barrabas has asked for the wisdom of the Perl Monks concerning the following question:

I would inquire of the monks: What are the best practices for storing locally generated libraries?

Background: I write a lot of perl functions that are used by local programs and for obvious reasons I keep them in library files somewhere in my home directory for reuse. These are not CPAN functions, just lots of things I have written over time and that are useful to my workflow.

I do work across several machines, and some of the programs need configuration parameters (think: IP addresses, usernames, port assignments and the like) so I also had a file "~/.config/perl/SiteConfig.pm" that exposes a hash of various bits of information of that nature.

I set PERL5LIB so that any program I ran could access these libraries and everything worked well enough.

All of this went horribly wrong when I went to apt-get install something as superuser - the root login carries over the PERL5LIB from the base user, and some of my files conflicted with system ones and the entire install process got borked. (Clearing PERL5LIB in root/bashrc fixes the problem, but doesn't fix the broken installs.)

So my question is: what is the *preferred* placement and configuration for locally generated libraries and programs specific to a login?

For instance, I have a directory "Math" containing several files "EM.pm", "Levy.pm", "Log2.pm", and so on. Should I place my library dirs in a subdir "Site" so that I have to "use Site::SomeDir::SomeLib" for each library, to avoid namespace conflicts? Should I use the -I option on all shebang lines with a specific directory, or is it better to use PERL5LIB?

And is there a standard place (in my home directory) to put the directories containing the library files? For example, is .config/perl/ the right place for a shelf of library dirs?

What is "best practices" for this sort of setup?

A related question is where to put executable perl programs that are specific to my login. Is there a "bin" directory position within my home directory that's seen as standard or typical?

Replies are listed 'Best First'.
Re: Best practices for local libraries
by haukex (Bishop) on Dec 07, 2019 at 09:49 UTC
    some of my files conflicted with system ones

    Of the various things you wrote, I'd identify this as a central issue. I would suggest putting all your modules in their own namespace, for example MyProject:: or MyCompany:: - see also On The Naming of Modules, in particular, if these are modules that are always going to remain local and that you're not going to put on CPAN, you might also consider Local::.

    And is there a standard place (in my home directory) to put the directories containing the library files?

    There's More Than One Way To Do It... since you say it's just per-user, then really any place in your $HOME works, and using PERL5LIB set up in a ~/.bashrc or ~/.profile is fine (keeping in mind that you might want to append to the environment variables it if they're already set from elsewhere).

    You might have a look at local::lib, which will add ~/perl5/lib/perl5 to PERL5LIB by default. However, personally, I consider ~/perl5/lib a directory I can just clobber anytime so I can re-install everything from CPAN - so in fact, you might want to consider setting up a really basic Makefile.PL for your modules, such that you can make use of the default Perl installation mechanisms. See ExtUtils::MakeMaker - for example, consider developing your modules in some directory like say ~/dev/MyModule (which you can add to a version control system), which could have a directory layout like:

    ~/dev/MyModule/Makefile.PL ~/dev/MyModule/lib/Local/Math/EM.pm # Local::Math::EM ~/dev/MyModule/lib/Local/Math/Levy.pm # Local::Math::Levy ~/dev/MyModule/lib/Local/Math/Log2.pm # Local::Math::Log2 ~/dev/MyModule/bin/script.pl ~/dev/MyModule/t/*.t # (optional but recommended)

    And a Makefile.PL as simple as:

    use ExtUtils::MakeMaker; WriteMakefile( NAME => 'Local::Math', VERSION => '0.01', EXE_FILES => [ 'bin/script.pl' ], );

    Then, in ~/dev/MyModule/, you can just do the usual perl Makefile.PL, make, make test, make install, and with local::lib set up properly, it'll all just work; plus it makes it easy for you to expand your "distribution" with more tests, and it'll be easier to grow into a "more proper" distribution, like declaring dependencies and adding more tests. Also distributing to other machines will be as simple as:

    make distclean # if you've previously run "perl Makefile.PL" perl Makefile.PL make manifest make dist # will create a Local-Math-0.01.tar.gz for you!

    I think your idea of keeping your configuration files separately is a good one, since that's not something you'd want to keep in a "distribution" like the above. You could also have a look at the Config:: namespace on CPAN for plenty of pre-made solutions.

    A related question is where to put executable perl programs that are specific to my login. Is there a "bin" directory position within my home directory that's seen as standard or typical?

    ~/bin is added to PATH as part of the default ~/.profile on Debian-based systems, that's usually what I use for site-local stuff (Update: I probably should have said "user-local" stuff). Or, local::lib sets up ~/perl5/bin, which with the above suggestion will also "just work".

Re: Best practices for local libraries
by misc (Friar) on Dec 07, 2019 at 10:19 UTC
    >some of my files conflicted with system ones

    That's the reason I keep all my own modules within the namespace "Misc";
    and the physical location is /home/micha/prog/perl/Misc,
    having a symbolic link in /etc/perl/Misc.

    I don't know if there's a general preferred way.
    But this did work out for me.

    Having a unmodified homebrew perl installation on osx darwin here, I just looked the include path up:

    ~$ perl -e 'print map { "$_\n" } @INC' /usr/local/Cellar/perl/5.26.2/lib/perl5/site_perl/5.26.2/darwin-thread +-multi-2level /usr/local/Cellar/perl/5.26.2/lib/perl5/site_perl/5.26.2 /usr/local/Cellar/perl/5.26.2/lib/perl5/5.26.2/darwin-thread-multi-2le +vel /usr/local/Cellar/perl/5.26.2/lib/perl5/5.26.2 /usr/local/lib/perl5/site_perl/5.26.2
    So, I might prepend a BEGIN block and push the local path to @INC.

    ..
    I just looked it up, here's what I did in a script some time before:

    BEGIN{ use File::Basename; ($name,$path,$suffix) = fileparse ($0); print "path: $path"; push @INC, "$path/perl"; }

    I guess, there I followed the logic: Wherever this script lives, I'm going to store my modules within the same directory.

    In each case I'd recommend working with links to your (home) modules directory,
    just to keep things more tidy.

    And I'm utilizing github for storing backups / revision control.

    Finally, into some scripts I copied every module needed, also cpan modules, getting a huge single script.

    These are scripts I can execute nearly everywhere, without any cpan installation.

    I even built a statically linked perl out of this reason.
    hth, Michael
      So, I might prepend a BEGIN block and push the local path to @INC. I just looked it up, here's what I did in a script some time before: ...

      I'd suggest using the lib pragma (which unshifts instead of pushing) in combination with FindBin instead; $0 can in some cases be unreliable, and the code is shorter. So if there are .pm files in the same path as the script: use FindBin; use lib $FindBin::Bin;.

      Finally, into some scripts I copied every module needed, also cpan modules, getting a huge single script.

      See fatpack and pp, although with some modules they have issues.

        Thanks a lot, that's good news.
        You remember me why I married perl a long time ago ;)
Re: Best practices for local libraries
by eyepopslikeamosquito (Bishop) on Dec 08, 2019 at 10:46 UTC

    I have a directory "Math" containing several files "EM.pm", "Levy.pm", "Log2.pm", and so on. Should I place my library dirs in a subdir "Site" so that I have to "use Site::SomeDir::SomeLib" for each library, to avoid namespace conflicts?
    Yes, as you've discovered, avoiding namespace conflicts is crucial. Math was unfortunate because it clashed with a heavily used CPAN top level namespace. You need to choose a top level namespace for all your non-CPAN works that is unlikely to clash with a CPAN or core Perl module. I'd look for a name that is short, descriptive of your organisation, and easy to search for. "Site" is a bit too generic for me - searching CPAN for "Site" just now returned 2457 results, while "Barrabas" returned just one (an obscure module written by one Rajstennaj Barrabas :).

    What are the best practices for storing locally generated libraries?

    Perl provides a lot of flexibility, there are many good ways to do it. Some excellent free and general advice can be found at Modern Perl:

    You may choose never to release any of your code as public CPAN distributions, but you can use CPAN tools and conventions to manage even private code. The Perl community has built amazing infrastructure. Take advantage of it.
    For more detail on this amazing infrastructure and some specific tools, see Modern Perl's "CPAN Tools for Managing Distributions" sub-section.

    It's important to distinguish between a module and a distribution - again see Modern Perl for details, especially the "Designing Distributions" sub-section. Note that a distribution contains a unique name and single version number (often taken from its primary module). Instead of lumping all your modules into a single monolithic distribution, consider breaking them into a set of (highly cohesive, loosely coupled) distributions, all under the same top level namespace. Your comprehensive set of "Math" modules, for example, looks like a natural choice for a distribution.

    The most important aspect of any module is not how it implements the facilities it provides, but the way in which it provides those facilities in the first place.

    -- Damian Conway in Ten Essential Development Practices

    Oh, and don't forget about the importance of getting your interfaces right. A couple of related Perl Monks nodes I wrote a while ago:

Re: Best practices for local libraries
by karlgoethebier (Abbot) on Dec 07, 2019 at 19:55 UTC

    This is the way i do/did it: ~/lib for my libs and ~/bin for my executables. /usr/local/lib and /usr/local/bin for the stuff to share with other users - sometimes. Different namespaces like Company::Project or KGB::Gorgeous. Extensive usage of perlbrew. Using a versioning system may simplify things. Regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Re: Best practices for local libraries
by NERDVANA (Beadle) on Dec 08, 2019 at 02:13 UTC

    Echoing some other ideas here, I recommend putting all your utility code into modules in your own namespace (in a version-controlled directory) and then:

    1. Quick and dirty option: add a symlink to your modules into an existing perl include path
    2. More robust option: use Dist::Zilla to build actual packages of your modules, which you might even decide to publish on cpan

    The first option will get you up and running the quickest, and even help transfer your modules around (along with bug fixes for your modules) from computer to computer with the help of git. You use the existing perl lib path, so no need to clutter your scripts with "use lib ..." boilerplate. If package management looses your symlink you can put it back easily. You can also add your modules to every lib path of every perl you use while only maintaining one copy.

    Later, when you find time to learn Dist::Zilla and clean up your APIs, you can make them into official perl packages which you can install into system-perl or into per-user perlbrew using "cpanm MyPackage-version.tar.gz".

      Quick and dirty option: add a symlink to your modules into an existing perl include path
      The OP mentions using apt-get install, which strongly implies he's using a Debian or Debian-derived system. A default Debian install will have /usr/local/lib/site_perl in @INC for precisely this purpose. But this does, obviously, make any modules linked there available to all users on the machine rather than user-specific, which may or may not be acceptable to the OP.

      Really, though, the only real problem I see in the OP is the bit about the user's $PERL5LIB carrying over to root when using su. My solution to that is, quite simply, to use sudo instead, as it does not carry over the original user's environment:

      me@host:~$ export FOO=bar me@host:~$ echo $FOO bar me@host:~$ sudo -i [sudo] password for me: root@host:~# echo $FOO root@host:~# exit logout me@host:~$ sudo bash root@host:/home/me# echo $FOO root@host:/home/me#
      If you prefer su over sudo, you should be able to get the same effect by using su - or su --login to open a login shell with a fresh environment instead of preserving the previous shell's environment. From the Debian 10 version of man su:
             For backward compatibility, su defaults to not change the  current  di‐
             rectory  and to only set the environment variables HOME and SHELL (plus
             USER and LOGNAME if the target user is not root).  It is recommended to
             always use the --login option (instead of its shortcut -) to avoid side
             effects caused by mixing environments.
      
      and, from the Debian 8/9 versions of the man page:
                 Note that the default behavior for the environment is the
                 following:
      
                     The $HOME, $SHELL, $USER, $LOGNAME, $PATH, and $IFS environment
                     variables are reset.
      
                     If --login is not used, the environment is copied, except for
                     the variables above.
      
                     If --login is used, the $TERM, $COLORTERM, $DISPLAY, and
                     $XAUTHORITY environment variables are copied if they were set.
      
                     Other environments might be set by PAM modules.
      
      So just add --login to your su command and $PERL5LIB won't be carried over to your root shell.
Re: Best practices for local libraries
by Anonymous Monk on Dec 10, 2019 at 03:54 UTC
    But unfortunately for all of us ... the authors of Linux distro packages still possess a rather-imperial capacity to screw us if they do not know – or care – what they are doing.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11109789]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (7)
As of 2021-04-19 16:09 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?