Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Collapsing smaller scripts into a larger one, request for comment

by Lady_Aleena (Curate)
on May 29, 2020 at 17:35 UTC ( #11117478=perlquestion: print w/replies, xml ) Need Help??

Lady_Aleena has asked for the wisdom of the Perl Monks concerning the following question:

I have been trying to decide for weeks whether or not I want to collapse most of my individual page-scripts (scripts that act like web pages) into their indexes and use a touch of CGI:Minimal to switch the text on a click. Below are the page-scripts for my Collections where you can see how similar they are above the __DATA__.

Link to all of the Collections; links to the modules: Base::Page (page and story), Util::StoryMagic, Util::StoryMagic::Collection.

index.pl
#!/usr/bin/perl # This is the index for Collections. use strict; use warnings FATAL => qw( all ); use CGI::Carp qw(fatalsToBrowser); use lib '../files/lib'; use Base::Page qw(page story); use Util::StoryMagic::Collection qw(collection_magic); my $magic = collection_magic; page( 'code' => sub { story(*DATA, { 'line magic' => $magic }) }); __DATA__ Welcome to Lady Aleena's B<collections>, which is lists of A<novels|hr +ef="Fiction.pl">, A<books|href="Non-fiction.pl">, A<music|href="Music +_and_comedy.pl">, A<movies|href="Movies.pl">, A<tie-ins|href="Tie-ins +.pl">, and A<programs|href="Programs.pl"> I am willing to admit I own + or use. The list of movies here is just those movies I own and shoul +d not to be confused with my more general interest in A<movies|href=" +../Movies">. Tie-ins are books and music connected to movies or telev +ision series. I also share my A<fandom|href="../Fandom"> elsewhere. Here is a key for the notations after each title with the exception of + programs.
Fiction.pl
#!/usr/bin/perl use strict; use warnings FATAL => qw( all ); use CGI::Carp qw(fatalsToBrowser); use lib '../files/lib'; use Base::Page qw(page story); use Util::StoryMagic::Collection qw(collection_magic); my $magic = collection_magic; page( 'code' => sub { story(*DATA, { 'doc magic' => $magic, 'line magi +c' => $magic }) }); __DATA__ This is my B<fiction collection> of SPAN<hardcovers|^hardcovers^>, SPA +N<trade paperbacks|^trades^>, and SPAN<mass market paperbacks|^massma +rkets^>.
Movies.pl
#!/usr/bin/perl use strict; use warnings FATAL => qw( all ); use CGI::Carp qw(fatalsToBrowser); use lib '../files/lib'; use Base::Page qw(page story); use Util::StoryMagic::Collection qw(collection_magic); my $magic = collection_magic; page( 'code' => sub { story(*DATA, { 'doc magic' => $magic, 'line magi +c' => $magic }) }); __DATA__ This is my B<movie collection> of SPAN<blu-rays|^brds^>, SPAN<DVDs|^dv +ds^>, and SPAN<VHSs|^vhss^>.
Music_and_comedy.pl
#!/usr/bin/perl use strict; use warnings FATAL => qw( all ); use CGI::Carp qw(fatalsToBrowser); use lib '../files/lib'; use Base::Page qw(page story); use Util::StoryMagic::Collection qw(collection_magic); my $magic = collection_magic; page( 'code' => sub { story(*DATA, { 'doc magic' => $magic, 'line magi +c' => $magic }) }); __DATA__ This is my B<music and comedy collection> of SPAN<LPs|^lps^>, SPAN<45s +|^ffs^>, SPAN<cassettes|^cassettes^>, and SPAN<CDs|^cds^>. If it is n +ot on a CD, I have not heard it in a long long time. SPAN<&#9785;|cla +ss="bigger">
Programs.pl
#!/usr/bin/perl use strict; use warnings FATAL => qw( all ); use CGI::Carp qw(fatalsToBrowser); use lib '../files/lib'; use Base::Page qw(page story); use Util::StoryMagic qw(program_magic); my $magic = program_magic; page( 'code' => sub { story(*DATA, { 'line magic' => $magic }) }); __DATA__ This is a list of B<programs> that I am using or have used. I can not +account for I<all> the software we have had and used over the years. +Some of it was so bad, we blanked it out of our heads. This list does + not include a full list of hardware drivers either. So much software +, so little time or in this case patience.
Tie-ins.pl
#!/usr/bin/perl use strict; use warnings FATAL => qw( all ); use CGI::Carp qw(fatalsToBrowser); use lib '../files/lib'; use Base::Page qw(page story); use Util::StoryMagic::Collection qw(collection_magic); my $magic = collection_magic; page( 'code' => sub { story(*DATA, { 'line magic' => $magic }) }); __DATA__ This is my film and television B<tie-in collection>. For books: SPAN<hardcovers|^hardcovers^>, SPAN<trade paperbacks|^trade +s^>, and SPAN<mass market paperbacks|^massmarkets^>. For music: SPAN<CDs|^cds^>, SPAN<cassettes|^cassettes^>, SPAN<45s|^ffs +^>, and SPAN<LPs|^lps^>.

With the above being so similar, I can collapse them into one umbrella script that is below.

#!/usr/bin/perl # This is the index for Collections. use strict; use warnings FATAL => qw( all ); use CGI::Carp qw(fatalsToBrowser); use CGI::Minimal; use HTML::Entities qw(encode_entities); use lib '../files/lib'; use Base::Page qw(page story); use Util::Data qw(file_directory file_list); use Util::StoryMagic qw(program_magic); use Util::StoryMagic::Collection qw(collection_magic); my $cgi = CGI::Minimal->new; my $collection = $cgi->param('collection') ? encode_entities($cg +i->param('collection'),'<>"') : undef; my $magic = $collection && $collection eq 'Program' ? progr +am_magic : collection_magic; my $collections_dir = file_directory('Collections'); my @collections_list = file_list($collections_dir); my @collections = map { $_ =~ s/\.txt//; $_ =~ s/_/ /g; $_ } grep {/^\p{uppercase}/} @collections_list; my $heading = q(Lady Aleena's ); $heading .= $collection && grep( $collection =~ /$_/, @coll +ections ) ? lc "$collection collection" : 'collections'; my $collection_file = $collection ? "$collections_dir/$collection.txt +" : "$collections_dir/index.txt"; $collection_file =~ s/ /_/g; open(my $collection_fh, '<', $collection_file) || die $!; page( 'heading' => $heading, 'code' => sub { story($collection_fh, { 'doc magic' => $magic, 'line magic' => $magic }); } );

(file_directory and file_list are in Util::Data in the above and in the sub below.)

The above script works as expected, but I would not be posting here if there were not a problem. The problem is my site's menu is built by recursing through the directory structure of my site and uses the files and directories it finds to make the menu. If I were to collapse the 155 page-scripts into the 25 index scripts, I would lose the links to the topics those separate page-scripts now have. I did something very close to this a while ago, and I have not liked that the directory does not have entries for each topic. That directory may not get any traffic because it appears that there is nothing much there.

Side note, I did not realize I had so many page-scripts.

I use my base_menu sub (below) to make the menu for my site and would like to know how you would populate the directories that have only one file in them. With base_menu, all I have to do is add a page-script and refresh, and the menu is updated. Without those page-scripts, the menu will be almost empty.

sub base_menu { my %opt = @_; my $directory = $opt{'directory'}; my $curr_cwd = cwd; my @contents = file_list($directory); @contents = grep {/^\p{uppercase}/} @contents if (!$opt{'full'} || $ +opt{'full'} !~ /^[yt1]/); # Thank you [tye]! # Thank you [davido]! my $sub = $directory =~ /(Other_poets|Player_characters|Spellbooks)$ +/ ? \&name_sort : \&article_sort; @contents = sort { $sub->($a, $b, { 'misc' => $opt{'misc'} }) } @con +tents; my (@files, @directories); for my $content (@contents) { my $long_content = "$directory/$content"; my $link = File::Spec->abs2rel($long_content); my $text = $content !~ /^\./ ? textify($content) : $content; if (-f $long_content) { my $active = realpath($0) eq $long_content ? 'active' : 'inactiv +e'; my $color = $opt{'color'} == 1 ? link_color($content,1) : undef +; my $inlist = $active eq 'active' && $opt{'file menu'} ? ['u', $o +pt{'file menu'}, { 'class' => 'sub_menu' }] : undef; $active .= $active eq 'active' && $opt{'file menu'} ? ' open' +: ''; if (-M $long_content < 3) { $active .= ' updated'; } push @files, [anchor($text, { 'href' => $link, 'title' => $text, + 'style' => $color }), { 'class' => $active, 'inlist' => $inlist } ]; } if (-d $long_content) { my $active = $curr_cwd =~ /$long_content/ ? 'open active' : 'clo +sed inactive'; my $file_list; my $index = "$long_content/index.pl"; if (-e $index) { $link .= "/index.pl"; my $color = $opt{'color'} == 1 ? link_color($link,1) : undef; $text = anchor($text, { 'href' => $link, 'title' => $text, 'st +yle' => $color }); $file_list = "$curr_cwd/$0" =~ /$index/ && $active =~ / active +$/ && $opt{'file menu'} ? $opt{'file menu'} : undef; } my $next_list = base_menu( 'directory' => $long_content, 'color' => $opt{'color'}, 'full' => $opt{'full'}, 'misc' => $opt{'misc'}, 'file menu' => $opt{'file menu'} ); push @$next_list, @$file_list if $file_list; my $inlist = $next_list ? ['u', $next_list] : undef; $active =~ s/^(?:open|closed) // if !$inlist; push @directories, [$text, { 'class' => $active, 'inlist' => $in +list}]; } } my @file_lines = (@files, @directories); return @file_lines > 0 ? \@file_lines : undef; }

(base_menu is in Util::Menu, name_sort and article_sort are in Util::Sort, textify is in Util::Convert.)

So, why did I post this? I would like to know if this way appears better? I also need ideas on how to collapse these page-scripts into their indexes without losing the automatic addition to the menu that I have now. There are a few scripts that do not use story, and those are the hardest to figure out how to add to the menu since they will not have a text file to go with it.

Thank you for reading this latest bit of my craziness.

There is an update below.

My OS is Debian 10 (Buster); my perl versions are 5.28.1 local and 5.8.8 on web host.

Version control is a non-issue, I do not use it.

No matter how hysterical I get, my problems are not time sensitive. So, relax, have a cookie, and a very nice day!
Lady Aleena

Replies are listed 'Best First'.
Re: Collapsing smaller scripts into a larger one, request for comment
by duelafn (Vicar) on May 29, 2020 at 22:56 UTC

    Be careful with opening files from CGI parameters. collection=../../../../../../../home/aleena/passwords would reveal a bit more than you intended ("passwords.txt" in your home directory).

    Good Day,
        Dean

      Thank you for bringing this to my attention. If I ever store passwords somewhere, I will make sure to keep them out of the $collections_dir. I would probably put them in some deep dark corner of my directory structure with a name that does not look anything like the word 'password' and use something more secure than a plain text file.

      I could also add / and . to the encode entities to make sure that the string that the cgi param returns will not recurse. Adding those would make your string return the following.

      &#46;&#46;&#47;&#46;&#46;&#47;&#46;&#46;&#47;&#46;&#46;&#47;&#46;&#46; +&#47;&#46;&#46;&#47;&#46;&#46;&#47;home&#47;aleena&#47;passwords

      I will give that serious thought. Again, thank you.

      Update: Forward slashes will be html encoded.

      my $collection = $cgi->param('collection') ? encode_entities($cg +i->param('collection'),'/<>"') : undef;

      My OS is Debian 10 (Buster); my perl versions are 5.28.1 local and 5.8.8 on web host.

      Version control is a non-issue, I do not use it.

      No matter how hysterical I get, my problems are not time sensitive. So, relax, have a cookie, and a very nice day!
      Lady Aleena

        If it were me, I'd use the list of valid collections that you already have (assuming I understand your data correctly). Something like:

        # Note: Still has underscsores: my @collections = map { $_ =~ s/\.txt//; $_ } grep {/^\p{uppercase}/} @collections_list; # $collection guaranteed to match one of your file names. my $collection_raw = $cgi->param('collection'); my ($collection) = grep { $_ eq $collection_raw } @collections; $collection //= 'Deafult'; # If your file names might have <>", you can still do this: $collection = encode_entities($collection); # Without underscores my @collection_labels = map { $_ =~ s/_/ /g; $_ } @collections;

        Generally, it is safest to do something like this before using $collections for anything.

        Good Day,
            Dean

Re: Collapsing smaller scripts into a larger one, request for comment
by jcb (Vicar) on May 29, 2020 at 23:42 UTC

    How is that a problem? You seem to still be storing one file per page, just with a .txt suffix instead of a .pl suffix, unless I misunderstand. Another option is to ensure that the list of valid collections is stored in some recognizable way at each index, and have the menu builder parse the indexes.

      Here is the difference:

      ./Collections # is where scripts live and is where base_men +u looks for menu items ./files/data/Collections # is where data lives and is not where base_m +enu looks for menu items

      Taking the data outside of individual scripts means I need to figure out how I am going to rewrite base_menu to find items to add to my site menu that will work for all the index files (see below).

      ./Collections/index.pl ./Fandom/index.pl ./Fandom/Crossovers/index.pl ./Fandom/Crossovers/Big_fake_companies/index.pl ./Fandom/Crossovers/Westphall_crossovers/index.pl ./Fandom/Fictional_family_trees/index.pl ./Fandom/The_Riftwar/index.pl ./Fandom/Xanth/index.pl ./Miscellany/index.pl ./Miscellany/Geeky_thoughts/index.pl ./Miscellany/Political_opinions/index.pl ./Movies/index.pl ./Role_playing/index.pl ./Role_playing/Locations/Afma/index.pl ./Role_playing/Locations/index.pl ./Role_playing/Magic_items/index.pl ./Role_playing/Miscellany/index.pl ./Role_playing/Monsters/index.pl ./Role_playing/Player_characters/index.pl ./Role_playing/Reference_tables/index.pl ./Writing/index.pl ./Writing/Erotic_fiction/index.pl ./Writing/Fan_fiction/index.pl ./Writing/Poetry/index.pl

      The data for those indexes will be in the following.

      ./files/data/Collections/ ./files/data/Fandom/ ./files/data/Fandom/Crossovers/ ./files/data/Fandom/Crossovers/Big_fake_companies/ ./files/data/Fandom/Crossovers/Westphall_crossovers/ ./files/data/Fandom/Fictional_family_trees/ ./files/data/Fandom/The_Riftwar/ ./files/data/Fandom/Xanth/ ./files/data/Miscellany/ ./files/data/Miscellany/Geeky_thoughts/ ./files/data/Miscellany/Political_opinions/ ./files/data/Movies/ ./files/data/Role_playing/ ./files/data/Role_playing/Locations/ ./files/data/Role_playing/Locations/Afma/ ./files/data/Role_playing/Magic_items/ ./files/data/Role_playing/Miscellany/ ./files/data/Role_playing/Monsters/ ./files/data/Role_playing/Player_characters/ ./files/data/Role_playing/Reference_tables/ ./files/data/Writing/ ./files/data/Writing/Erotic_fiction/ ./files/data/Writing/Fan_fiction/ ./files/data/Writing/Poetry/

      This is what I am grappling with in my head.

      My OS is Debian 10 (Buster); my perl versions are 5.28.1 local and 5.8.8 on web host.

      Version control is a non-issue, I do not use it.

      No matter how hysterical I get, my problems are not time sensitive. So, relax, have a cookie, and a very nice day!
      Lady Aleena

        For each index.pl in the equivalent of find . -name index.pl, try ($dir = $indexfile) =~ s!^[.]/!./files/data/! and glob $dir.'*.txt' (obviously untested) as a starting point, if the files in those directories correspond to menu items.

Re: Collapsing smaller scripts into a larger one, request for comment
by perlfan (Vicar) on May 31, 2020 at 06:19 UTC
    use lib '../files/lib';
    I'd use FindBin here. Regarding what all is in __DATA__, you could separate the driver code and turn each set of __DATA__ into a module. And rather than having a __DATA__ section, have an our DATA variable - since you're clearly editing this manually anyway, the work to maintain it will be the same after you convert it. This brings you closer to the suggestion, which I agree with, to separate your code from your data.

    For example, index.pl becomes:

    use strict; use warnings; package site::index; our $DATA = q{Welcome to Lady Aleena's B<collections>, which is lists +of A<novels|href="Fiction.pl">, A<books|href="Non-fiction.pl">, A<mus +ic|href="Music_and_comedy.pl">, A<movies|href="Movies.pl">, A<tie-ins +|href="Tie-ins.pl">, and A<programs|href="Programs.pl"> I am willing +to admit I own or use. The list of movies here is just those movies I + own and should not to be confused with my more general interest in A +<movies|href="../Movies">. Tie-ins are books and music connected to m +ovies or television series. I also share my A<fandom|href="../Fandom" +> elsewhere. Here is a key for the notations after each title with the exception of + programs.}; 1;
    Fiction.pl becomes:
    use strict; use warnings; package site::collection::Fiction; our $DATA = q{This is my B<fiction collection> of SPAN<hardcovers|^har +dcovers^>, SPAN<trade paperbacks|^trades^>, and SPAN<mass market pape +rbacks|^massmarkets^>.} 1;
    ...etc

    Then you'd register these modules in your main driver, and based on whatever it is that you're dispatching on you could use the appropriate full package name to get the content via $site::index::DATA, $site::collection::Fiction::DATA, etc. I suggest this because it seems you like this style for its maintenance ease, but not for it's Perl sophistication. The solution above keeps the maintenance overhead similar to now, but is more sophisticated Perl by allowing you to eliminate the repeated calling code and implement some dispatching. And you can even keep the same .pl file names and directory structure provided you require them (which you'd have to anyway.

      I looked at FindBin and can not tell what purpose it serves. It seems I would still need to know the relative path to my lib, so why add this to it?

      And so you know, the __DATA__ I put in the original post was just first paragraphs. Some __DATA__ fields are hundreds of lines long. So the modules you are suggesting would be tidier with __DATA__ fields and exporting those fields as a glob (faux file handle). I am also not sure that use statements could be put into loops.

      However, I may be misunderstanding what you are suggesting. The text still needs extensive munging any way I do it.

      My OS is Debian 10 (Buster); my perl versions are 5.28.1 local and 5.8.8 on web host.

      Version control is a non-issue, I do not use it.

      No matter how hysterical I get, my problems are not time sensitive. So, relax, have a cookie, and a very nice day!
      Lady Aleena
Re: Collapsing smaller scripts into a larger one, request for comment (no)
by Anonymous Monk on May 31, 2020 at 02:53 UTC

    No.

    Keep all your files.

    But keep your data and your code separate.

    Remove the code from your scripts because it doesn't belong. Remove everything before __DATA__.

    Rename your .pl templates to .lat or .dat

    Collapse any other page specific code needed into subroutines inside a module.

    This is all that should be in your master script

    use lib ...; use MyStuff; MyStuff::run_cgi( '/path/to/root/or/config/if/nonstandard' );

    Eventually get inspired by https://metacpan.org/release/Template-TT2Site/tt2site and make the dynamically generated static parts of your site truly static

      I intend to keep all the text in the __DATA__ fields, but not in their current directories. I am not sure what the difference is between me leaving the files in place and renaming them or moving the text to a new directory with a similar new filename.

      ./Collections/Fiction.pl # to ./Collections/Fiction.txt # or ./files/text/Collections/Fiction.txt

      Leaving them in place would mean adding another grep to exclude all .txt files in the directories in base_menu along with initial lower case files. I'm having difficulty rewriting base_menu for the new structure I am contemplating.

      I think .dat would expand to "data", so what would .lat expand to be? I am curious.

      My OS is Debian 10 (Buster); my perl versions are 5.28.1 local and 5.8.8 on web host.

      Version control is a non-issue, I do not use it.

      No matter how hysterical I get, my problems are not time sensitive. So, relax, have a cookie, and a very nice day!
      Lady Aleena
Re: Collapsing smaller scripts into a larger one, request for comment
by Lady_Aleena (Curate) on Jun 11, 2020 at 07:44 UTC

    Update: I am done mostly with writing the indexes. There are a few parts of the code that I am still mulling over. The following is a basic index script.

    I have 5 page-scripts that do not use story, so they are still independent scripts for now until I can figure out how to get them into the menu automatically if I put their code into their indexes. I mentioned in the OP that I had to rewrite my base_menu subroutine. Lines 16-28 took me several hours over a couple of days to get right.

    My OS is Debian 10 (Buster); my perl versions are 5.28.1 local and 5.8.8 on web host.

    Version control is a non-issue, I do not use it.

    No matter how hysterical I get, my problems are not time sensitive. So, relax, have a cookie, and a very nice day!
    Lady Aleena
      After our CB convo, I decide to try to follow my own advice, this is what I got for Programs.pl:
      #!/usr/bin/perl use strict; use warnings FATAL => qw( all ); package my::site::Programs; sub get_data { my $self = shift; # not used but passed with '->' notation return q{This is a list of B<programs> that I am using or have use +d. I can not account for I<all> the software we have had and used ove +r the years. Some of it was so bad, we blanked it out of our heads. T +his list does not include a full list of hardware drivers either. So +much software, so little time or in this case patience.}; } 1; # not sure if this is actually needed here, but would be if you mov +ed this to a .pm that is "use"d or "require"d # replicate the functionality of the original Programs.pl package main; use CGI::Carp qw(fatalsToBrowser); use lib '../files/lib'; use Base::Page qw(page story); use Util::StoryMagic qw(program_magic); my $magic = program_magic; page( 'code' => sub { story(my::site::Programs->get_data(), { 'line ma +gic' => $magic }) }); # note you'll have to change how you handle wha +t was passed as "*DATA" # # NOTE, since you're now not using __DATA__ (via *DATA handle), you ca +n make the "main" content # minimal by passing in just the code reference: # # package main; # use lib '../files/lib'; # Base::Page->page( code => \&my::site::Program::get_data, story => q{ +Util::StoryMagic} ); #

      The goal here was to show what I meant by making the catagory files into modules and to drop __DATA__. This offers no change to how things work now, but by converting all of your .pl files to a module internally, this sets you up for some things:

      1. incrementally put your data into module-like libraries
      2. move them to functional libraries by turning the above into a modulino - allows you to  use require q{../file/Programs.pl}, e.g.,
      3. once they work as they do now, but can be used us libraries as modulinos, you are free to experiment with Dancer2 or Mojo since they will fundamentally differ from your existing code being router/dispatch based and not file based (e.g., require content modules in the route handlers)
      4. then you can decide how to proceed, but this allows you to retain the functionality of the site now while exploring a more modern approach

      Then eventually if you made them full libraries, you'd pull them into to the framework like:

      # in framework route handing require q{path/to/Programs.pl}; my $data = my::site::Programs->get_data(); #...now do something with $data
Re: Collapsing smaller scripts into a larger one, request for comment
by Anonymous Monk on Jun 05, 2020 at 18:14 UTC
    This all looks very unwieldy to say the least. You would give your self an easier ride just using wordpress. If you want to write code for the sake of writing code that's one thing. It's like you are tripping yourself up deliberately, and enjoying it. Look at the conversation about Dancer2/Mojo, there's a good reason nobody writes stuff like you do.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://11117478]
Approved by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others meditating upon the Monastery: (5)
As of 2020-11-25 02:04 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?