sub do_em {
    ...
    my $module = 'Crunch::' . $company;
    require $module; 
    $module->do_it( $dir );
  }
}
[download]

Update: Actually that should be:

    my $module = 'Crunch/' . $company . '.pm';
[download]

...per mpeters' note below.

There are also some CPAN modules for doing plugins which will scan a directory and load each module fitting some criteria, so you don't even have to do the require.

No, this won't actually work. From perldoc:

But if you try this:

  $class = ’Foo::Bar’;
  require $class;      # $class is not a bareword
  #or
  require "Foo::Bar";  # not a bareword because of the ""
[download]

The require function will look for the "Foo::Bar" file in the @INC array and will complain about not finding "Foo::Bar" there. In this case you can do:

  eval "require $class";
[download]

But you could do:

  my $module = 'Crunch/' . $company .'.pm';
  require $module;
[download]


-- More people are killed every year by pigs than by sharks, which shows you how good we are at evaluating risk. -- Bruce Schneier

Thanks for the cluebricks on eval, and on the keyword "plugin", which hadn't occurred to me. Searching for "plugin" in CPAN brings down about 600 hits, most of which are not general enough (e.g. Siesta::Plugin, PXP::Plugin, etc.), so I have some wading to do. If you remember any specific names, please let know.

Update: Thanks to mpeters for reminding me why I was fussing with eval.

the lowliest monk

It's actually not so much a 'plugin' architecture, but a 'factory' design patter. Check out Class::Factory.

-- More people are killed every year by pigs than by sharks, which shows you how good we are at evaluating risk. -- Bruce Schneier

Perhaps it's an aversion to using eval,

# eval "require $module; $module\::do_it( '$dir' ); 1"
  require "$module";
  $module->do_it( $dir );
[download]

my $m = Church->new('GeneralMotors') or die;
# $m is Church::GeneralMotors
[download]

Your factory approach is clean, but it seems like you're just moving tlm's problem from one place to another. How is Church::new implemented?

Flavio
perl -ple'$_=reverse' <<<ti.xittelop@oivalf

Don't fool yourself.

package Church;
sub factory_create
{
  my ( $class, %p ) = @_;
  my $new_class = ref($class) || $class . "::" . $p{firm};
  eval "require $new_class" or die $@;
  return $new_class->new(%p);
}
[download]

Boris

How about just having a separate perl script for each company? It seems like the modularized approach isn't really doing all that much except managing a namespace and it adds a lot of complexity for little value that I can see. You could keep all the scripts in a single directory and just execute them all whenever you need to, via cron or via some control routine that just executes all the scripts in a given directory. Each script can contain the location to store raw and processed files.

That's not that different from your approach using 100 modules, but why use require and eval when do or system would work just as well? Ideally, there would be some reuseable bits that you could put into modules to share between these scripts so that each script is just a small wrapper to drive some common way of gathering data and some shared output functions.

How about just having a separate perl script for each company? It seems like the modularized approach isn't really doing all that much except managing a namespace and it adds a lot of complexity for little value that I can see.

I'm glad you brought this up. My main reason for using the module approach was so I could test the various internal functions that support the do_it API function. I find that there's a lot more support available for testing functions if they belong to a loadable module, though, admittedly, this rationale has a tail-wagging-the-dog ring to it.

the lowliest monk

You can have it both ways. See brian d foy's article 'How a script becomes a module'. Make it a module for testing purposes and a script otherwise.

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

I'd use the eval, but only for the require. I'd use $module->doIt() or, more likely, $module->new(...) followed by some other method calls.

Note that I'd likely not use inheritance to factor out commonality here. I encourage you to try to factor out common parts, but by making modules that your main modules use, not modules that your main modules try to BE (as in @ISA).

Your situation sounds like a nearly perfect situation for discovering how inheritance binds so tightly and can leave you in a bind. But learning that from reading instead of from personal, painful experience is a sign of intelligence. (:

I'm reading Head First Design Patterns, which says to favor composition over inheritance. Encapsulate what changes, but there are other ways to do that besides inheritance, like delegation. What you say, Tye, reminds me of what I've read.

As for the calling mechanism, eval will certainly work, as would a system() call... Have you thought about a dispatch table? It probably wouldn't be any cleaner, but it might be.

I have something quite similar where I have lots of detailed expect scripts that go off and do some function to lots of different network equipment. In that case, I'm actually calling those via system(); and yes I know about the CPAN expect module, I just haven't had the tuits to redo all the work already done in expect...

HTH,

-Scott

Rather than eval or some equivelent, use a Perl script in each company's folder that does the refresh work. Your update script then just itterates over the folders that are currently present and executes the refresh script in each folder.

To add a new company, create a folder and the refresh script.

It would seem best to put all your 'data_gathering' processes in one module. There has to be a lot of commonality in the code for each company.

In the module you might have the following subs.

• gather_data_A .. n using WWW::Mechanize, FTP, or access an RDBMS, etc
• process_data_A .. n
• save_data_A .. n

Now you need to tell the module which process combination to use. For this you could use a hash, outside the module, that defines the process for each company. That way you don't need to mess with the module when you add a new company provided, of course, that the required sub exists in the module.

use Crunch;

%process = (
    'WalMart' => 'A:A:A',
    'ExxonMobil' => 'B:D:A',
    ...
    'CiscoSystemsInc' => 'C:D:A',
);

sub do_em {
  my $path_to_companies = shift;
  opendir $dh, $path_to_companies or die "Can't opendir $path_to_compa
+nies: $!\n";
  while ( my $company = readdir( $dh ) ) {
    next if $company ~= /^\./;
    my $dir = catdir( $path_to_companies, $company );
    next unless -d $dir;
    Crunch::do_it(
        dir => $dir
        company => $company
        process => $process{$company);
    );
  }
}
[download]

There has to be a lot of commonality in the code for each company.

No, I tried to be very explicit and emphatic about this point. That's precisely the problem: each company requires mostly idiosyncratic code.

the lowliest monk

I'd be very careful about this. Because if you look for similarities, there likely will be many. I realise that the idiosyncracies probably outweigh the similarities, but even encapsulating those few similarities (which, of course, won't be similar among all companies, but probably a significant subset will share one similarity while another subset shares another similarity, etc.) can pay dividends.

For example, what I have done in a similar (yet completely dissimilar) situation ... where I have a bunch of tasks to perform, each one is almost completely unique from the other, and I have a task manager object that reorders the tasks (order can be important for me - not likely for you, but the manager could do other things to help, such as figure out that some companies have been updated recently enough to skip, or something) and then executes them serially.

The manager object is actually almost completely uninteresting here.

package TaskManager;

use Task;

sub new {
  # create.
  # receive task objects or task names, store them via add_task.
}

sub add_task {
  # receive task objects or task names
  my $self = shift;
  push @{$self->{_tasks}}, map {
    ref $_ ? $_ : Task::find_task($_)
  } @_;
}

sub execute_tasks {
  my $self = shift;

  # do we need to do them?
  my @tasks = grep {
    $_->needed()
  } @{$self->{_tasks}}

  for my $t (@tasks)
  {
    $t->perform_task();
  }
}

1;
[download]

For the most part, that's it. I skipped the sorting part, for example. And strict/warnings - they're in the real module. And then, the Task module looks like this:

package Task;

sub new {
  # creation of generic object.
}

my %task_cache;
sub find_task {
  my $task = shift;
  $task = shift if $task eq __PACKAGE__; # can call as Task-> or Task:
+:

  return $task_cache{$task} if $task_cache{$task};

  (my $mod_name = $task . '.pm') =~ s{::}{/};
  eval { require $mod_name };
  if ($@) { 
    # handle error - return error, die, whatever.
  }
  $task_cache{$task} = $mod_name->new(@_);
}

sub needed
{
  #default - yes.
  1
}

sub perform_task
{
  # default... can't.
  my $self = shift;
  die "Forgot to override perform_task in " . ref($self) . "\n";
}

1;
[download]

Now I didn't really show you how to reuse stuff. But what I've started here is a framework. A framework upon which you can build further frameworks. For example:

package Task::WWW;

use WWW::Mechanize;

sub perform_task
{
  my $self = shift;
  # set up proxy, ...
  my $www = WWW::Mechanize->new($self->get_hostname());
  $self->handle_page($www);
}
1;
[download]

Now you just need to derive your HTTP-gathering info from Task::WWW, override two simpler functions, and can rest assured that if your application needs to change proxies for http requests, they all go through this code and can be easily modified. I realise that LWP::Simple probably uses environment variables for this, but it's just an example. I'm sure the more imaginative out there can find another use for this type of abstraction. Perhaps you now need to authenticate to get out of your company intranet. Or something equally silly. Whatever. By abstracting out the common parts, you not only can save time in adding new requirements like that, but you can concentrate on smaller pieces of the overall puzzle at a time in each function in each module.

You'll have well over 100 modules this way, but each one will be simpler.

(I think I need to formalise this and put it on CPAN...)

This would make me go with something akin to:

DataCollector
DataCollector::Mechanized
DataCollector::Mechanized::WalMart
DataCollector::Mechanized::GeneralElectric
DataCollector::RDBMS
DataCollector::RDBMS::ExxonMobil
DataCollector::FTP
DataCollector::FTP::GeneralMotors
DataCollector::Scrape
DataCollector::Scrape::FordMotorCompany
DataCollector::Scrape::CiscoSystemsInc
. . .
etc.
[download]

Your driver program would then, unfortunately, need to know all of the DataCollector leaf classes or devise a method to dynamically load and run them. But for each of these classes, you could call the above mentioned methods. Those methods would make private method calls on down until you get to the ugly details in the individual implementation classes. These implementation classes would only need to know where it's going for data and how to pull the real data from raw data source. Up one level would be how to talk to the data source type, based on information in the implementation classes. Up in the top level is the detail of how to write out the data.

I hope this makes sense, isn't too vague, etc. Good luck.