http://qs321.pair.com?node_id=11137774


in reply to Re^2: database and deployment questions
in thread Newbie question

Perl arrays and hashtables support nesting (e.g. hash-of-hash-of-array-of-hash etc.), mixed "data types" and slices (a convenient form of indexing). There is also PDL if your data is numeric. Additionally, there is Inline::Python if you want to inline python code you already have (untested by me).

storing millions of rows and hundreds of columns in 2d array if possible

well that's illegal in some states and some of us Perl-ers still work on Spectrums, :)

Update: There's also DBIx::Array which offers this:

foreach my $row ($dbx->sqlarrayhash($sql, @bind)) {
  do_something($row->{"id"}, $row->{"column"});
}

[download]

And there's DBIx::Class as a long-term investment.

bw, bliako

Replies are listed 'Best First'.
Re^4: database and deployment questions - updated for DBIx
by SpaceCowboy (Acolyte) on Oct 21, 2021 at 19:02 UTC
    Thank you! this seems like a start. While I would try to get as much as done in SQL way, there are some things like transformations, transposes, joins I really wish I could do it in Perl. Is there a package that would do cross-tab or pivot table in Perl? I found Data::Pivot and would love to test out others when I get a chance...
[reply]

      SpaceCowboy, a bit of giggling led me to DBIx::SQLCrosstab and Data::Pivoter. For the record, these are the terms I had searched: cross tabulation cpan perl. CPAN is the repository of Perl modules and where we usually install modules from. On the left tab of each module's page you will notice the date last updated and also the result of testing. Both modules are old but have no failed tests. I would definetely give them a try based on that information.

      There are a few ways to install modules from CPAN to your local machine. The most basic is by using cpan which ships with Perl. There's a more convenient module though: cpanm provided by App::cpanminus. In short: cpan App::cpanminus, bearing in mind that you may need to configure it the first time you use it, pressing enter for using the defaults works usually fine. And then cpanm DBIx::SQLCrosstab. Just mentioning these in case you are new to Perl.

      There are two more points to mention in case you are new to Perl and not aware. Where are modules installed? You can install them system-wide for all users if you want and do have system-admin privileges, or you can use your own personal library (see local::lib, but that's a whole new question) when with no privileges. Secondly and most importantly, Perl is used by many applications in our computers (ehm computer=unix-based OS) and by the OS itself. And that's why there is what we call system perl. By installing modules system-wide you are risking affecting that system perl's behaviour and breaking your system (for example it may rely on a certain module being on a certain version but you upgraded it with your admin privileges, rare but possible). The best way to tackle this is either to use local::lib or install another Perl, living in parallel with the "system Perl". This is much easier than it sounds, thanks to https://perlbrew.pl/ and it is highly recommended when you are dealing with unix systems. The system will be using its own Perl and you will be using your own (one or more, no problem) and the two will live happily together.

      reference: http://www.cpan.org/modules/INSTALL.html

      bw, bliako

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom