Re: Hash Search is VERY slow

It's working! I used choroba's advice and removed the whole if...else construct. Resultant loop looks like this:

while (my $row= $csv->getline_hr($fh))
{
        $linecounter++;
        my $ip=$row->{'Source address'};
        my $url=$row->{'URL/Filename'};

        push @{ $ipURL{$ip} }, $url;

        if (!($linecounter % 50000))
        {
                print "Lines: $linecounter\n";
        }
}
        formatOutput(\%ipURL);
#       print Dumper \%ipURL;
[download]

The entire thing runs in about 45 seconds now:

perl test-urlListbyIP.pl
Lines: 50000
Lines: 100000
Lines: 150000
Lines: 200000
Lines: 250000
Lines: 300000
Lines: 350000
Lines: 400000
Lines: 450000
Lines: 500000
Lines: 550000
Lines: 600000
Lines: 650000
Lines: 700000
Lines: 750000
Lines: 800000
Lines: 850000
Formatting Output...
List End:1316
Execution Time: 44.18 s
[download]

Comment on Re: Hash Search is VERY slow Select or Download Code

Replies are listed 'Best First'.
Re^2: Hash Search is VERY slow by AnomalousMonk (Archbishop) on Sep 29, 2021 at 19:13 UTC
I think you'll find that it was choroba's advice to process the file on a line-by-line (CSV-record-by-record in this case) basis that did the trick. :) Give a man a fish: `<%-{-{-{-<`	[reply] [d/l]
Re^3: Hash Search is VERY slow by NERDVANA (Deacon) on Sep 29, 2021 at 22:37 UTC
I’m going to bet it was bliako’s observation that the array was getting cloned every time an element was added. That’s where the N^2 behavior came from.	[reply]
Re^4: Hash Search is VERY slow by LanX (Saint) on Sep 29, 2021 at 23:14 UTC
Oh I missed that. => Re: Hash Search is VERY slow Brilliant! > that the array was getting cloned I think it's not so much the copying alone but the allocation of a new array each time. Because the arrays are growing, old released memory often can't be reused, leading to a fragmentation and increasing waste of space. Fun! :) Tho I might be wrong, the reserved space for an array grows by doubling IIRC, this should be easier to reuse ... Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery}	[reply]


Pathologically Eclectic Rubbish Lister
	PerlMonks