Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re: best way to use grep

by Laurent_R (Canon)
on Dec 27, 2018 at 22:11 UTC ( [id://1227768]=note: print w/replies, xml ) Need Help??


in reply to best way to use grep

In terms of performance, the single most important change to be done is to use a hash enabling direct lookup instead of an array to store ther $xref data, as already suggested by Corion. Depending on the size of the $xref file, this change alone can make your program thousands times (or even much more) faster.

A few other changes might improve your code. Insert these pragmas:

use strict; use warnings;
at the top of all your Perl programs, and declare your variables (with the my keyword).

You don't need to store your customer file into an array (@lines) and then read the array, this leads you to process twice the same data, which is a waste of time. Do your checks when reading your customer file line by line;

Use the three-argument syntax of open and use lexical file handles (see examples below).

This could lead to something like this (untested):

use strict; use warnings; # Read the values from XREF into a hash my $xref = "input_file.txt"; # put here real filename open my $XREF, "<", $xref or warn "Could not open $xref $!"; # A more + detailed message can sometimes be useful my %xreflines = map { chomp; $_ => 1 } grep /\S/, <$XREF>; # popu +lating the hash with the content of the xref file close $XREF; my $goodfile = ...; # insert the name +of the output file here open my $XFILE, ">>", $goodfile or warn "Could not open $goodfile $!"; my $outfile = ...; # insert the name +of the customer file here open my $CUSTOMERFILE, "<", $outfile or warn "Could not open $outfile +$!"; while (my $line = <$CUSTOMERFILE>) { my ($dist, $cust) = (split /;/, $line)[0,1]; # no need to chomp + the fields, there are other fields afterwards in the line my $key = "$dist$cust"; # Building the loo +kup key unless (exists $xreflines{$key}) { # hash lookup print $XFILE "$line"; # printing the lin +es whose ID is not found in the cross ref file } } close $XFILE; close $CUSTOMERFILE;
Update: removed some extra quote marks left out from the OP code when changing the open syntax. Thanks to choroba for pointing out.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1227768]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (7)
As of 2024-03-28 08:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found