Perl Monk, Perl Meditation | |
PerlMonks |
Re: Hash Search is VERY slowby kcott (Archbishop) |
on Sep 29, 2021 at 00:06 UTC ( [id://11137103]=note: print w/replies, xml ) | Need Help?? |
G'day rtjensen, Welcome to the Monastery. "I have a script that loads a CSV file of around 800k lines, they're firewall logs, I'm trying to pull out the IP address and the URL they're hitting." I use very large CSV files at $work. In my case, they hold biological data; however, that's completely immaterial. I have one file which I use for volume testing which is over 2Gb. I expect that's probably comparable to your logfiles. I would take a different approach to what you show and use Text::CSV (if you also have Text::CSV_XS installed, it will run a lot faster). What follows is example code and data to show the technique; adapt it for your specific needs. The data:
The code:
The output:
See also: autodie and Data::Dump. — Ken
In Section
Seekers of Perl Wisdom
|
|