Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: How to optimize a regex on a large file read line by line ?

by RichardK (Parson)
on Apr 16, 2016 at 14:51 UTC ( #1160643=note: print w/replies, xml ) Need Help??


in reply to How to optimize a regex on a large file read line by line ?

How long are the lines in your file? and how many lines is it reading in total? Maybe reading it a line at a time is not the best approach for your data set.

  • Comment on Re: How to optimize a regex on a large file read line by line ?

Replies are listed 'Best First'.
Re^2: How to optimize a regex on a large file read line by line ?
by John FENDER (Acolyte) on Apr 16, 2016 at 14:59 UTC
    How long ? Well, it's could vary regarding the extract you can make and the data you would analyze. Some logs are huges, more than 2Gbs... For starting 10000000 lines for passwords log 185866729 lines for the dictionnary file The entry are not very long, nothing more than 8 or 16 chars i would say.

      There's no point trying to optimize your code if you're not sure what your data looks like. However index will be faster than a regex if you're only looking for a fixed string.

      As other people have recommended, profile your code and find out where the time is going.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1160643]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (4)
As of 2020-05-27 04:16 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If programming languages were movie genres, Perl would be:















    Results (152 votes). Check out past polls.

    Notices?