Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked

Re^2: How to improve speed of reading big files

by djp (Hermit)
on Sep 18, 2009 at 06:03 UTC ( #796033=note: print w/replies, xml ) Need Help??

in reply to Re: How to improve speed of reading big files
in thread How to improve speed of reading big files

I can second the idea of using standard Unix tools. In particular awk(1) can deliver huge performance improvements over perl. We had a text-file processing app which went from 20 minutes to 20 seconds when we recoded in awk. YMMV of course.
  • Comment on Re^2: How to improve speed of reading big files

Replies are listed 'Best First'.
Re^3: How to improve speed of reading big files
by Marshall (Canon) on Sep 18, 2009 at 08:58 UTC
    This is unbelievable! Most likely your awk guys didn't know how to write Perl very well. They coded awk more efficiently because that is what they knew how to do.
      No it's not unbelievable at all. It was a Perl expert who recoded the app in awk. We don't have 'awk guys', does anyone?
        Yes there are awk guys out there. I get awk code and have to deal with it from 2 people who will remain anonymous. Their awk code is good awk code. I write Perl stuff that "stitches their awk together" and we get a good result (meaning application that works).

        As far as "Perl experts" go, some folks are considerably better at Perl than others. I remain skeptical that a well coded Perl app would be of lower performance than an awk app. I will add that fewer lines of Perl doesn't necessarily mean higher performance. I will also add that the three of us have an app and process that works and we don't see any need to optimize the performance. I would argue before taking on an "optimizing" mission, a relevant questions is: does anybody care and does it matter? The main thing is: does it work?

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://796033]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (4)
As of 2023-10-04 00:41 GMT
Find Nodes?
    Voting Booth?

    No recent polls found