Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re^6: efficient perl code to count, rank

by haj (Vicar)
on Jul 19, 2021 at 07:21 UTC ( [id://11135158]=note: print w/replies, xml ) Need Help??


in reply to Re^5: efficient perl code to count, rank
in thread efficient perl code to count, rank

Yeah, the RAM problem is one one which becomes immediately apparent when looking at the code: Hence my wording "a first guess". As has already been written in this thread (and demonstrated by tybalt89's code), it can be eliminated by working through the file line by line, so with a small change in code the RAM problem does no longer exist. Also, it has nothing to do with sorting, it's just the attempt to slurp a 62GB file into an array. In the followups to the article you quoted "sorting" isn't even mentioned, because it is irrelevant.

We are still speculating what exactly he wanted to be ranked/sorted.

Looking at the code presented in the original posting should be considered an option. tybalt89 came up with the following explanation, which matches my own interpretation:

You were doing the ranking sort for each column...
I'm simply assuming that the OP's code performs the operation they want to be done, albeit inefficient. In that code there is not one sort over 14M entries, but there are thousand sorts (one per column). The OP's code does these 1000 sorts 14M times, that's why it won't finish in time, even for small arrays.

I hope that the Monks will eventually give tybalt89's article the ranking it deserves.

  • Comment on Re^6: efficient perl code to count, rank

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11135158]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (7)
As of 2024-04-23 13:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found