Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re^2: efficient perl code to count, rank

by haj (Vicar)
on Jul 18, 2021 at 18:49 UTC ( [id://11135136]=note: print w/replies, xml ) Need Help??


in reply to Re: efficient perl code to count, rank
in thread efficient perl code to count, rank

It should be noted that at no point does the code sort 14M arrays. The variable @rows actually holds the fields for the current line. The code sorts, for every column, the numbers of occurrences of different values. While a column could hold 14M different values in 14M lines, this is not the case here: With 14M lines of 1400 fields each, and 62GB in total, the average column has a data width of two bytes. You can only cram so much different values into two bytes (especially if it's text) - that's several orders of magnitude away from 14M and should fit into memory quite easily.

The sorting problem in the code is, as has been pointed out, that the sorting is done 14M times instead of once.

Replies are listed 'Best First'.
Re^3: efficient perl code to count, rank
by LanX (Saint) on Jul 18, 2021 at 19:33 UTC
    Maybe, maybe not.

    Without example IN and OUT it's hard to guess what an OP really tried to achieve in messy code... 🤷🏾

    That's why I asked for an SSCCE

    And ranking columns looks weird.

    Anyway, the question how to sort data which doesn't fit into memory is more interesting for me!°

    Call it a thread drift, but the monastery is currently not really busy answering interesting questions. ;)

    And I learned a lot about sorting ...

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

    °) like https://www.geeksforgeeks.org/external-sorting/

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11135136]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (2)
As of 2024-04-25 19:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found