http://qs321.pair.com?node_id=53945


in reply to Re: Sorting data that don't fit in memory
in thread Sorting data that don't fit in memory

While I'm looking into the possibilities provided by the other posters, these tips don't require more advanced knowledge than I have ;-).

Thank you for warning me about the anonymous array overhead. Will change that immediately.

Furthermore, I'm on little endian, so the string comparison should work just fine. However, I don't want to sort on the remainder of the string as it is now. Will run a test first, and post the results here.

Thanks a lot,

Jeroen
"We are not alone"(FZ)

Update: Must have made some difference in memory usage, but it's still way too much. Made a rough calculation of the memory usage with system monitor (gnome) and this little script:

#!/usr/bin/perl my $size=1E6; my $f = 2; print "Increasing memory print each step by a factor $f\n\n"; while(1){ print "\tCreating array of $size items...Press enter to continue.... +"; my $b = <STDIN>; my @a=1..$size; print "...done. Press enter to continue."; $b = <STDIN>; chomp $b; $b=~ /[qxQX]/ and exit; $size *= $f; }
It's about 44 megabytes for every million items in an array. No way my 12 M records are going to fit in physical memory. I'm back to Radixsort and alike.