in reply to Re: Sorting data that don't fit in memory
in thread Sorting data that don't fit in memory
While I'm looking into the possibilities provided by the
other posters, these tips don't require more advanced
knowledge than I have ;-).
Thank you for warning me about the anonymous array overhead. Will change that immediately.
Furthermore, I'm on little endian, so the string comparison should work just fine. However, I don't want to sort on the remainder of the string as it is now. Will run a test first, and post the results here.
Thanks a lot,
Jeroen
"We are not alone"(FZ)
Update: Must have made some difference in memory usage, but it's still way too much. Made a rough calculation of the memory usage with system monitor (gnome) and this little script:
It's about 44 megabytes for every million items in an array. No way my 12 M records are going to fit in physical memory. I'm back to Radixsort and alike.#!/usr/bin/perl my $size=1E6; my $f = 2; print "Increasing memory print each step by a factor $f\n\n"; while(1){ print "\tCreating array of $size items...Press enter to continue.... +"; my $b = <STDIN>; my @a=1..$size; print "...done. Press enter to continue."; $b = <STDIN>; chomp $b; $b=~ /[qxQX]/ and exit; $size *= $f; }
|
---|
Replies are listed 'Best First'. | |
---|---|
(tye)Re: Sorting data that don't fit in memory
by tye (Sage) on Jan 24, 2001 at 19:40 UTC | |
by jeroenes (Priest) on Jan 24, 2001 at 20:07 UTC |
In Section
Seekers of Perl Wisdom