|Problems? Is your data what you think it is?|
Re: Re: Sorting data that don't fit in memoryby jeroenes (Priest)
|on Jan 24, 2001 at 16:04 UTC||Need Help??|
While I'm looking into the possibilities provided by the other posters, these tips don't require more advanced knowledge than I have ;-).
Thank you for warning me about the anonymous array overhead. Will change that immediately.
Furthermore, I'm on little endian, so the string comparison should work just fine. However, I don't want to sort on the remainder of the string as it is now. Will run a test first, and post the results here.
Thanks a lot,
Update: Must have made some difference in memory usage, but it's still way too much. Made a rough calculation of the memory usage with system monitor (gnome) and this little script:
It's about 44 megabytes for every million items in an array. No way my 12 M records are going to fit in physical memory. I'm back to Radixsort and alike.