Re: Re: Heap sorting in perl

I have a mod_perl process that slurps up a large dataset from a database, sorts it, then displays the first N items. This happens to balloon the memory consumption of the apache process to an unacceptable level.

An obvious question, perhaps, but have you ruled out getting the database to do the work for you? That'd be a no-brainer if the column you're interested in is indexed. I'm guessing it isn't. Still, having the database do the sort might still be faster than scanning the entire table, pulling the data across a process boundary a row at a time, and then filtering it in the Apache process.

Assuming you have a fairly large table, here's an ugly heuristic you might try: Probe the first 2*M rows of the table, sort, and find the M'th largest value. Then run a query that does a sort on the database side, subsetting the data before sorting by adding a WHERE clause to reject any row whose target column has a value greater than the M'th value you found the first time through. You can tune this by trying 3*M rows, 4*M rows, etc. The idea is to reduce the set that the database needs to sort.

Comment on Re: Re: Heap sorting in perl

Replies are listed 'Best First'.
Re3: Heap sorting in perl by blakem (Monsignor) on Apr 06, 2003 at 08:37 UTC
Letting the datbase do the sort is a good approach, but I was hoping for a more general, memory efficient solution that allows for arbitrary perl comparison routines. Its a common enough situation that I would like to create a subroutine that returns the rows I want and takes a database handle, a sorting coderef, M, and an optional offset. I'm envisioning something like: `my $sortedrows = db_heap_sort($dbh, $coderef, $M, $offset);` [download] The question is, is there a perl heap implementation that would be suitable for implementing the above sub. Heap.pm seems like the obvious choice, but I have doubts about its current status. I have emailed the Heap.pm author, and will follow up with any new information I find out. -Blake	[reply] [d/l]
Re: Re: Re: Heap sorting in perl by pg (Canon) on Apr 06, 2003 at 03:17 UTC
If those are rows in a database, I agree with dws, just let database do it. If that particular column is not indexed, I suggest index it, as you need it. "order by" is usually not too expansive, especially when it only involves indexed range scan. Depends on your database, some database directly support syntax that allows to retrieve first N rows.	[reply]
Re: Re: Re: Re: Heap sorting in perl by dws (Chancellor) on Apr 06, 2003 at 04:04 UTC
If that particular column is not indexed, I suggest index it, as you need it. If this is a write-mostly, query-infrequently table, indexing could significantly degrade performance. For log/audit files that record large numbers of miscellaneous value, indexing the values is rarely an effective strategy.	[reply]
Re: Re: Re: Re: Re: Heap sorting in perl by pg (Canon) on Apr 06, 2003 at 04:31 UTC
When this is generally correct, I don't really see a reason why a person want to "sort by" some column containing some miscellaneous values. If unfortunately that's the case, then should look into the data, and have some sort of key information extracted. Any way, I believe blakem has his stuff well organized, and what he is doing is reasonable.	[reply]


There's more than one way to do things
	PerlMonks