http://qs321.pair.com?node_id=413743


in reply to Performance Trap - Opening/Closing Files Inside a Loop

woah! way to get distracted! I just realised I've spent way too much time messin with this.. back to work..

not really a "run once" as it would need tuning to the machine/data etc and the sorts could certainly do with optimising..

max records and max hash keys are configurable dependant on available memory. When there are too many records, the largest file is dumped, when there are too many files (hash keys) we dump the smallest.

#!/usr/bin/perl use strict; use warnings; my $max_records = 10; my $max_files = 4; my %rec; my $record_count = 0; my $total_record_count = 0; my $total_dump_count = 0; while ( <DATA> ) { $record_count++; print $record_count."] $_"; my @field = split /,/; my $file = splice @field, 2, 1; push(@{$rec{$file}},$_); if ($record_count >= $max_records){ print " Too many records..\n"; my ($big_file) = sort { @{$rec{$b}} <=> @{$rec{$a}} } keys %rec; dumping($big_file); } if (scalar keys %rec > $max_files){ print " Too many files..\n"; my ($lil_file) = sort { @{$rec{$a}} <=> @{$rec{$b}} } keys %rec; dumping($lil_file); } $total_record_count++; } print " Data Stopped..\n"; foreach(keys %rec){ dumping($_); } print "Total Records: $total_record_count\n"; print "Total Dumps: $total_dump_count\n"; sub dumping{ my $file = shift; $total_dump_count++; print " Dumping $file (".@{$rec{$file}}." records)\n"; open( FILE, '>>', $file ); print FILE @{$rec{$file}}; close(FILE); $record_count -= @{$rec{$file}}; delete $rec{$file}; } __DATA__ 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 3,4,foo,5 6,7,bar,8 9,0,baz,1 2,3,barbar,4 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 5,6,bazbaz,7 3,4,foo,5 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 3,4,foo,5 6,7,bar,8 9,0,baz,1 ...output... 1] 1,2,foo,3 2] 4,5,bar,6 3] 7,8,foo,9 4] 0,1,baz,2 5] 3,4,foo,5 6] 6,7,bar,8 7] 9,0,baz,1 8] 2,3,barbar,4 9] 1,2,foo,3 10] 4,5,bar,6 Too many records.. Dumping foo (4 records) 7] 7,8,foo,9 8] 0,1,baz,2 9] 5,6,bazbaz,7 Too many files.. Dumping bazbaz (1 records) 9] 3,4,foo,5 10] 1,2,foo,3 Too many records.. Dumping bar (3 records) 8] 4,5,bar,6 9] 7,8,foo,9 10] 0,1,baz,2 Too many records.. Dumping baz (4 records) 7] 3,4,foo,5 8] 6,7,bar,8 9] 9,0,baz,1 Data Stopped.. Dumping bar (2 records) Dumping baz (1 records) Dumping foo (5 records) Dumping barbar (1 records) Total Records: 21 Total Dumps: 8

cheers,

J