Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: Performance Trap - Opening/Closing Files Inside a Loop

by edoc (Chaplain)
on Dec 10, 2004 at 03:21 UTC ( [id://413743]=note: print w/replies, xml ) Need Help??


in reply to Performance Trap - Opening/Closing Files Inside a Loop

woah! way to get distracted! I just realised I've spent way too much time messin with this.. back to work..

not really a "run once" as it would need tuning to the machine/data etc and the sorts could certainly do with optimising..

max records and max hash keys are configurable dependant on available memory. When there are too many records, the largest file is dumped, when there are too many files (hash keys) we dump the smallest.

#!/usr/bin/perl use strict; use warnings; my $max_records = 10; my $max_files = 4; my %rec; my $record_count = 0; my $total_record_count = 0; my $total_dump_count = 0; while ( <DATA> ) { $record_count++; print $record_count."] $_"; my @field = split /,/; my $file = splice @field, 2, 1; push(@{$rec{$file}},$_); if ($record_count >= $max_records){ print " Too many records..\n"; my ($big_file) = sort { @{$rec{$b}} <=> @{$rec{$a}} } keys %rec; dumping($big_file); } if (scalar keys %rec > $max_files){ print " Too many files..\n"; my ($lil_file) = sort { @{$rec{$a}} <=> @{$rec{$b}} } keys %rec; dumping($lil_file); } $total_record_count++; } print " Data Stopped..\n"; foreach(keys %rec){ dumping($_); } print "Total Records: $total_record_count\n"; print "Total Dumps: $total_dump_count\n"; sub dumping{ my $file = shift; $total_dump_count++; print " Dumping $file (".@{$rec{$file}}." records)\n"; open( FILE, '>>', $file ); print FILE @{$rec{$file}}; close(FILE); $record_count -= @{$rec{$file}}; delete $rec{$file}; } __DATA__ 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 3,4,foo,5 6,7,bar,8 9,0,baz,1 2,3,barbar,4 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 5,6,bazbaz,7 3,4,foo,5 1,2,foo,3 4,5,bar,6 7,8,foo,9 0,1,baz,2 3,4,foo,5 6,7,bar,8 9,0,baz,1 ...output... 1] 1,2,foo,3 2] 4,5,bar,6 3] 7,8,foo,9 4] 0,1,baz,2 5] 3,4,foo,5 6] 6,7,bar,8 7] 9,0,baz,1 8] 2,3,barbar,4 9] 1,2,foo,3 10] 4,5,bar,6 Too many records.. Dumping foo (4 records) 7] 7,8,foo,9 8] 0,1,baz,2 9] 5,6,bazbaz,7 Too many files.. Dumping bazbaz (1 records) 9] 3,4,foo,5 10] 1,2,foo,3 Too many records.. Dumping bar (3 records) 8] 4,5,bar,6 9] 7,8,foo,9 10] 0,1,baz,2 Too many records.. Dumping baz (4 records) 7] 3,4,foo,5 8] 6,7,bar,8 9] 9,0,baz,1 Data Stopped.. Dumping bar (2 records) Dumping baz (1 records) Dumping foo (5 records) Dumping barbar (1 records) Total Records: 21 Total Dumps: 8

cheers,

J

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://413743]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (5)
As of 2024-04-24 22:36 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found