When passing the hash back to the calling function, the process 
stalled indefinitely but there was a great deal of memory usage 
in the background and one CPU saturated.
[download]

First, please don't put your entire posts into <c></c> or <code><code> tags. These are reserved for code.

Second, you are probably passing the hash back like that

    return %hash;
[download]

which passes back a long, long, really long list of tuples (key/value pairs). Pass back a reference.

    return \%hash;
[download]

See perlref.

which passes back a long, long, really long list of tuples (key/value pairs). Pass back a reference.

*cough* Its just a flat list, there are no tuples (k/v pairs), or its one big tuple

use Time::HiRes qw(time);

my $n = $ARGV[0];
my %h;
for (1..$n) {
    my $key = sprintf "%06d", $_;
    $h{$key} = 1;
}

my @a;
my $t1 = time;
@a = keys %h;
my $t2 = time;
printf "time = %.3f; secs per 1M keys: %.3f\n",
    ($t2-$t1), ($t2-$t1) / $n * 1_000_000;
[download]

$ perl foo 1000000
time = 0.400; secs per 1M keys: 0.400
$ perl foo 2000000
time = 0.845; secs per 1M keys: 0.423
$ perl foo 4000000
time = 1.752; secs per 1M keys: 0.438
$ perl foo 8000000
time = 3.792; secs per 1M keys: 0.474
$ perl foo 16000000
time = 9.213; secs per 1M keys: 0.576
[download]

Dave.

To echo what's already been said, please don't format your entire post in code blocks. That should be used for code and program output. I'm sure many people have passed on helping because it was hard for them to read and understand the problem.

There seems to be a mismatch between your code segment and the example outputs. The first output says:

CCS: Fsize 216913920, pix 36152320
ETime=28.648 min, =97.60%, Event 'CCS: Extract RGB2C keys'
ETime=0.380 min, = 1.30%, Event 'CCS: Read_and_hash'
ETime=0.325 min, = 1.11%, Event 'CCS: Write_RGBC'
97.60% -> CCS: Extract RGB2C keys
 1.30% -> CCS: Read_and_hash
 1.11% -> CCS: Write_RGBC
Elapsed time = 29.58 min
[download]

But the time_event() calls in the snippet are:

&time_event('CCS: Counting RGB hash keys', \%e2at...
&time_event('CCS: Extract RGB2C keys', \%e2at,...     
&time_event('CCS: >open Output file', \%e2at,     
&time_event('CCS: Write_RGBC', \%e2at,...     
&time_event('CCS: Close_RGBC', \%e2at, $debug*0);
printf("CCS: %d bytes written to fn '$ofile'\n", -s $ofile);
[download]

But most of these do not appear in your output. It's also not entirely clear from the code how time_event() works. I could see some possibilities of its functioning that would mean you're measuring the reading time from the file, when you think you're measuring the call to keys.

BrianP,

I tried to stay out of this one, but 'what the heck!'

The type of work your doing requires a deeper understanding of Perl than you want to spend time on. In your write-up you mention passing a hash from a subroutine back to the main program, but failed to include that in your limited code description. As others have already pointed out, passing a hashref to the subroutine would have avoided having to copy a 27MM key/value hash back to the main program. As you show below it looks like it's in the main program anyway.

Tiny, active code segment
...    
$sr_len = sysread(IN, $buf, $bsize);  # SysRead Length    
last  if $sr_len == 0;      
while($buf)  {       
    $rgb=substr($buf, 0, 6, '');  # Nibble 6 bytes       
    $rgb2c{$rgb}++;    
}
[download]

    
@rgb = keys %rgb2c;  << 1 line takes 28.648 min
[download]

my $fsize = -s [your file];   ## Find out how big the image is?
my $arrsize = $fsize / 6;     ## Size of the array and hash
my $counter = 0;
my %rgb2c;
keys %rgb2c = $arrsize;       ## Allocate one large memory hash!
my @rgb[$arrsize] = '';       ## Allocate one large memory array!
while ( 1 )
{   $sr_len = sysread(IN, $buf, $bsize);  # SysRead Length    
    last  if $sr_len == 0;      
    while($buf)
    {   $rgb=substr($buf, 0, 6, '');  # Nibble 6 bytes       
        $rgb2c{$rgb}++;
        $rgb[$counter} = $rgb;        # Build array as you go along
        $counter++;
    }
}
[download]

At this point you have a hash for telling you the number of colors and an array that represents the exact image in 48 bit increments. By pre-allocating the hash and array you make only one call to the operating system for memory for each, instead of millions of calls.

Spend a little more time learning Perl and using efficient algorithms, and you'll have tools that will make you proud.

Regards...Ed