Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^2: MCE segmentation fault

by Anonymous Monk
on Mar 03, 2020 at 07:25 UTC ( [id://11113687]=note: print w/replies, xml ) Need Help??


in reply to Re: MCE segmentation fault
in thread MCE segmentation fault

This has been rather interesting. There is an overall speedup, but not what I expected.

Thank you Mario. I was trying to use the Imager object because of failing to implement the scalar solution (for dumb reasons, like omitting type => 'gif' on the write method), as you showed in the heatmap node, with errors like:

write_multi: image 1 is not an Imager image object
or
Usage: i_writegif_wiol(IO,hashref, images...)
I've only had time to try your first demo and of course it works wonders! The speed gain is more impressive if another 9 is added to the count to make a 100,000 frame GIF (with 8 workers on 3.1GHz i7):
real	3m43.591s # Perl
real	1m19.701s # MCE
MCE also appears to use much less memory. The plain Perl version grows to almost 2GB, while the MCE workers consume a mere 6MB each until the final process that consolidates the GIF only grows to 1GB. This makes an 80MB GIF. I tried a million frames too but the heat throttled my laptop CPU to 2 GHz while it used 9GB of RAM before swapping so that took 18 minutes to make an 820MB GIF, no fault of MCE...

MCE makes the hard things easy—thanks for supercharging Perl!

Replies are listed 'Best First'.
Re^3: MCE segmentation fault
by Anonymous Monk on Mar 03, 2020 at 07:58 UTC
    Sorry, just realized I compared apples to oranges. The Perl version that used 2GB was collecting Imager objects in the array, while MCE was using scalars. When changed to collect scalars plain Perl used the same amount of RAM as MCE, about 1GB, but then it was even slower than MCE:
    real	4m50.307s Perl
    real	5m24.204s Perl
    real	5m26.991s Perl
    
    real	1m16.036s MCE
    real	1m29.944s MCE
    real	1m28.977s MCE
    
    For some reason my CPU frequency is different when comparing the plain Perl version to MCE. MCE sends it straight to 3GHz while the plain Perl hangs out around 2-2.5GHz. I don't know why that happens but it must contribute to the difference...

      Wow :) MCE continues to boggle my mind after all these years.

        Excellent demonstration as usual, marioroy!

        Well, just slightly offtopic, w/r/t/ the efforts

        to speed up animated GIF generation

        the disturbing truth is that the main time-waster, above, appears to be Imager collecting, in single thread, 999(9...) decoded GIF images into final animation. Just look what it does: GIF files/scalars are read into internal Imager format (thankfully, kept palletized i.e. not converted to "true color"), the LZW compression is discarded/forgotten, it apparently tries palette(s) optimization(?), then individual frames are LZW-compressed again, very likely to the state they were initially stored in files/scalars. What a waste. To be fair, depending on image/scene, quantization requires much more work than LZW compression, so parallelization, as is, pays off well. However, one can't help but wonder if it all can be accelerated.

        Actually, yes, some tools appear to be more fit for the job, -- GD. It also, I think, decompresses GIF frames/scalars, but then either keeps (and then uses) originals, or is just optimized 100% ++.

        The example below is marioroy's code, with a few changes to setup. I run it on 4 cores, image is slightly larger (to give a computer some work to do, maybe closer to real life use) and has "interesting" background, otherwise compression cancels our larger image size, and it may again be closer to kind of images/animations OP is working with, it seems to me, -- or at least that was an idea. Number of frames, chunk size, etc. are changed according to setup above.

        use strict; use warnings; use feature 'say'; use Imager; use MCE::Loop; use Time::HiRes 'time'; use GD; STDOUT->autoflush; my $start = time; my $count = 0; my @data; my $bkg; my $TEST_GD = $ARGV[ 0 ] // 1; MCE::Loop->init( max_workers => MCE::Util::get_ncpu(), chunk_size => 40, init_relay => '', gather => sub { if (@_ == 1) { print "\r", $count++; } else { push @data, @{ $_[1] }; } }, user_begin => sub { $bkg = Imager->new(xsize=>800, ysize=>600); $bkg->filter(type=>"gradgen", xo=>[ 100, 300, 600 ], yo=>[ 100, 300, 100 ], colors=>[ qw(red blue green) ]); $bkg->filter(type=>"noise", amount=>12, subtype=>0) }, ); mce_loop { my ($mce, $chunk_ref, $chunk_id) = @_; my @i_data; for my $x (@{ $chunk_ref }) { my $i = $bkg-> copy; $i->string( text => $x, color => Imager::Color->new('ffffff'), font => Imager::Font->new( # file => '/usr/share/fonts/truetype/msttcorefonts/cour.ttf', # file => '/System/Library/Fonts/Courier.dfont', face => 'Courier New', # mswin size => 420, aa => 1), x => 25, y => 500, ); $i->write(data => \my $data, type => 'gif'); push @i_data, $data; MCE->gather($x); } MCE::relay { MCE->gather($chunk_id, \@i_data) }; } [ 0 .. 319 ]; MCE::Loop->finish; print " frame GIF done!\n"; printf "compute time: %0.3fs\n", time - $start; if ( $TEST_GD ) { my $image = GD::Image-> newFromGifData( $data[ 0 ]); my $gifdata = $image-> gifanimbegin(0,0); $gifdata .= $image-> gifanimadd( 1,0,0,1,1 ); for (1..$#data) { my $frame = GD::Image-> newFromGifData( $data[ $_ ]); $gifdata .= $frame-> gifanimadd( 1,0,0,1,1 ); } $gifdata .= $image-> gifanimend; open my $fh, '>', 'gd.gif'; binmode $fh; print $fh $gifdata; close $fh; } else { Imager->write_multi({ file => 'im.gif', type => 'gif', gif_loop => 0, gif_delay => 1, }, map { Imager->new(data => \$_) } @data) or die Imager->errstr; } printf "Total: %0.3fs\n", time - $start; __END__ 319 frame GIF done! compute time: 10.932s Total: 45.646s 319 frame GIF done! compute time: 10.738s Total: 17.107s

        1st is testing (final animation to be collected by) Imager, 2nd -- GD. Nice. But then, from here it's natural to also parallelize what's being collected in "$gifdata" -- staying in the same "mce_loop" block! 2 fragments: (1) a "push" to be replaced with (should be re-written better, it's too late here already):

        if ( $TEST_GD ) { my $gd = GD::Image-> newFromGifData( $data ); push @i_data, !$x ? $gd-> gifanimbegin . $gd-> gifanimadd( 1,0,0,1,1 ) : $gd-> gifanimadd( 1,0,0,1,1 ) } else { push @i_data, $data }

        and (2) final "if" to be replaced with:

        if ( $TEST_GD ) { open my $fh, '>', 'gd.gif'; binmode $fh; print $fh join '', @data; close $fh; } else { Imager->write_multi({ file => 'im.gif', type => 'gif', gif_loop => 0, gif_delay => 1, }, map { Imager->new(data => \$_) } @data) or die Imager->errstr; } __END__ 319 frame GIF done! compute time: 11.765s Total: 12.015s

        Now that's some "speeding up animated GIF generation". The whole test -- using Imager to generate a GIF, then GD to re-save -- is maybe somewhat artificial as an example, and may or may not be close to "real life". Just to show that parallelization (== optimization) helps as much as other tools/setup allow.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11113687]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (5)
As of 2024-04-25 14:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found