http://qs321.pair.com?node_id=11113692


in reply to Re^3: MCE segmentation fault
in thread MCE segmentation fault

Wow :) MCE continues to boggle my mind after all these years.

Replies are listed 'Best First'.
Re^5: MCE segmentation fault
by vr (Curate) on Mar 04, 2020 at 01:57 UTC

    Excellent demonstration as usual, marioroy!

    Well, just slightly offtopic, w/r/t/ the efforts

    to speed up animated GIF generation

    the disturbing truth is that the main time-waster, above, appears to be Imager collecting, in single thread, 999(9...) decoded GIF images into final animation. Just look what it does: GIF files/scalars are read into internal Imager format (thankfully, kept palletized i.e. not converted to "true color"), the LZW compression is discarded/forgotten, it apparently tries palette(s) optimization(?), then individual frames are LZW-compressed again, very likely to the state they were initially stored in files/scalars. What a waste. To be fair, depending on image/scene, quantization requires much more work than LZW compression, so parallelization, as is, pays off well. However, one can't help but wonder if it all can be accelerated.

    Actually, yes, some tools appear to be more fit for the job, -- GD. It also, I think, decompresses GIF frames/scalars, but then either keeps (and then uses) originals, or is just optimized 100% ++.

    The example below is marioroy's code, with a few changes to setup. I run it on 4 cores, image is slightly larger (to give a computer some work to do, maybe closer to real life use) and has "interesting" background, otherwise compression cancels our larger image size, and it may again be closer to kind of images/animations OP is working with, it seems to me, -- or at least that was an idea. Number of frames, chunk size, etc. are changed according to setup above.

    use strict; use warnings; use feature 'say'; use Imager; use MCE::Loop; use Time::HiRes 'time'; use GD; STDOUT->autoflush; my $start = time; my $count = 0; my @data; my $bkg; my $TEST_GD = $ARGV[ 0 ] // 1; MCE::Loop->init( max_workers => MCE::Util::get_ncpu(), chunk_size => 40, init_relay => '', gather => sub { if (@_ == 1) { print "\r", $count++; } else { push @data, @{ $_[1] }; } }, user_begin => sub { $bkg = Imager->new(xsize=>800, ysize=>600); $bkg->filter(type=>"gradgen", xo=>[ 100, 300, 600 ], yo=>[ 100, 300, 100 ], colors=>[ qw(red blue green) ]); $bkg->filter(type=>"noise", amount=>12, subtype=>0) }, ); mce_loop { my ($mce, $chunk_ref, $chunk_id) = @_; my @i_data; for my $x (@{ $chunk_ref }) { my $i = $bkg-> copy; $i->string( text => $x, color => Imager::Color->new('ffffff'), font => Imager::Font->new( # file => '/usr/share/fonts/truetype/msttcorefonts/cour.ttf', # file => '/System/Library/Fonts/Courier.dfont', face => 'Courier New', # mswin size => 420, aa => 1), x => 25, y => 500, ); $i->write(data => \my $data, type => 'gif'); push @i_data, $data; MCE->gather($x); } MCE::relay { MCE->gather($chunk_id, \@i_data) }; } [ 0 .. 319 ]; MCE::Loop->finish; print " frame GIF done!\n"; printf "compute time: %0.3fs\n", time - $start; if ( $TEST_GD ) { my $image = GD::Image-> newFromGifData( $data[ 0 ]); my $gifdata = $image-> gifanimbegin(0,0); $gifdata .= $image-> gifanimadd( 1,0,0,1,1 ); for (1..$#data) { my $frame = GD::Image-> newFromGifData( $data[ $_ ]); $gifdata .= $frame-> gifanimadd( 1,0,0,1,1 ); } $gifdata .= $image-> gifanimend; open my $fh, '>', 'gd.gif'; binmode $fh; print $fh $gifdata; close $fh; } else { Imager->write_multi({ file => 'im.gif', type => 'gif', gif_loop => 0, gif_delay => 1, }, map { Imager->new(data => \$_) } @data) or die Imager->errstr; } printf "Total: %0.3fs\n", time - $start; __END__ 319 frame GIF done! compute time: 10.932s Total: 45.646s 319 frame GIF done! compute time: 10.738s Total: 17.107s

    1st is testing (final animation to be collected by) Imager, 2nd -- GD. Nice. But then, from here it's natural to also parallelize what's being collected in "$gifdata" -- staying in the same "mce_loop" block! 2 fragments: (1) a "push" to be replaced with (should be re-written better, it's too late here already):

    if ( $TEST_GD ) { my $gd = GD::Image-> newFromGifData( $data ); push @i_data, !$x ? $gd-> gifanimbegin . $gd-> gifanimadd( 1,0,0,1,1 ) : $gd-> gifanimadd( 1,0,0,1,1 ) } else { push @i_data, $data }

    and (2) final "if" to be replaced with:

    if ( $TEST_GD ) { open my $fh, '>', 'gd.gif'; binmode $fh; print $fh join '', @data; close $fh; } else { Imager->write_multi({ file => 'im.gif', type => 'gif', gif_loop => 0, gif_delay => 1, }, map { Imager->new(data => \$_) } @data) or die Imager->errstr; } __END__ 319 frame GIF done! compute time: 11.765s Total: 12.015s

    Now that's some "speeding up animated GIF generation". The whole test -- using Imager to generate a GIF, then GD to re-save -- is maybe somewhat artificial as an example, and may or may not be close to "real life". Just to show that parallelization (== optimization) helps as much as other tools/setup allow.

      Greetings vr,

      Cool gradient background! I tried adding frames simultaneously while running by modifying the gather code block.

      gather => sub { if (@_ == 1) { print "\r", $count++; } else { if (!defined $gifdata) { $image = GD::Image-> newFromGifData( shift @{ $_[1] } ); $gifdata = $image-> gifanimbegin( 0,0 ); $gifdata .= $image-> gifanimadd( 1,0,0,1,1 ); } while ( my $data = shift @{ $_[1] } ) { my $frame = GD::Image-> newFromGifData( $data ); $gifdata .= $frame-> gifanimadd( 1,0,0,1,1 ); } } },

      That results in workers waiting for the manager process which is fine (i.e. the manager process is now the bottleneck). In that case chunk_size 1 may be faster to have the manager process quickly add one frame. Thereby able to respond to other workers with lesser latency.

      use strict; use warnings; use feature 'say'; use Imager; use MCE::Loop; use Time::HiRes 'time'; use GD; STDOUT->autoflush; my $start = time; my $count = 0; my $bkg; my $image; my $gifdata; MCE::Loop->init( max_workers => MCE::Util::get_ncpu(), chunk_size => 1, init_relay => '', gather => sub { if (@_ == 1) { print "\r", $count++; } else { if (!defined $gifdata) { $image = GD::Image-> newFromGifData( shift @{ $_[1] } ); $gifdata = $image-> gifanimbegin( 0,0 ); $gifdata .= $image-> gifanimadd( 1,0,0,1,1 ); } while ( my $data = shift @{ $_[1] } ) { my $frame = GD::Image-> newFromGifData( $data ); $gifdata .= $frame-> gifanimadd( 1,0,0,1,1 ); } } }, user_begin => sub { $bkg = Imager->new(xsize=>800, ysize=>600); $bkg->filter(type=>"gradgen", xo=>[ 100, 300, 600 ], yo=>[ 100, 300, 100 ], colors=>[ qw(red blue green) ]); $bkg->filter(type=>"noise", amount=>12, subtype=>0) }, ); mce_loop { my ($mce, $chunk_ref, $chunk_id) = @_; my @i_data; for my $x (@{ $chunk_ref }) { my $i = $bkg-> copy; $i->string( text => $x, color => Imager::Color->new('ffffff'), font => Imager::Font->new( # file => '/usr/share/fonts/truetype/msttcorefonts/cour.ttf', # + Ubuntu # file => '/System/Library/Fonts/Courier.dfont', face => 'Courier New', # mswin size => 420, aa => 1), x => 25, y => 500, ); $i->write(data => \my $data, type => 'gif'); push @i_data, $data; MCE->gather($x); } MCE::relay { MCE->gather($chunk_id, \@i_data) }; } [ 0 .. 319 ]; MCE::Loop->finish; print " frame GIF done!\n"; printf "compute time: %0.3fs\n", time - $start; $gifdata .= $image-> gifanimend; open my $fh, '>:raw', 'gd.gif'; print $fh $gifdata; close $fh; printf "Total: %0.3fs\n", time - $start;

      Regards, Mario

        MCE::Channel is included in recent MCE releases. MCE::Flow allows running multiple tasks simultaneously.

        So here is a great demonstration for MCE::Flow and MCE::Channel. The benefit is that the manager process does little work and better able to accommodate workers requesting input with minimum latency. The bottleneck is now the gifout task which is a separate worker. Notice user_begin and user_end called for the first task only by checking MCE->task_id. Notice also max_workers taking an array ref for specifying how many workers for each task.

        The following demonstration was made possible by PerlMonks. Thanks vr for providing GD tips. Thank you Anonymous Monk for posting.

        use strict; use warnings; use feature 'say'; use Imager; use MCE::Flow; use MCE::Channel; use Time::HiRes 'time'; use GD; STDOUT->autoflush; my $start = time; my $chnl = MCE::Channel->new(); my $count = 0; my $bkg; MCE::Flow->init( max_workers => [ MCE::Util::get_ncpu() - 1, 1 ], chunk_size => 1, init_relay => '', gather => sub { print "\r", $count++; }, user_begin => sub { if (MCE->task_id == 0) { $bkg = Imager->new(xsize=>800, ysize=>600); $bkg->filter(type=>"gradgen", xo=>[ 100, 300, 600 ], yo=>[ 100, 300, 100 ], colors=>[ qw(red blue green) ]); $bkg->filter(type=>"noise", amount=>12, subtype=>0) } }, user_end => sub { if (MCE->task_id == 0) { $chnl->end; } } ); sub task_frames { my ($mce, $chunk_ref, $chunk_id) = @_; my @i_data; for my $x (@{ $chunk_ref }) { my $i = $bkg-> copy; $i->string( text => $x, color => Imager::Color->new('ffffff'), font => Imager::Font->new( # file => '/usr/share/fonts/truetype/msttcorefonts/cour.ttf', # + Ubuntu # file => '/System/Library/Fonts/Courier.dfont', face => 'Courier New', # mswin size => 420, aa => 1), x => 25, y => 500, ); $i->write(data => \my $data, type => 'gif'); push @i_data, $data; MCE->gather($x); } MCE::relay { $chnl->send(\@i_data) }; } sub task_gifout { my $image; my $gifdata; while ( my $i_data_ref = $chnl->recv ) { if (!defined $gifdata) { $image = GD::Image-> newFromGifData( shift @{ $i_data_ref } ) +; $gifdata = $image-> gifanimbegin( 0,0 ); $gifdata .= $image-> gifanimadd( 1,0,0,1,1 ); } while ( my $data = shift @{ $i_data_ref } ) { my $frame = GD::Image-> newFromGifData( $data ); $gifdata .= $frame-> gifanimadd( 1,0,0,1,1 ); } } $gifdata .= $image-> gifanimend; open my $fh, '>:raw', 'gd.gif'; print $fh $gifdata; close $fh; } mce_flow \&task_frames, \&task_gifout, [ 0 .. 319 ]; MCE::Flow->finish; print " frame GIF done!\n"; printf "compute time: %0.3fs\n", time - $start;

        There is a lot of parallel going on :) Although workers complete sooner, the manager process still waits for all tasks to finish before MCE::Flow exits.

        Regards, Mario