in reply to Re^3: MCE segmentation fault
in thread MCE segmentation fault
Wow :) MCE continues to boggle my mind after all these years.
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^5: MCE segmentation fault
by vr (Curate) on Mar 04, 2020 at 01:57 UTC | |
Excellent demonstration as usual, marioroy! Well, just slightly offtopic, w/r/t/ the efforts to speed up animated GIF generation the disturbing truth is that the main time-waster, above, appears to be Imager collecting, in single thread, 999(9...) decoded GIF images into final animation. Just look what it does: GIF files/scalars are read into internal Imager format (thankfully, kept palletized i.e. not converted to "true color"), the LZW compression is discarded/forgotten, it apparently tries palette(s) optimization(?), then individual frames are LZW-compressed again, very likely to the state they were initially stored in files/scalars. What a waste. To be fair, depending on image/scene, quantization requires much more work than LZW compression, so parallelization, as is, pays off well. However, one can't help but wonder if it all can be accelerated. Actually, yes, some tools appear to be more fit for the job, -- GD. It also, I think, decompresses GIF frames/scalars, but then either keeps (and then uses) originals, or is just optimized 100% ++. The example below is marioroy's code, with a few changes to setup. I run it on 4 cores, image is slightly larger (to give a computer some work to do, maybe closer to real life use) and has "interesting" background, otherwise compression cancels our larger image size, and it may again be closer to kind of images/animations OP is working with, it seems to me, -- or at least that was an idea. Number of frames, chunk size, etc. are changed according to setup above.
1st is testing (final animation to be collected by) Imager, 2nd -- GD. Nice. But then, from here it's natural to also parallelize what's being collected in "$gifdata" -- staying in the same "mce_loop" block! 2 fragments: (1) a "push" to be replaced with (should be re-written better, it's too late here already):
and (2) final "if" to be replaced with:
Now that's some "speeding up animated GIF generation". The whole test -- using Imager to generate a GIF, then GD to re-save -- is maybe somewhat artificial as an example, and may or may not be close to "real life". Just to show that parallelization (== optimization) helps as much as other tools/setup allow. | [reply] [Watch: Dir/Any] [d/l] [select] |
by marioroy (Prior) on Mar 04, 2020 at 04:36 UTC | |
Greetings vr, Cool gradient background! I tried adding frames simultaneously while running by modifying the gather code block.
That results in workers waiting for the manager process which is fine (i.e. the manager process is now the bottleneck). In that case chunk_size 1 may be faster to have the manager process quickly add one frame. Thereby able to respond to other workers with lesser latency.
Regards, Mario | [reply] [Watch: Dir/Any] [d/l] [select] |
by marioroy (Prior) on Mar 04, 2020 at 05:33 UTC | |
MCE::Channel is included in recent MCE releases. MCE::Flow allows running multiple tasks simultaneously. So here is a great demonstration for MCE::Flow and MCE::Channel. The benefit is that the manager process does little work and better able to accommodate workers requesting input with minimum latency. The bottleneck is now the gifout task which is a separate worker. Notice user_begin and user_end called for the first task only by checking MCE->task_id. Notice also max_workers taking an array ref for specifying how many workers for each task. The following demonstration was made possible by PerlMonks. Thanks vr for providing GD tips. Thank you Anonymous Monk for posting.
There is a lot of parallel going on :) Although workers complete sooner, the manager process still waits for all tasks to finish before MCE::Flow exits. Regards, Mario | [reply] [Watch: Dir/Any] [d/l] |