Re^2: Adding parallel processing to a working Perl script

Thank you for your reply, zentara.

Ah, this is my dilemma. I get profoundly odd, inconsistent behavior.

First, here's a successful run with the forking code commented out:

C:\>perl CountFilesRecords.pl ABCD\???\Z*
ExportVolumeFolder,TotalDATRecords,TotalTextFiles,TotalLFPRecords,Tota
+lImageFiles
ABCD/001/Z000003014V001,6,6,88,88
ABCD/002/Z000003015V001,66,66,201,201
ABCD/003/Z000003079V001,1,1,27,27
ABCD/004/Z000003080V001,1,1,32767,32767
ABCD/005/Z000003081V001,1,1,14297,14297
ABCD/006/Z000002503V001,9,9,45,45
ABCD/007/Z000002780V001,2106,2106,2907,2907
ABCD/008/Z000003020V001,49,49,51,51
ABCD/009/Z000003021V001,5,5,6,6
ABCD/010/Z000003069V001,2851,2851,4576,4576
ABCD/011/Z000003071V001,1259,1259,3242,3242
ABCD/012/Z000005594V001,439,439,708,708
ABCD/013/Z000003140V001,1,1,25,25
ABCD/014/Z000003141V001,1,1,275,275
ABCD/015/Z000003142V001,2,2,14,14
ABCD/016/Z000003143V001,1,1,36,36
ABCD/017/Z000003144V001,10,10,316,316
ABCD/018/Z000003145V001,2,2,835,835

C:\>
[download]

Now, here's a sequence of runs, one immediately after the other, with the forking code restored:

C:\>perl CountFilesRecords.pl ABCD\???\Z*
ExportVolumeFolder,TotalDATRecords,TotalTextFiles,TotalLFPRecords,Tota
+lImageFiles
ABCD/001/Z000003014V001,6,,88,
ABCD/005/Z000003081V001,1,,14297,
ABCD/004/Z000003080V001,1,1,32767,32767

C:\>perl CountFilesRecords.pl ABCD\???\Z*
ExportVolumeFolder,TotalDATRecords,TotalTextFiles,TotalLFPRecords,Tota
+lImageFiles
ABCD/004/Z000003080V001,1,1,32767,32767

C:\>perl CountFilesRecords.pl ABCD\???\Z*
ExportVolumeFolder,TotalDATRecords,TotalTextFiles,TotalLFPRecords,Tota
+lImageFiles
ABCD/002/Z000003015V001,-1,,201,201
ABCD/003/Z000003079V001,1,1,,27
ABCD/001/Z000003014V001,6,6,88,88
ABCD/006/Z000002503V001,9,9,45,45
ABCD/008/Z000003020V001,-1,49,51,51
ABCD/009/Z000003021V001,5,5,6,6
ABCD/007/Z000002780V001,2106,2106,2907,2907
ABCD/011/Z000003071V001,1259,1259,,3242
ABCD/012/Z000005594V001,439,439,,708
Error from open(IO::Handle=GLOB(0x2acc2e8), <&STDIN): Bad file descrip
+tor at C:/Perl64/site/lib/Capture/Tiny.pm line 99
        Capture::Tiny::_open('IO::Handle=GLOB(0x2acc2e8)', '<&STDIN') 
+called at C:/Perl64/site/lib/Capture/Tiny.pm line 176
        Capture::Tiny::_copy_std() called at C:/Perl64/site/lib/Captur
+e/Tiny.pm line 346
        Capture::Tiny::_capture_tee(1, 0, 0, 0, 'CODE(0x2a1e7a8)') cal
+led at CountFilesRecords.pl line 113
        main::probe_volume('HASH(0x2962728)') called at CountFilesReco
+rds.pl line 99
ABCD/004/Z000003080V001,1,,32767,

C:\>perl CountFilesRecords.pl ABCD\???\Z*
ExportVolumeFolder,TotalDATRecords,TotalTextFiles,TotalLFPRecords,Tota
+lImageFiles
ABCD/001/Z000003014V001,6,6,88,88
[download]

You can plainly see the bizarre, inconsistent output from one run to the next. The last run stalled. In fact, it's still running as I type this, neither finishing nor producing more output.

This is why I'd hoped some kind brother on PerlMonks—one who has much more experience using Parallel::ForkManager than I do—might look at this script and immediately recognize what's wrong with it.

Comment on Re^2: Adding parallel processing to a working Perl script Select or Download Code

Replies are listed 'Best First'.

Re^3: Adding parallel processing to a working Perl script
by BrowserUk (Patriarch) on Apr 21, 2014 at 15:58 UTC

Error from open(IO::Handle=GLOB(0x2acc2e8), <&STDIN): Bad file descrip +tor at C:/Perl64/site/lib/Capture/Tiny.pm line 99
[download]

The problem is that each of the 'fork's, it re-opening the same glob for stdin. Under unix, this works because each fork is a new process, so the re-used glob is unique within its own process space.

But under windows, each 'fork' is actually just a separate thread, within the same process space, so the re-used glob -- despite that it is cloned at the Perl level -- is trying to concurrently reuse the same underlying per-process OS buffers and data-structures; with the inevitable consequences.

When it works, it is by pure chance. Mostly it won't.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]
[d/l]


Perl-Sensitive Sunglasses
	PerlMonks