Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Re: Can Perl write multiple files at the same time?

by biohisham (Priest)
on Oct 20, 2014 at 22:14 UTC ( #1104501=note: print w/replies, xml ) Need Help??

in reply to Can Perl write multiple files at the same time?

You might as well give the module Many Core Engines (MCE) a go.

A 4 year old monk

Replies are listed 'Best First'.
Re^2: Can Perl write multiple files at the same time?
by marioroy (Parson) on Dec 13, 2014 at 02:39 UTC
    The MCE 1.515 was moved to back pan some time back. Here is the URL.

    There is another example Re^4: Using MCE to write to multiple files..

    The MCE->sendto method is available. A nice thing is that MCE caches the file handle, therefore only opening the file handle once during the run. The sendto method appends, thus the reason for unlink.

    Below, using the Flow model as input_data is not required to run.

    use MCE::Flow max_workers => 8; my $file1 = "/path/to/file1.txt"; my $file2 = "/path/to/file2.txt"; unlink $file1 if -e $file1; unlink $file2 if -e $file2; mce_flow sub { my $wid = MCE->wid; if (MCE->wid % 2 == 0) { MCE->sendto("file:$file1", "sunny day from $wid\n"); } else { MCE->sendto("file:$file2", "raining day from $wid\n"); } };

    Another way is via MCE->print, MCE->printf, MCE->say. MCE serializes data from workers to the manager process. The output is done by the manager process. These methods are beneficial when many workers write to same file simultaneously.

    use MCE::Loop chunk_size => 1, max_workers => 8; my @input = (100..199); open my $fh_1, '>', '/path/to/file1.txt'; open my $fh_2, '>', '/path/to/file2.txt'; mce_loop { my ($mce, $chunk_ref, $chunk_id) = @_; if ($chunk_id % 2 == 0) { MCE->say($fh_1, "id: $chunk_id: input $_"); } else { MCE->say($fh_2, "id: $chunk_id: input $_"); } } @input; close $fh_1; close $fh_2;

    In case the reader missed it, the syntax for mce_flow requires the sub in front of the opening brace unlike mce_loop which takes a block. The reason is that mce_flow can take many anonymous blocks; e.g. mce_flow sub { ... }, sub { ... }.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1104501]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (1)
As of 2021-10-21 04:24 GMT
Find Nodes?
    Voting Booth?
    My first memorable Perl project was:

    Results (82 votes). Check out past polls.