Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Re: how do I create parrallel processing in this script?

by tirwhan (Abbot)
on Feb 05, 2010 at 09:40 UTC ( #821526=note: print w/replies, xml ) Need Help??

in reply to how do I create parrallel processing in this script?

Additionally to moritz's correct answer, I'll caution that you are unlikely to experience a huge difference in processing time by parallelizing the input operation. IO is limited by the disk speed, and turning a single process reading in one file at a time into several processes that read these files in parallel will likely affect performance detrimentally, because the disk head has to constantly jump to a new position while reading in. So, unless you have a very unusual drive configuration (or weird limits set on your single process by the OS), this will probably slow down the entire process, rather than speeding it up.

Also, reading in the file into an array in a forked child and then printing that array in the parent won't work, data structures don't remain shared in a forked process, you need some form of IPC for that.

All dogma is stupid.

Replies are listed 'Best First'.
Re^2: how do I create parrallel processing in this script?
by avanta (Beadle) on Feb 05, 2010 at 10:17 UTC
    Thanks a lot for the suggestion. I think I would be needing another help, in this code i wanted to read files and split into threads the process and then join the output of the threads into one file. How an I do it other than using fork(),

      Hmm. As I said above, there's not much point in doing this, because you almost certainly won't improve performance. However, if you did want to do this you could use one of the following methods:

      - print to the output file from the children instead of the parent. You'll need to lock the output file in some way (see "perldoc -q lock" for some ways on doing that) so that the writes are not completely jumbled.

      - use some form of Inter Process Communication to read the file content into the parent process. Read the documentation I linked to or search cpan for IPC for various ways of doing this.

      - use threads and a shared variable instead of fork.

      All dogma is stupid.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://821526]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2022-08-10 06:11 GMT
Find Nodes?
    Voting Booth?

    No recent polls found