Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re^2: Easily parallelize program execution

by bennymack (Pilgrim)
on Aug 24, 2006 at 02:43 UTC ( [id://569270]=note: print w/replies, xml ) Need Help??


in reply to Re: Easily parallelize program execution
in thread Easily parallelize program execution

I'm afraid I can't think of any other examples off the top of my head that will run anywhere but on my system.

I like the way this example illustrates a couple of principals. It shows that the script only takes as long as the longest process. It also demonstrates how one can aggregate the results of each process.

I don't think it takes too much imagination to see how this can be useful. For instance, it could be used to connect to 100 servers, grep their access log for a pattern, return the line count, then display the total line count. Or, on a 4 processor system, look through a directory of gzipped files 4 files at a time, do something useful, then return the aggregate result.

It's not limited to aggregation either. One could create a complex data structure also. For instance, building upon the gzip example, look for a particular query string parameter in a directory of gzipped log files and generate a hash of the values of the params and the count of times they appear.

Ok so that last one was kind of aggregation too. Of course, the reduce_sub is optional. One can simply print the results of each task...

  • Comment on Re^2: Easily parallelize program execution

Replies are listed 'Best First'.
Re^3: Easily parallelize program execution
by BrowserUk (Patriarch) on Aug 24, 2006 at 02:58 UTC

    Thankyou for taking my request seriously. I agree that it is difficult to come up with good examples that are simple enough to allow the unfamiliar to see what's going on but complex enough to demonstrate the api.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re^3: Easily parallelize program execution
by pemungkah (Priest) on Sep 12, 2006 at 23:15 UTC
    I'll try using it to parallelize some testing we're doing and see if it works for that. Should be a big runtime win if so.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://569270]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (7)
As of 2024-04-24 09:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found