Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery

Re: Parallel-processing the code

by Marshall (Canon)
on May 17, 2018 at 00:17 UTC ( #1214699=note: print w/replies, xml ) Need Help??

in reply to Parallel-processing the code

Your application doesn't appear to be well suited for parallel processing. That's because you have a single input stream, what appears to be minimal processing and a single output DB that each thread or process would have to interact heavily with.

I get from your question that the root problem is "I want my application to run faster". You assume that multi-processing is the answer for that and you are asking us how to do it.

Let's back up a bit. Without sample data, it is a bit hard for me to figure out exactly what you are doing. How long does this app take? How big is the input file? What triggers a "new run"? It could be that if say 90% of the data is the same between runs and 10% is "new", some strategy that saves the results from the 90% that was the same from last time will result in getting the complete results faster? Anyway before jumping to a "solution", I'd like to understand a bit more about what you are doing...

It could also be that some relatively minor tweaks to your code could provide some performance increase, although I doubt anything truly dramatic.

But for example, I have one application that calculates some results hourly in a very efficient manner. At the end of the fiscal year, we recalculate everything and it takes about 6 hours. Nobody cares that one year's of data takes 6 hours to process because that work is spread out in very small increments on an hourly basis throughout the year. Update: we do the complete reprocessing as a "double check" on the incremental process. In theory the results are the same.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1214699]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (3)
As of 2020-12-01 00:42 GMT
Find Nodes?
    Voting Booth?

    No recent polls found