Your application doesn't appear to be well suited for parallel processing. That's because you have a single input stream, what appears to be minimal processing and a single output DB that each thread or process would have to interact heavily with.

I get from your question that the root problem is "I want my application to run faster". You assume that multi-processing is the answer for that and you are asking us how to do it.

Let's back up a bit. Without sample data, it is a bit hard for me to figure out exactly what you are doing. How long does this app take? How big is the input file? What triggers a "new run"? It could be that if say 90% of the data is the same between runs and 10% is "new", some strategy that saves the results from the 90% that was the same from last time will result in getting the complete results faster? Anyway before jumping to a "solution", I'd like to understand a bit more about what you are doing...

It could also be that some relatively minor tweaks to your code could provide some performance increase, although I doubt anything truly dramatic.

But for example, I have one application that calculates some results hourly in a very efficient manner. At the end of the fiscal year, we recalculate everything and it takes about 6 hours. Nobody cares that one year's of data takes 6 hours to process because that work is spread out in very small increments on an hourly basis throughout the year. Update: we do the complete reprocessing as a "double check" on the incremental process. In theory the results are the same.