Using TheSchwartz - 2 threads pick the same jobs. Any help ?

gube has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks, We using TheSchwartz module in our application for Exports async job. Basically, 3 schwartz threads running parallelly to accomplish exports faster for multiple users. Occasionally, we ran into issue in production sometimes 2 threads picks the same schwartz job from the job table and processing. So, user in export excel get the same data twice. Is there a way to control / parameter in Theschwartz module ? Thanks, Gubs

Comment on Using TheSchwartz - 2 threads pick the same jobs. Any help ?

Replies are listed 'Best First'.
Re: Using TheSchwartz - 2 threads pick the same jobs. Any help ? by Anonymous Monk on Dec 06, 2011 at 19:59 UTC
Sounds like your job queue is not thread safe. Are you just pushing and popping from an array?	[reply]
Re^2: Using TheSchwartz - 2 threads pick the same jobs. Any help ? by BrowserUk (Patriarch) on Dec 06, 2011 at 21:08 UTC
"thread safety" & "arrays" don't come into it. The "queue" in this module appears to be a database table. And the "threads" are probably forked processes. I say "probably" because after 20 minutes of source diving, I'm still not sure. What I can say is that I saw no sign of threading. I can also say that this is the single most horrendously complex, over-engineered, stupefyingly over-architected module I've yet encountered. That doesn't mean it doesn't work, or that it might not work very well. Just that I wouldn't want to be the one responsible for deciding that it has been adequately tested. Or trying to track down bugs. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]
Re^3: Using TheSchwartz - 2 threads pick the same jobs. Any help ? by spx2 (Deacon) on Dec 07, 2011 at 13:08 UTC
Well how 'bout locking/unlocking them tables to prevent this from happening ? About TheSchwartz being over-*, well.. I actually saw a lot of people using it, and lots of jobs on jobs.perl.org featuring TheSchwartz as a required skill .. so I'd imagine we're missing something from the picture, and that it probably is a good module. Equally, one could easily write a distributed job queue, using Redis, MongoDB, zeromq, rabbitmq and many others.	[reply]
Re^4: Using TheSchwartz - 2 threads pick the same jobs. Any help ? (locks--) by tye (Sage) on Dec 07, 2011 at 14:41 UTC
Re^5: Using TheSchwartz - 2 threads pick the same jobs. Any help ? (locks--) by Tanktalus (Canon) on Dec 07, 2011 at 15:17 UTC
Some notes below your chosen depth have not been shown here
Re^4: Using TheSchwartz - 2 threads pick the same jobs. Any help ? by BrowserUk (Patriarch) on Dec 07, 2011 at 22:07 UTC

Back to Seekers of Perl Wisdom