Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Using TheSchwartz - 2 threads pick the same jobs. Any help ?

by gube (Parson)
on Dec 06, 2011 at 17:37 UTC ( #942068=perlquestion: print w/replies, xml ) Need Help??

gube has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks, We using TheSchwartz module in our application for Exports async job. Basically, 3 schwartz threads running parallelly to accomplish exports faster for multiple users. Occasionally, we ran into issue in production sometimes 2 threads picks the same schwartz job from the job table and processing. So, user in export excel get the same data twice. Is there a way to control / parameter in Theschwartz module ? Thanks, Gubs
  • Comment on Using TheSchwartz - 2 threads pick the same jobs. Any help ?

Replies are listed 'Best First'.
Re: Using TheSchwartz - 2 threads pick the same jobs. Any help ?
by Anonymous Monk on Dec 06, 2011 at 19:59 UTC

    Sounds like your job queue is not thread safe. Are you just pushing and popping from an array?

      "thread safety" & "arrays" don't come into it.

      The "queue" in this module appears to be a database table. And the "threads" are probably forked processes.

      I say "probably" because after 20 minutes of source diving, I'm still not sure. What I can say is that I saw no sign of threading.

      I can also say that this is the single most horrendously complex, over-engineered, stupefyingly over-architected module I've yet encountered. That doesn't mean it doesn't work, or that it might not work very well.

      Just that I wouldn't want to be the one responsible for deciding that it has been adequately tested. Or trying to track down bugs.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?

        Well how 'bout locking/unlocking them tables to prevent this from happening ?

        About TheSchwartz being over-*, well.. I actually saw a lot of people using it, and lots of jobs on featuring TheSchwartz as a required skill .. so I'd imagine we're missing something from the picture, and that it probably is a good module.

        Equally, one could easily write a distributed job queue, using Redis, MongoDB, zeromq, rabbitmq and many others.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://942068]
Approved by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (3)
As of 2020-05-27 03:57 GMT
Find Nodes?
    Voting Booth?
    If programming languages were movie genres, Perl would be:

    Results (152 votes). Check out past polls.