decoupling processes via queues: advice wanted

morgon has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

in my current project we will have several cooperating Perl-processes running on several machines.

What we are currently thinking about is not having these processes talk directly to each other but decoupling them via a queuing middle-tier in which each processes would communicate indirectly via (persistent) queues.

We are running on Linux and use Oracle for the backend.

The question is now how to implement such a queuing infrastructure.

I can currently see four ways for us to achieve this:

1) Roll your own.

2) Use Spread.

3) Use database-tables are queues.

4) Use Oracle Advanced Queuing.

Currently I am trying to evalute options 2-4 and would welcome any feedback on specific pros and cons that more experienced monks see.

Personally I quite like Spread (so far - I don't have a lot of experience with it), however I am also seeing the advantages of Oracle Advanced Queuing as it would permit us to have a process dequeue a message, do it's business-logic, update some database-state and finally enqueue new messages all in one database-transaction.

So could you please provide some enlightenment?

Many thanks

Comment on decoupling processes via queues: advice wanted

Replies are listed 'Best First'.
Re: decoupling processes via queues: advice wanted by rcaputo (Chaplain) on Apr 19, 2009 at 18:33 UTC
You may avoid rolling your own by looking on CPAN. There are a few message+queue modules, some of which seem to be well-developed. `-- Rocco Caputo - http://poe.perl.org/`	[reply]
Re: decoupling processes via queues: advice wanted by dHarry (Abbot) on Apr 19, 2009 at 18:19 UTC
You´re not to clear on why you want to do it. If you´re having problems with your current implementation you might want to investigate the cause. Under the assumption that decoupling indeed is a good idea: Sounds to me like you want a message queue. I would not try to implement functionality like this yourself. My experience is limited to MQ Series (nowadays called WebSphere MQ). If I understand correctly Oracle Advanced Queuing is a similar product. So this is one alternative to research. Instead of going for broke it might be wise to do a small proof of concept. Messaging is not a silver bullet. I don´t know Oracle AQ, I assume you have to spend money on it. There are several free/open source products on the market. See Message queue for somepointers. I advice to do a small PoC to see if it will work for you	[reply]
Re^2: decoupling processes via queues: advice wanted by morgon (Priest) on Apr 19, 2009 at 18:38 UTC
decoupling indeed is a good idea I believe it is a good idea. What we want is that processes to not depend on other processes to be running. In our scenario the processes don't know about each other, they know only about the queues they use to either receive messages from or to post messages to. Sounds to me like you want a message queue. Evidently - that's what I said in the title, or didn't I? My experience is limited to MQ Series I've used MQ and would like to use it again but we don't have it and won't get it. If it was an option I would have listed it as number 5. Oracle Advanced Queuing is a similar product. So this is one alternative to research. This is why I posted this message...I had hoped for some feedback of people who have used it. I don´t know Oracle AQ, I assume you have to spend money on it It is part of Oracle. If you pay for Oracle you get AQ for free - that's why it is an option for us. There are several free/open source products on the market For the time being I would like to restict the discussions to the above listed options... But many thanks anyway.	[reply]
Re^3: decoupling processes via queues: advice wanted by dHarry (Abbot) on Apr 20, 2009 at 07:21 UTC
I believe it is a good idea. It might be a great idea but I cannot judge that based on the information you provide. I don’t know what the Perl scripts are doing and why they communicate. I would just like to say that not all synchronous communication can be changed to asynchronous. In general asynchronous messaging allows for more parallelism because of the non-blocking nature but it also introduces issues like what happens if a message cannot be delivered? Or maybe the receiver needs to return some transaction id to the sender? etc. I have seen home grown messaging middleware developed at a time when messaging products were not available. Nowadays I would take an existing product rather than reinventing wheels. Dedicated Message Oriented Middleware normally takes care of much of the complexity. If Oracle AQ is available at no extra cost why not give it a try? I read up on Oracle AQ, and it is implemented in database tables (your option 3). Cheers, dHarry	[reply]
Re^4: decoupling processes via queues: advice wanted by morgon (Priest) on Apr 20, 2009 at 09:31 UTC
Re: decoupling processes via queues: advice wanted by andreas1234567 (Vicar) on Apr 20, 2009 at 12:14 UTC
The Message queue (wikipedia article) lists 5-6 other open source alternatives to the ones already listed. -- No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]	[reply]
Re: decoupling processes via queues: advice wanted by mrons (Beadle) on Apr 21, 2009 at 04:04 UTC
I've implemented a queuing infrastructure using IPC::DirQueue Some of the other persistent queue implementations on CPAN don't allow multiple processes accessing queues. IPC::DirQueue can work with multiple processes and even multiple hosts via NFS. Also the CPAN queue implementations I've looked at don't have the idea of a job on the queue being active, they only do push and shift. An active job stays on the queue until you say "I've finished with that job". Other processes don't pick up the job while active. Great module. It may fit your needs if you don't want to dive into oracle queues.	[reply]

Back to Seekers of Perl Wisdom