Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re^3: Reliable Work Queue Manager

by ph713 (Pilgrim)
on Oct 27, 2005 at 18:20 UTC ( [id://503432]=note: print w/replies, xml ) Need Help??


in reply to Re^2: Reliable Work Queue Manager
in thread Reliable Work Queue Manager

Having a truly transactionally coherent ACID relational database at your disposal of course makes things like persistent jobs queues a breeze. Then you just need a "jobs" table with an autoincrementing serial number for a key and you're pretty much done.

But I'd be careful of thinking that using DBM-based solutions buys you the same gaurantees as a real database, I would imagine there are a lot of corner cases for recovery that don't work out so well in certain failure scenarios.

But, IMHO, relying on having a full-blown RDBMS installed anwhere you use your queueing library isn't very good planning either. A lot of good n-tier application architectures can be built assuming you have a disk-persistent network-transparent queueing system in place that you can run on arbitrary machines without the support of an RDBMS or specific dbm implementation, along the lines of IBM MQseries in terms of functionality. You don't want to have to put local RDBMS's on every node involved in the architecture in order for it to work.

My opinion is that for a general-purpose queueing module one should build a solution based on queue directory management using whatever atomic and/or locking resources you have available. POSIX specifies atomic rename() and good fcntl() locks. Then you can deploy the module in any reasonable environment which implements the POSIX primitives you depend on (excuse my *nix centricity, I don't take Windows seriously at all, and I'm not really sure whether it supports POSIX in that regard).

Come to think of it, sendmail/postfix + (E)SMTP amounts to a disk-persistent network-transparent job queueing system using POSIX primitives and a queue directory on the local machines. It makes a fine example, spam problems notwithstanding :)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://503432]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (3)
As of 2024-04-26 02:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found