Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Message Queue library

by pwagyi (Monk)
on Jun 21, 2018 at 01:49 UTC ( [id://1217087] : perlquestion . print w/replies, xml ) Need Help??

pwagyi has asked for the wisdom of the Perl Monks concerning the following question:

Greeting Monks! I would like to seek advice from senior monks on this matter. I am developing automatic data re-processing system. Basically it is pretty similar to TCP data re-transmission.

A process will generate a bunch of data di (d1,d2,d3...dn), and it must be transferred to receiver process. Sender sends di+1 if receiver reply di (with OK) otherwise, Sender will re-send di again. Basically Receiver should receive data in order (d1,d2,..dn) and sender should at the end received replies with (OK) in order (d1,d2,..dn) Note: Sender and receiver are different processes (potentially across network) and there are time gaps between data generation (d1, 1/2 hour ,d2 , 1 hr, d3 ..)

My plan currently is as follow: There will be persitence queue with some sort of locking mechanism. I have found some modules ( but it does not seem to provide locking the queue.

OnDataGenerationEvent(Data d) - queue.add(d, {state => PENDING_TRANSFER} ) process(queue) OnDataReceivedEvent(Data reply) - lock(queue) { d = dequeue() if reply.status = OK then # no Op else # status is NAK # add back to front of queue queue.add_front(d,{state => PENDING_TRANSFER}) endif } process(queue) sub process(queue) lock(queue ){ d = dequeue(queue) if d.state = PENDING_TRANSFER then transfer(d) queue.add(d, {state => PENDING_REPLY } ) endif }

Replies are listed 'Best First'.
Re: Message Queue library
by haukex (Archbishop) on Jun 21, 2018 at 07:54 UTC

    I think that a decent message queue should guarantee "transmission" of messages - under what conditions would you expect to lose messages? Maybe you could expand on your explanation, because I also don't understand what's wrong with just using a TCP connection?

Re: Message Queue library
by Anonymous Monk on Jun 21, 2018 at 16:39 UTC
    Why does a queue need a lock? Isn't the entire idea that it's a flexible buffer where readers wait for something to read and writers maybe wait for space in which to write?
A reply falls below the community's threshold of quality. You may see it by logging in.