Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things

Re: Semaphore puzzle

by jcb (Parson)
on Feb 23, 2021 at 02:29 UTC ( [id://11128677] : note . print w/replies, xml ) Need Help??

in reply to Semaphore puzzle

The better answer that I have used in the past for loosely-coordinated multi-processing scripts is to use the filesystem. I will presume that neither the data nor status file reliably arrives first, so we must detect when a pair of files is present, lock the pair, and then proceed with processing.

The simple solution (assuming all workers are running on the same node, as network filesystems can screw this up) is to create a third "flag" file, using sysopen with O_CREAT|O_EXCL from Fcntl, the successful creation of which acts as acquiring a lock for that pair. A process that fails to acquire this lock simply moves on to the next pair, confident that another worker has already claimed that pair.

There is a small potential problem here with stale flag files, but that can be remedied by either shutting the worker horde down and cleaning up any flags left (as I have used in the past when I needed this for a quick-and-dirty multi-process solution) or writing the PID into the flag file to allow a single cleanup process to remove any flags left by workers that are no longer running. When cleanup removes a flag, another worker will eventually find that pair and process it. You will need to manually supervise this kind of operation, because the most likely reason for a worker to fail to complete processing is that that pair exposes a bug in the worker code and will reliably crash the worker process.