Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??

The better answer that I have used in the past for loosely-coordinated multi-processing scripts is to use the filesystem. I will presume that neither the data nor status file reliably arrives first, so we must detect when a pair of files is present, lock the pair, and then proceed with processing.

The simple solution (assuming all workers are running on the same node, as network filesystems can screw this up) is to create a third "flag" file, using sysopen with O_CREAT|O_EXCL from Fcntl, the successful creation of which acts as acquiring a lock for that pair. A process that fails to acquire this lock simply moves on to the next pair, confident that another worker has already claimed that pair.

There is a small potential problem here with stale flag files, but that can be remedied by either shutting the worker horde down and cleaning up any flags left (as I have used in the past when I needed this for a quick-and-dirty multi-process solution) or writing the PID into the flag file to allow a single cleanup process to remove any flags left by workers that are no longer running. When cleanup removes a flag, another worker will eventually find that pair and process it. You will need to manually supervise this kind of operation, because the most likely reason for a worker to fail to complete processing is that that pair exposes a bug in the worker code and will reliably crash the worker process.


In reply to Re: Semaphore puzzle by jcb
in thread Semaphore puzzle by jerryhone

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others contemplating the Monastery: (3)
As of 2024-04-24 04:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found