The better answer that I have used in the past for loosely-coordinated multi-processing scripts is to use the filesystem. I will presume that neither the data nor status file reliably arrives first, so we must detect when a pair of files is present, lock the pair, and then proceed with processing.
The simple solution (assuming all workers are running on the same node, as network filesystems can screw this up) is to create a third "flag" file, using sysopen with O_CREAT|O_EXCL from Fcntl, the successful creation of which acts as acquiring a lock for that pair. A process that fails to acquire this lock simply moves on to the next pair, confident that another worker has already claimed that pair.
There is a small potential problem here with stale flag files, but that can be remedied by either shutting the worker horde down and cleaning up any flags left (as I have used in the past when I needed this for a quick-and-dirty multi-process solution) or writing the PID into the flag file to allow a single cleanup process to remove any flags left by workers that are no longer running. When cleanup removes a flag, another worker will eventually find that pair and process it. You will need to manually supervise this kind of operation, because the most likely reason for a worker to fail to complete processing is that that pair exposes a bug in the worker code and will reliably crash the worker process.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|