http://qs321.pair.com?node_id=714818

suaveant has asked for the wisdom of the Perl Monks concerning the following question:

I have a report writer that was designed to fork for better speed, it takes chunks of the report and runs them in parallel, writing the results to a file which is later sorted. When we were on Solaris we had no known issues with this, but when we moved to Linux all of a sudden we started getting inconsistencies in the data. Basically lines were getting cut off, lost, etc... it would seem the children were stomping on each other's toes.

I have been hacking at this for half a day and nothing is working. I build chunks of data, all the lines for my child's output, flock the output file and write the chunk to it. Originally I use open and print with flock, then I tried manually flushing the file handle (which apparently flock already does anyway), I tried converting everything to sysopen and syswrite. I made sure LOCK_EX and LOCK_UN were defined... I just can't figure out what is going on. I though flock was fine as long as everything was using it....

Here is the code snippet I am currently using to no avail:

flock( $rpt, LOCK_EX ); # seek($rpt, 0, 2); # found this in PerlIO, didn't help. sysseek broke + everything for some reason syswrite($rpt,$repout,length($repout)); flock( $rpt, LOCK_UN );
Any ideas? Or even better ways to write data to the file that is fork-safe?

                - Ant
                - Some of my best work - (1 2 3)

Replies are listed 'Best First'.
Re: What the flock!? Concurrency issues in file writing.
by kyle (Abbot) on Oct 01, 2008 at 15:59 UTC

    I would start with code in the flock documentation and work from there.

    I would also try to be sure that nobody else is trying to write to the file. You might be able to ensure that by changing the filename that you write to. That is, instead of writing to "report", change it to "report-locksafe" in the places where you're doing locking. If you see something show up in "report" anyway, there's another writer somewhere.

    Also, the "do or die" idiom is good here. Check to see if your flock fails and be sure to output $! if it does.

      I checked flock, no erroring there... each report generates a unique filename based on various stats and the time, so no worry of other processes adding on. I also did a run where I commented out my file writes to make sure I wasn't doing writes from any non-flocking part of the system.

      Maybe I can make up a test script to recreate the problem...

                      - Ant
                      - Some of my best work - (1 2 3)

Re: What the flock!? Concurrency issues in file writing.
by moritz (Cardinal) on Oct 01, 2008 at 15:57 UTC
    You don't check the return value of flock. It can fail, like every other system call.
      Well, here is an example I ran on the redhat boxes at work and my debian box at home, it shows bad data on both systems if run enough (usually 20 iterations will do it). I also added a check for syswrite... doesn't seem to be writing short data.

      #!/usr/bin/perl use Fcntl qw(:DEFAULT :flock); use strict; my $FH; my $fn = 'flockdata'; my $test_data = (join('','a'..'z')."\n") x 10; #print length($test_data),"\n"; sysopen($FH,$fn,O_WRONLY|O_CREAT); for(1..10) { last unless fork(); } for(1..20) { die "FLOCK ERROR\n" unless flock($FH,LOCK_EX); die "SYSWRITE ERROR\n" unless syswrite($FH,$test_data,length($test_d +ata)) == \ length($test_data); flock($FH,LOCK_UN); } close $FH;
      and to run it when named testflock.pl
      perl -e 'system(q{perl testflock.pl ; ls -al flockdata; rm flockdata}) + for 1..20'

                      - Ant
                      - Some of my best work - (1 2 3)

        As I mentioned before you must put a sysseek in there (e.g. after the flock, but before the syswrite). If you don't you will see corruption since a single process will have a stale version of the current EOF for it's file descriptor.

        I also see corruption with your code on my mac. I don't see it if there is a sysseek($FH, 0, 2). You mentioned that sysseek() "broke everything". When you put it back in this code, and check for error returns, what do you see?

      But it should block indefinitely (hes not using LOCK_NB), so it shouldn't fail ... syswrite/seek still could :)
        If there were no failure conditions for flock, then I'm sure the documentation wouldn't say
        Returns true for success, false on failure

        I'm not quite sure what the exact failure condition is in which flock neither waits for the lock nor dies, but it seems to exist.

        (Update: It seems that it only will fail if LOCK_NB is used, but I'm not entirely sure).

      Hrm... good point, but I just put in logging to check that, still getting bad data, but flock never returning false.

                      - Ant
                      - Some of my best work - (1 2 3)

Re: What the flock!? Concurrency issues in file writing.
by bluto (Curate) on Oct 01, 2008 at 16:41 UTC
    You've commented out "seek" but you say using "sysseek" was a problem. "seek" will not work with syswrite, you must use sysseek. If you were using sysseek, how did it break things (i.e. compared to how they are broken without it)?

    If you are going to sort the output anyway, one way of handling this might be to have each writer send output to it's own file, then concatenate/sort the files all at the same time at the end.

    If you are writing output lines one at a time, you may be able to just open the file in append mode and write that way, without locks. I think this is OS/stdlib dependent so YMMV. UPDATE: Now that I think about it, this last suggestion just sounds bad. I forgot that I always use locks even in this case since there is no guarantee that the perl/OS/stdlib/whatever will support this in the future even on a machine I believe works this way today.

Solution, it seems.
by suaveant (Parson) on Oct 01, 2008 at 19:14 UTC
    I think I found the solution, but it makes me wonder if I am doing something wacky or if I found a bug in the Perl or Linux (most likely Linux) IO system....

    If I use O_SYNC and O_APPEND on the sysopen with flock my problems seem to go away... but I need both, along with O_CREAT and O_WRONLY... seems very wrong, but so far my tests look good.

                    - Ant
                    - Some of my best work - (1 2 3)

      DBM::Deep won't do for your needs?

      My criteria for good software:
      1. Does it work?
      2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
        I am keeping away from databased solutions for now for a couple reasons: space (current report is over 100 million records); speed (I've had bad luck with db stuff slowing me down because of the added overhead that I don't really need); compatibility (eventually I want to have at least some of this be C, for increased performance, which makes DBM::Deep a bad choice); brevity (for the moment I am changing as little as I can to get this out quickly, we have a client waiting, more changes mean more possible points of failure without extra testing)

        In the future I may look into some improvements, for now I want it to work. (not to mention its driving me nuts not knowing what is screwed up :)

                        - Ant
                        - Some of my best work - (1 2 3)

Re: What the flock!? Concurrency issues in file writing.
by Illuminatus (Curate) on Oct 01, 2008 at 17:55 UTC
    I just have to ask, because it hasn't been mentioned... You are not writing to an NFS filesystem, right?

    You could use a semaphore instead of flock (IPC::SysV::Semaphore).

    Of course, writing lots of data from multiple sources for ultimate sorting sounds like a DB application to me. SQLite or MySQL might seem to be overkill, but they are really easy to install and interface with. If you are storing these reports over time, this is a much better archive system as well. Finally, if your retrieval needs increase in the future, your options are wide open.
      Semaphore doesn't work, either... it must be something at the file buffer level or something, but I've tested syswrite and it returns the proper number of bytes written.

                      - Ant
                      - Some of my best work - (1 2 3)

      Not NFS.

      I am only sorting to get the data back in line after parallel writing, since data gets written out of order (and is not fixed, so I can't just seek to the proper position and write)

                      - Ant
                      - Some of my best work - (1 2 3)

        If you have to sort the results anyway, just dodge the bullet and have each of your processes write to separate files. Then sort the files into one. You can't sort until you get the last line anyway, so there is no loss of parallelism.

        Also, a threaded solution is possible if you are interested.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
Re: What the flock!? Concurrency issues in file writing.
by grizzley (Chaplain) on Oct 02, 2008 at 07:10 UTC
    ++ for beautiful title. If there was some ranking of PerlMonks' best titles, you would for sure get to top 10 :)
Re: What the flock!? Concurrency issues in file writing.
by ikegami (Patriarch) on Oct 01, 2008 at 19:08 UTC
    Just a quick comment: Don't mix seek (buffered I/O) and syswrite (unbuffered I/O). sysseek (unbuffered I/O) would be the tool to use.
Re: What the flock!? Concurrency issues in file writing.
by massa (Hermit) on Oct 01, 2008 at 16:55 UTC
    Try this (works in some tests I did):
    use strict; use warnings; sub report($@) { my $file = shift; my $temp = "$file.$$"; 1 until -w $file; rename $file, $temp or die "rename [$file] [$temp]: $!"; open my $t, '+<', $temp or die "open [$temp]: $!"; seek $t, 0, 2 or die "seek [$temp]: $!"; print $t @_; close $t; rename $temp, $file or die "rename [$temp] [$file]: $!" }
    As the two 'rename' operations are atomic, this might work, and the file inode is preserved, so all should be well. Change the "$file.$$" for other (better) tempfilename if you expect concurrent accesses by the same process or thru the network.
    []s, HTH, Massa (κς,πμ,πλ)
      This won't work all of the time since there is a race condition. If two processes get past the 1 until -w $file line simultaneously. One will end up die'ing when the rename fails. Update: changed "while" to "until".