http://qs321.pair.com?node_id=106340


in reply to (Ovid) Re: Writing to a file
in thread Writing to a file

The problem with all your solutions is that want me to write to multiple files and combine them the only problem with that is that myscript.prl is actually being called locally on different wkstations when by 
system("rshto $wks myscript.prl >> $tmp_file");

[download]
and we have over 200,000 wkstations here were the script is supposed to run. I thought about writing to different files too, but the sheer number of temp log files generated make this impractical, and the extra code to put these files back by date/time stamp and then unlink("$tmp_file") is also needed. I was wondering if there is a way in perl to know if the file is being written too currently, and if its is being wriiten to wait until no other process is writing to it.

Replies are listed 'Best First'.
Re: Re: (Ovid) Re: Writing to a file
by THRAK (Monk) on Aug 21, 2001 at 00:25 UTC
    Within "myscript.prl" instead of just printing and capturing STDOUT to $tmp_file, open it instead and write to it. You will want to checkout flock which may help prevent the overwriting problem. Still, if you have hundreds of thousands of processes/machines all trying to write to the same file, you are creating a huge bottleneck. What about running the command on each machine as you appear to want to do, but write it to a local temporary accumulation file. Then either retrieve each one, or send them to a common queue (on a periodic basis) where a second process can collate them into this one behemoth file you desire? Just a thought.

    -THRAK
    www.polarlava.com
[reply]
Re: Re: (Ovid) Re: Writing to a file
by dragonchild (Archbishop) on Aug 21, 2001 at 00:24 UTC
    There is flock, which would lock the file. But, each process has to request what the flock status is and I'm not very conversant on how that works.

    Now, what you're saying is that you're going to run this script on separate workstations. Why not just run it, store the logfile locally, then have another script which gathers together all the data?

    ------
    /me wants to be the brightest bulb in the chandelier!

    Vote paco for President!

[reply]
      Something in the back of my mind is scratching me and telling me there might be problems with using flock over NFS (I assume you're using NFS if all these workstations are writing to the same file?)

      Another potential approach? how about saving files as:

      localtime().hostname.extension?

      That way, you can sort the files date/time order simply by doing an ls -l.

      Note - this would assume that all the machines are synchronized to the same clock.

[reply]
Re: Re: (Ovid) Re: Writing to a file
by OzzyOsbourne (Chaplain) on Aug 22, 2001 at 21:54 UTC

    Couldn't you utilize the users home directories for a location for the temp file, and then run one script to comb the homedirs and conglomerate them all into a master file?

    -OzzyOsbourne

[reply]

In Section Seekers of Perl Wisdom