Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Re^2: Multi threading

by sandy1028 (Sexton)
on Apr 06, 2009 at 07:17 UTC ( [id://755651] : note . print w/replies, xml ) Need Help??

in reply to Re: Multi threading
in thread Multi threading

I am using this code to create the threads using fork. This will read all the files in a directory and writes to a log file. Now this code creates th elog file. But some of the files are missed due to overlapping. How can I avoid this using locks
#!/usr/bin/perl use Parallel::ForkManager; my $processor = shift; $tc=5; # threads $fc = 100; # splits.. each thread should process 100 at a time my $pm = new Parallel::ForkManager($tc+1); $pm->run_on_finish( sub { my ($pid, $exit_code, $ident) = @_; $tmp +Files[$ident] = undef; } ); foreach my $i (0..$#tmpFiles) { # Forks and returns the pid for the child: my $pid = $pm->start($i) and next; $SIG{INT} = 'DEFAULT'; my $filename = $tmpFiles[$i]->filename(); my $file = IO::File->new("<$filename") or die "Can't open $filen +ame\n"; while((my $line) = $file->getline()) { last unless defined($line); chomp $line; my ($dir, $file) = split(/\t/, $line); $processor->($dir, $file, $config, $log); } $pm->finish; # Terminates the child process }

Replies are listed 'Best First'.
Re^3: Multi threading
by cdarke (Prior) on Apr 06, 2009 at 07:55 UTC
    As Corion said, you create processes with fork(), not threads. If you have multiple writers to a file then there are several solutions - none of them particularly good!
    The simplest is to lock the whole file for each writer - but that defeats the object of having multiple writers.
    Another is to allocate specific regions (by byte offset) of the file for each thread/process ensuring that these do not overlap. That avoids locking, but requires some planning and management. Use seek to position each thread/process.
    If you turn off buffering for the write you will reduce the overlap, but probably not get rid of it altogether.
      Can you please tell me how to lock it. Any tutorial or any example related to this
Re^3: Multi threading
by sandy1028 (Sexton) on Apr 07, 2009 at 10:05 UTC
    Can you please tell me how can I create a separate log files for each process. Not all process writing to a single file?