Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re: problems with fork from thread

by DrManhattan (Chaplain)
on Mar 01, 2012 at 16:43 UTC ( #957256=note: print w/replies, xml ) Need Help??

in reply to problems with fork from thread

I don't know the answer to your question, but this article seems relevant:

If you just need some simple code to do 10 things at the same time, you might like Parallel::ForkManager. Example code looks something like this:

use Parallel::ForkManager; my $pm = new Parallel::ForkManager(10); foreach (1..10) { # Forks and returns the pid for the child: my $pid = $pm->start and next; &something_1(); $pm->finish; # Terminates the child process } $pm->wait_all_children;


Replies are listed 'Best First'.
Re^2: problems with fork from thread
by cristi1979 (Novice) on Mar 01, 2012 at 17:49 UTC
    Forking is not an ideal solution, because I exchange data between the threads and I will have to rewrite a lot to go with forks.

    If I use only threads, I have no problems.

    My problem is that I'm using n threads doing work with HTML::TreeBuilder. Somewhere in there I have some memory leaks, because the memory used by the script is increasing until it will be killed because of OOM.

    So my solution (that will not involve rewriting the entire application with forks) was to fork only the HTML::TreeBuilder part and so I will get read of the accumulated memory leaks.

    Seems I hit an other problem and I have to look on a way to go with fork and sockets.

    I still hope that a solution will appear for this.

      May I ask if you are successfully using threading for your program, why to you want to fork as well? What are you going to do after you fork?

      Note: I'm not condemning the use of fork here, just trying to see a way to a solution for you that avoids needing to re-write your entire application.

      If, for example, you need to run an external command, then there are a couple of ways of approaching that that are known to work.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?

        Like I said, I'm forking in order to not accumulate the memory leaks in the main program. Forking will increase the memory size of the new process only and get rid of it when it dies.

        I have $tree = $tree->delete() and also undef ($tree). Also I put everything in a class and undefed the class, but I still see the memory increasing.
      I have same problem recently (threaded app with HTML::TreeBuilder::XPath).

      I have downloaded and processed less than 6000 entries (html pages) but my program uses 2Gb! I have fixed this issue by adding $tree->delete() after I don't use it.


Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://957256]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others examining the Monastery: (3)
As of 2023-12-10 07:34 GMT
Find Nodes?
    Voting Booth?
    What's your preferred 'use VERSION' for new CPAN modules in 2023?

    Results (38 votes). Check out past polls.