Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Critical sections; In the Perl interpreter

by Wiggins (Hermit)
on Feb 16, 2017 at 18:45 UTC ( #1182156=perlquestion: print w/replies, xml ) Need Help??

Wiggins has asked for the wisdom of the Perl Monks concerning the following question:

My interest is in critical sections in the Perl interpreter itself.
I was asked by a friend, why after starting 150 web spider threads, his system would bog down until it seem to stop, but was still running?

I proposed that each of those threads was a separate process, with each having a full Perl interpreter. But they were all sharing the same .so libraries, and the number of critical sections(CS) that the interpreter had to go through would create single file bottle-neck after bottle-neck.

Are there CS in the interpreter? Are there certain functional areas that have significantly more CS than others

It is always better to have seen your target for yourself, rather than depend upon someone else's description.

  • Comment on Critical sections; In the Perl interpreter

Replies are listed 'Best First'.
Re: Critical sections; In the Perl interpreter
by davido (Cardinal) on Feb 16, 2017 at 20:25 UTC

    I was asked by a friend, why after starting 150 web spider threads, his system would bog down until it seem to stop, but was still running?

    He may be asking the wrong question. The real question might be something similar to, "Is it possible to do 150 simultaneous requests without dragging the system to its knees?" Look at the problem that needs to be solved and remain open to lighter-weight solution that might not have previously been considered.

    # Concurrent non-blocking requests (synchronized with a delay) Mojo::IOLoop->delay( sub { my $delay = shift; $ua->get('mojolicious.org' => $delay->begin); $ua->get('cpan.org' => $delay->begin); }, sub { my ($delay, $mojo, $cpan) = @_; say $mojo->result->dom->at('title')->text; say $cpan->result->dom->at('title')->text; } )->wait;

    ...from the Mojo::UserAgent docs.

    So that sub containing the gets could look like:

    sub { my $delay = shift; $ua->get($_ => $delay->begin) for @list_of_urls; },

    Dave

Re: Critical sections; In the Perl interpreter
by BrowserUk (Pope) on Feb 16, 2017 at 19:03 UTC
    why after starting 150 web spider threads ... I proposed that each of those threads was a separate process,

    Which is it? Threads or processes?

    If it is threads, then there are internal locks, but they are mostly to do with (explicitly) shared memory.

    More likely is that those 150 threads are over-subscribing his memory and forcing swapping. Have him check top/Process Manager.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Critical sections; In the Perl interpreter (no)
by tye (Sage) on Feb 21, 2017 at 00:56 UTC
    But they were all sharing the same .so libraries, and the number of critical sections(CS) that the interpreter had to go through would create single file bottle-neck after bottle-neck.

    No. Critical sections are only needed to protect shared and modifiable resources. Shared libraries are indeed shared, but they are not modifiable. If the code in a shared library implements a critical section around some block of code, then each process that passes through that code is going to have their own semaphore and they will not block each other at all.

    - tye        

Re: Critical sections; In the Perl interpreter
by andal (Hermit) on Feb 19, 2017 at 08:02 UTC

    Is this Linux or Windows? If talking about Linux, then it works approximately like this. Each process gets section of memory that contains "code". Then there's section containing non-changeable data. And there's section for changeable data and stack. The latter 2 are writable, the first 2 are not, so they can be shared between processes. Threads share all sections of the process, but each thread gets its own portion of the stack.

    Perl is no different, but one has to remember, that perl executes scripts and they are loaded into changeable section, so multiple perl processes may end up having their own copies of the script they execute. I say "may end up" because normally Linux does not allocate separate copy of memory page until process attempts to write into it (copy on write).

      APerl is no different,

      Actually, Perl's threads are very different from your description.

      And, copy on write has almost zero benefit in perl scripts, because almost every access to every scalar (and every other internals data structure based upon them) is a write access. Either the reference count is incremented or decremented; or the status bits are changed; or a number is referenced in a string context and the SV field is written, or vice versa; or strings are encoded or decoded.

      Each and every change, even the setting or clearing of a single bit (IV_OK etc.) causes the entire 4kb it is located in to be allocated to the new process and duplicated. CoW buys Perl almost nothing.

      The bottom line is that a working knowledge of the basic executable structure doesn't tell you much about Perl processes and almost nothing about Perl threading.


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
      In the absence of evidence, opinion is indistinguishable from prejudice.
        And, copy on write has almost zero benefit in perl scripts

        Well, there's nothing to argue about here. I was not talking about benefits for perl scripts. I've just mentioned existing feature. So, you are right when you say that it does not bring much of benefits for perl because perl scripts are not in code segments but in data segments.

        Actually, Perl's threads are very different from your description

        It all depends on what is "difference". I only mentioned basic idea of thread vs process. That idea stays the same: thread shares memory of process with other threads. Of course there are different strategies on how that memory is shared and particular sharing may only worsen memory consumption at the end.

        The bottom line is that a working knowledge of the basic executable structure doesn't tell you much about Perl processes and almost nothing about Perl threading.

        That is true. I just didn't see that the OP wanted to have knowledge about Perl threading. I thought, that the question was more about memory management for Perl. So I pointed out, that perl scripts are loaded into data segment and they don't get shared like segments of perl itself. If I misunderstood the question, then sorry.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1182156]
Approved by stevieb
Front-paged by kcott
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (4)
As of 2020-10-30 05:11 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My favourite web site is:












    Results (277 votes). Check out past polls.

    Notices?