I was asked by a friend, why after starting 150 web spider threads, his system would bog down until it seem to stop, but was still running?

He may be asking the wrong question. The real question might be something similar to, "Is it possible to do 150 simultaneous requests without dragging the system to its knees?" Look at the problem that needs to be solved and remain open to lighter-weight solution that might not have previously been considered.

# Concurrent non-blocking requests (synchronized with a delay)
Mojo::IOLoop->delay(
  sub {
    my $delay = shift;
    $ua->get('mojolicious.org' => $delay->begin);
    $ua->get('cpan.org'        => $delay->begin);
  },
  sub {
    my ($delay, $mojo, $cpan) = @_;
    say $mojo->result->dom->at('title')->text;
    say $cpan->result->dom->at('title')->text;
  }
)->wait;
[download]

...from the Mojo::UserAgent docs.

So that sub containing the gets could look like:

  sub {
    my $delay = shift;
    $ua->get($_ => $delay->begin) for @list_of_urls;
  },
[download]

why after starting 150 web spider threads ... I proposed that each of those threads was a separate process,

Which is it? Threads or processes?

If it is threads, then there are internal locks, but they are mostly to do with (explicitly) shared memory.

More likely is that those 150 threads are over-subscribing his memory and forcing swapping. Have him check top/Process Manager.

But they were all sharing the same .so libraries, and the number of critical sections(CS) that the interpreter had to go through would create single file bottle-neck after bottle-neck.

No. Critical sections are only needed to protect shared and modifiable resources. Shared libraries are indeed shared, but they are not modifiable. If the code in a shared library implements a critical section around some block of code, then each process that passes through that code is going to have their own semaphore and they will not block each other at all.

Is this Linux or Windows? If talking about Linux, then it works approximately like this. Each process gets section of memory that contains "code". Then there's section containing non-changeable data. And there's section for changeable data and stack. The latter 2 are writable, the first 2 are not, so they can be shared between processes. Threads share all sections of the process, but each thread gets its own portion of the stack.

Perl is no different, but one has to remember, that perl executes scripts and they are loaded into changeable section, so multiple perl processes may end up having their own copies of the script they execute. I say "may end up" because normally Linux does not allocate separate copy of memory page until process attempts to write into it (copy on write).

APerl is no different,

Actually, Perl's threads are very different from your description.

And, copy on write has almost zero benefit in perl scripts, because almost every access to every scalar (and every other internals data structure based upon them) is a write access. Either the reference count is incremented or decremented; or the status bits are changed; or a number is referenced in a string context and the SV field is written, or vice versa; or strings are encoded or decoded.

Each and every change, even the setting or clearing of a single bit (IV_OK etc.) causes the entire 4kb it is located in to be allocated to the new process and duplicated. CoW buys Perl almost nothing.

The bottom line is that a working knowledge of the basic executable structure doesn't tell you much about Perl processes and almost nothing about Perl threading.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.

In the absence of evidence, opinion is indistinguishable from prejudice.

And, copy on write has almost zero benefit in perl scripts

Well, there's nothing to argue about here. I was not talking about benefits for perl scripts. I've just mentioned existing feature. So, you are right when you say that it does not bring much of benefits for perl because perl scripts are not in code segments but in data segments.

Actually, Perl's threads are very different from your description

It all depends on what is "difference". I only mentioned basic idea of thread vs process. That idea stays the same: thread shares memory of process with other threads. Of course there are different strategies on how that memory is shared and particular sharing may only worsen memory consumption at the end.

The bottom line is that a working knowledge of the basic executable structure doesn't tell you much about Perl processes and almost nothing about Perl threading.

That is true. I just didn't see that the OP wanted to have knowledge about Perl threading. I thought, that the question was more about memory management for Perl. So I pointed out, that perl scripts are loaded into data segment and they don't get shared like segments of perl itself. If I misunderstood the question, then sorry.