Modperl2 + mpm_event + seemingly forgotten global vars

spellila has asked for the wisdom of the Perl Monks concerning the following question:

Greeting everyone, i get stucked using modperl2 on apache 2.4 (mpm_event) and ubuntu (upgraded on daily base form ubuntu-repo). I try to keep some filehandles and datastructures simply on a "our" variable. This works fine until i generate some network load for this server (1000 persistent connections each wants to get the same stuff). Than it looks like the "our" is forgeting its value. This is happening much faster if you are using more threads per child. Here a simple example:

package example::PerlResponseHandler;

use strict;
use warnings;
use APR::OS ();
use Apache2::Const -compile => qw(OK);

our $obj;

sub handler {
    my $r = shift;
    my $tid = APR::OS::current_thread_id();

    if(!$obj) {
        $obj = {one => 1};
        warn "[pid $$ tid $tid] New object generated";
    }

    $r->content_type('text/html');
    $r->print('mod_perl rules!' . $obj->{one});

    return Apache2::Const::OK;
}

1;
[download]

Using this package as responsehandler in your apache2.conf and the `ab`-tool targeting this handler, you will see at your error.log that the our var is getting setted. After some time you will see that this variable is reseted on and on.

Using 5000 MaxRequestWorkers splitted by threads 250 on each child i could see that only 18 threads was able to hold this variable without any interaction after it was initialized (over days!).

Even if i used only 3 threads per child and 1500 possible servers, the error was reproduced after the 264 thread was spawned. 263 threads was keeping their states over days.

The server have enough ressources over and no limits being reached.

I want to achive that all my threads holding the states of their our vars.

After some sleepless days i decided to ask the perlmonk community. Hopefully somebody can help further.

Comment on Modperl2 + mpm_event + seemingly forgotten global vars Download Code

Replies are listed 'Best First'.
Re: Modperl2 + mpm_event + seemingly forgotten global vars by shmem (Chancellor) on Mar 28, 2017 at 20:15 UTC
Hmm... `if(!$obj) { $obj = {one => 1}; ... $r->print('mod_perl rules!' . $obj->{1});` [download] You probably meant `$r->print('mod_perl rules!' . $obj->{one});` [download] or vice versa. Is that hashref `$obj` being used elsewhere in your code? perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'	[reply] [d/l] [select]
Re^2: Modperl2 + mpm_event + seemingly forgotten global vars by spellila (Initiate) on Mar 29, 2017 at 10:46 UTC
Yeah, thanks. Was my fault, i corrected it.	[reply]
Re: Modperl2 + mpm_event + seemingly forgotten global vars by Marshall (Canon) on Mar 28, 2017 at 23:15 UTC
You say "I try to keep some filehandles and datastructures simply on a "our" variable." When using threads that modify shared data which is "global the the entire process", there needs to be some kind of locking mechanism. A thread has to acquire an "exclusive write lock" versus its other "thread peers" in order to update that commonly shared information. I don't see any locking code that would adjudicate write/read conflicts amongst its peer threads. It is possible that one thread might read a variable during the writing of that variable by another thread. I don't know much about the server software that you are using, but I'm sure that some other Monks do. However, unlocked shared variable sounds like a likely suspect to me.	[reply]
Re^2: Modperl2 + mpm_event + seemingly forgotten global vars by spellila (Initiate) on Mar 29, 2017 at 11:24 UTC
I agree, but only under following condition: - the threads are sharing their vars - the threads are changing their vars I don't see any condition matching at this case. The global var is shared only to the scope it was created. This means to thread it self. Thats ok for my case. On this way i can initate handles and keep them per thread. Each thread having its own handles should be no problem, do it? So every thread will create its own "our" variable. And after the creation it should stay there. This works fine for low numbers of connections. But under pressure it gets reinitilized on and on.	[reply]
Re: Modperl2 + mpm_event + seemingly forgotten global vars by Krambambuli (Curate) on Mar 29, 2017 at 17:29 UTC
If what you get here in the monastery doesn't help, try visiting and reposting to the modperl mailing list. Even if it is a bit hard to find, the modperl list is active and can be very helpful.	[reply]
Re: Modperl2 + mpm_event + seemingly forgotten global vars by Anonymous Monk on Mar 29, 2017 at 11:32 UTC
How many filehandles is it? There are limits on how many filehandles you can have open on some OS not sure about ubuntu	[reply]
Re^2: Modperl2 + mpm_event + seemingly forgotten global vars by spellila (Initiate) on Mar 29, 2017 at 18:23 UTC
In my case there are exactly zero filehandles. But every handle will face this same problem. At the end them are only references. And if a thread can't remember the value of this variable, it will reinitilize it. There as many handles as threads. There should be no problem. I'm also able to open as many (and more) handles in a single script, without modperl. Here is my `ulimit -a`: core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 64000 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited [download] I would also await an error message for something like that. But there is nothing. This global vars should loose its value only under the condition the thread ends. A java developer pointed me to the garbagge colector, but at the moment my knowhow at this area is not at that level i would want it. (Never was thinking about garbagge collection while using perl) Let's assume them are ending, why is there a little rest of threads allowed to keep living and holding their vars? (I mean it is a computer. I was thinking everything in it has to be deterministic, so someone has to delete this vars)	[reply] [d/l]


Keep It Simple, Stupid
	PerlMonks