Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: System call doesn't work when there is a large amount of data in a hash

by marioroy (Parson)
on Apr 29, 2020 at 17:19 UTC ( #11116227=note: print w/replies, xml ) Need Help??


in reply to System call doesn't work when there is a large amount of data in a hash

Hi Nicolasd,

One may spin up a worker early and communicate via a channel. The worker makes the system call and notifies once completed. The 'Simple' channel is specified due to just one background worker (i.e. no mutex locking needed). Otherwise, 'Mutex' is the default when not specified. Lastly, this works on Unix OSes and Windows.

Update: Pass back the status. See system on PerlDocs for how to inspect the status.
Update: Including error message.

use strict; use warnings; use MCE::Child; use MCE::Channel; my $chnl = MCE::Channel->new( impl => 'Simple' ); # spin up worker early before creating big hash mce_child { while ( my ($cmd, @args) = $chnl->recv ) { local ($?, $!); system($cmd, @args); $chnl->send2($?, $!); } }; # create big hash my %big_hash; my ($status, $errmsg); # pass command and optionally args to worker $chnl->send('ls'); ($status, $errmsg) = $chnl->recv2; # ditto, sleep for 2 seconds $chnl->send('sleep', '2'); ($status, $errmsg) = $chnl->recv2; # notify no more work, then reap worker $chnl->end; MCE::Child->waitall;

The background worker awaits for the next system call to make. Waiting involves no CPU time. No hash copy either because the worker is spun early.

See also Child in meta::cpan.

Regards, Mario

  • Comment on Re: System call doesn't work when there is a large amount of data in a hash
  • Download Code

Replies are listed 'Best First'.
Re^2: System call doesn't work when there is a large amount of data in a hash
by marioroy (Parson) on Apr 29, 2020 at 22:26 UTC

    Hi again,

    I made an improved version including worker sending the error message. Replace system with syscmd in your script. That sends the command along with any arguments to the background worker. The worker receives the command to run and any arguments. The status and error message are sent to the main process after running the command.

    syscmd:

    use strict; use warnings; use MCE::Child; use MCE::Channel; my $chnl = MCE::Channel->new( impl => 'Simple' ); # spin up worker early before creating big hash mce_child { local $SIG{__WARN__} = sub {}; while ( my ($cmd, @args) = $chnl->recv ) { local ($?, $!); system($cmd, @args); $chnl->send2($?, $!); } }; sub syscmd { my $cmd = shift; return unless $cmd; $chnl->send($cmd, @_); my ($status, $errmsg) = $chnl->recv2; if ($status == -1) { print "SYSTEM: failed to execute ($cmd): $errmsg\n"; } elsif ($status & 127) { printf "SYSTEM: $cmd died with signal %s, %s coredump\n", ($status & 127), ($status & 128) ? 'with' : 'without'; } else { printf "SYSTEM: $cmd exited with status %d\n", $status >> 8; } } # create big hash my %big_hash; # pass command and optionally args syscmd('ls'); # attempt to run a command not found syscmd('something'); # sleep for 2 seconds syscmd('sleep', '2'); # notify no more work, then reap worker $chnl->end; MCE::Child->waitall;

    output:

    ls output from syscmd SYSTEM: ls exited with status 0 SYSTEM: failed to execute (something): No such file or directory SYSTEM: sleep exited with status 0

    Regards, Mario

      Using the memory gobbling technique by Corion here, let's do some testing. I divided by 2 to reflect the actual memory consumption desired.

      My virtual CentOS 7 machine has 4 GB of RAM allocated to it. This creates a 3 GB scalar key-value pair to consume 75%. The system command called by the main process fails, no ls output on the 2nd time. Calling syscmd succeeds due to the background worker spun early.

      syscmd

      use strict; use warnings; use MCE::Child; use MCE::Channel; my $chnl = MCE::Channel->new( impl => 'Simple' ); # spin up worker early before creating big hash mce_child { local $SIG{__WARN__} = sub {}; while ( my ($cmd, @args) = $chnl->recv ) { local ($?, $!); system($cmd, @args); $chnl->send2($?, $!); } }; sub syscmd { my $cmd = shift; return unless $cmd; $chnl->send($cmd, @_); my ($status, $errmsg) = $chnl->recv2; if ($status == -1) { print "SYSTEM: failed to execute ($cmd): $errmsg\n"; } elsif ($status & 127) { printf "SYSTEM: $cmd died with signal %s, %s coredump\n", ($status & 127), ($status & 128) ? 'with' : 'without'; } else { printf "SYSTEM: $cmd exited with status %d\n", $status >> 8; } } # My CentOS VM has 4 GB of RAM # create big hash my $memory_eaten = 3 * 1024*1024*1024 / 2; # 3 GB, adjust to fit my %memory_eater = ( foo => scalar( ' ' x $memory_eaten ), ); # pass command and optionally args syscmd('ls'); # this one works; see status that it succeeded system('ls'); # this one fails; no ls output the 2nd time # attempt to run a command not found syscmd('something'); # sleep for 2 seconds syscmd('sleep', '2'); # busy loop, see top output in another terminal # notice the memory consumption (i.e. RES) # press Ctrl-C to exit or let it finish 1 for 1..3e8; # notify no more work, then reap worker $chnl->end; MCE::Child->waitall;

      output:

      ls output from syscmd SYSTEM: ls exited with status 0 SYSTEM: failed to execute (something): No such file or directory SYSTEM: sleep exited with status 0

      Regards, Mario

        Hi again,
        I tested your example and it seems to work great, so the problem is solved!
        Thank you so much!
        I quickly tested your script on the Centos, and it works perfectly! I will implement it in my script tomorrow, hopefully it will work as good. You can send me your name, I would like to thank you when the tool will be published.
        Wow thanks so much for your help! I will test it tomorrow for sure! I thought I had to give up the system call, would be amazing if it works!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11116227]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (6)
As of 2020-09-24 15:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If at first I donít succeed, I Ö










    Results (134 votes). Check out past polls.

    Notices?