Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re^6: Abusing Map

by AnomalousMonk (Bishop)
on May 17, 2018 at 18:27 UTC ( #1214774=note: print w/replies, xml ) Need Help??


in reply to Re^5: Abusing Map
in thread Abusing Map

... for big lists ...

I agree about the time/space advantages, sometimes life-saving advantages, of in-place mutation for big lists.

... postfix for ... generate[s] a list as big as the array.

I thought a for-loop, postfix or otherwise, was guaranteed to have a space complexity (if that's the correct term) of zero | 1 (per ikegami here) — if it's written right! That, e.g., either
    for (@humongous_array) { $_ = $_ + $_ }
or
    $_ = $_ + $_ for @humongous_array;
are guaranteed to be just fine (if @humongous_array could fit in memory in the first place), whereas
    @humongous_array = map $_ + $_, @humongous_array;
can kill you in terms of both time and space. In fact, isn't it the case that beginning with Perl version 5.mumble, map called in void context produces no intermediate list/array, so even
    map $_ = $_ + $_, @humongous_array;
should be perfectly safe (although for aesthetic reasons, I'm not a fan of this particular usage)?

And let's just not talk about Fight Club | grep anymore.


Give a man a fish:  <%-{-{-{-<

Replies are listed 'Best First'.
Re^7: Abusing Map
by BrowserUk (Pope) on May 17, 2018 at 19:03 UTC

    The problem comes when you need to build a list of indexes as with the OPs code:

    sub checkMem{ `tasklist /nh /fi "pid eq $_[0]"`; };; print checkMem( $$ );; perl.exe 3040 Console 1 +9,544 K $#a = 1e6;; print checkMem( $$ );; perl.exe 3040 Console 1 1 +7,428 K map{ $_ } 0 .. $#a;; print checkMem( $$ );; perl.exe 1224 Console 1 7 +3,272 K ### 56MB to build the list of 1e6 indices ###

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
    In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit
      The problem comes when you need to build a list of indexes ...

      Ah, ok, I see now what you're talking about.


      Give a man a fish:  <%-{-{-{-<

      I remember reading that for ($start..$end) kind of loops are also optimized not to produce an intermediate list:

      $ perl -s -e' sub mymem { system("ps -q $$ -o vsz") }; mymem(); $#a = $size; mymem(); $a[$_]=0 for 0..$#a; mymem() ' -- -size=1e7 VSZ 18724 VSZ 96852 VSZ 333528 $ perl -s -e' sub mymem { system("ps -q $$ -o vsz") }; mymem(); $#a = $size; mymem(); $a[--$size]=0 while $size; mymem() ' -- -size=1e7 VSZ 18724 VSZ 96852 VSZ 333528 $ perl -E'say $^V' v5.24.1
      At least, they seem to be on my machine.

      This doesn't cover map, though:

      $ perl -s -e' sub mymem { system("ps -q $$ -o vsz") }; mymem(); $#a = $size; mymem(); map {$_} 0..$#a; mymem() ' -- -size=1e7 VSZ 18724 VSZ 96852 VSZ 573664

        From PerlOP:
        In the current implementation, no temporary array is created when the range operator is used as the expression in foreach loops, but older versions of Perl might burn a lot of memory when you write something like this:

        Many of my efficiency habits formed back in 5.6/5.8 days, and I've never seen the need to change them.

        One problem is that optimisation only kicks in in certain circumstances -- and I don't remember if I ever knew what they exactly were. I have a sneaking suspicion that in 32-bit perls, anything over 2**31 may have caused a list, but I haven't used a 32-bit perl for a decade, so I cannot check.

        One example that bit me was the iterator moving outside the range of integers:

        1 for 1e200 .. 1e200+1e6;; [Range iterator outside integer range at (eval 10) line 1, <STDIN> lin +e 2.

        Easy enough to re-write that one, but not so much if the numbers come from outside or are generated.

        The while/until loop approach handles anything 64-bit floats can, and (from) memory, is usually faster than postfix for even for integers; though that could have changed a lot since I last benchmarked it.

        I like to program in a consistent way -- where it doesn't compromise performance to do so -- and tend to stick with patterns once I've established ones that work.

        Eg. I still use '<:raw' as a matter of course, even though things changed and it was deprecated when they re-vamped PerlIO, and it was demonstrated that some other combination of pushing, popping and/or applying of layers was apparently faster. The whole thing just got too complex to remember and too convoluted to test.

        The whole PerlIO thing was something of a debacle IMO; a vast amount of effort and change for something that 95% of Perl users have never used :(


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". The enemy of (IT) success is complexity.
        In the absence of evidence, opinion is indistinguishable from prejudice. Suck that fhit
Re^7: Abusing Map
by ikegami (Pope) on May 18, 2018 at 08:14 UTC

    I thought a for-loop, postfix or otherwise, was guaranteed to have a space complexity (it that's the correct term) of zero

    Of 1 (constant).

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1214774]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (5)
As of 2020-11-25 01:01 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?