Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

comment on

( #3333=superdoc: print w/replies, xml ) Need Help??

I actually think you would be hard pressed to even demonstrate an advantage in pre-allocating the array.

Update: The short answer is even with five million elements, and growing the non-pre-allocated array one element at a time, pre-allocating is a little SLOWER. What follows is my testing methodology and findings.

If I build up an array by pre-allocating, then adding elements one by one within the range of indexes that was preallocated, and then build up another array just by adding elements one by one, I can compare elapsed times. When I do, the non-preallocation method seems to win every time, presumably because there's one less statement to execute. Here's an example that attempts to isolate just the parts that involve preallocation and building up of the array's elements:

#!/usr/bin/env perl use strict; use warnings; use Time::HiRes qw(time); use Devel::Size qw(size); my $array_size = 5_000_000; my @array0; my @array1; print "Testing preallocation.\n"; my $t0 = time(); $#array0 = $array_size-1; for (my $i = 0; $i < $array_size; $i++) { $array0[$i] = $i; } my $t0_delta = time() - $t0; print scalar(@array0), " elements. Last element value is $array0[-1]\n +"; printf "Array size is %-0.2f megabytes\n", size(\@array0)/1024/1024; printf "With preallocation: %-0.4f seconds\n", $t0_delta; print "\nTesting without preallocation.\n"; my $t1 = time(); for (my $j = 0; $j < $array_size; $j++) { $array1[$j] = $j; } my $t1_delta = time() - $t1; print scalar(@array1), " elements. Last element value is $array1[-1]\n +"; printf "Array size is %-0.2f megabytes\n", size(\@array1)/1024/1024; printf "Without preallocation: %-0.4f seconds\n", $t1_delta;

On my system the output is:

Testing preallocation. 5000000 elements. Last element value is 4999999 Array size is 38.15 megabytes With preallocation: 0.3005 seconds Testing without preallocation. 5000000 elements. Last element value is 4999999 Array size is 44.55 megabytes Without preallocation: 0.2775 seconds

As you can see, without preallocation is actually faster. One might wonder why I didn't use push. The reason was that push was adding elements to the end of the preallocated array, skipping past the part that had been preallocated. So that raises one other reason not to preallocate (aside from it not actually being faster): It may trick you.

It's interesting that the size of the non-preallocated array is larger in terms of memory consumption than the preallocated one. This is both interesting and can be explained by explaining why preallocation isn't significantly faster than building the array up incrementally.

It would be grossly inefficient to grow memory for an array one element at a time. The reason is that as an array grows, if a new contiguous chunk needed to be allocated every time, the contents of the previous contiguous chunk would need to be copied over to the new one, for each new element. When Perl allocates space for arrays, it leaves the array space to grow to avoid this issue. I don't know the growth plan it uses, but let's say initially Perl leaves enough room for 50 elements. When those 50 are used, Perl allocates a chunk large enough for 100 elements and copies the array into that new chunk. When that is filled, it allocates enough space for 200, and so on. These numbers are totally made up, but the point is it almost always allocates enough space for the array to grow past its current size.

In the case of pre-allocation, the array growth algorithm hasn't applied. You requested a million elements, the actual array size in memory will be a million elements. Not 1.2 million (allowing for growth without further allocations). So in terms of memory consumption, the pre-allocation method was a little more efficient. But in terms of speed, the growth algorithm is already good enough to make the time it takes to grow an array inconsequential when compared to the time it takes to interpret and execute the statement that preallocates the array.

So the pre-allocation microoptimization, even for an array of five million elements, is useless if your goal is to improve speed. Perl's array growth algorithm is similar to a C++ std::vector container class. And has similar time complexity characteristics. For adding elements at the end, allowing the array to grow as needed, the operation is considered O(1) amortized time.

If speed is an issue, profile your code with Devel::Profile and solve the hotspots. Array allocation isn't going to be one of them.

I modified the code above to remove "extra" stuff and to put the guts into a main() subroutine so that I could profile that subroutine using Devel::NYTProf. When I profiled, I see that the loop over the pre-allocated array takes 601 milliseconds. And the loop over the non-preallocated takes 603 milliseconds. However, the preallocation step takes 6.69 milliseconds. So the preallocated array operation (allocating, then iterating) takes a total of 608ms, versus 603 for non-preallocated. Therefore, my assertion earlier, that the act of pre-allocating actually consumes more wallclock time interpreting and processing the line of code than just letting Perl allocate as needed.

# spent 1.30s (1.30+49s) within main::main which was called: # once (1.30s+49s) by main::RUNTIME at line 33 sub main { 8 1 600ns my $array_size = 5_000_000; 9 10 1 200ns my @array0; 11 my @array1; 12 13 1 12s 1 6s my $t0 = time(); # spent 6s making 1 call to Time::HiRes::time 14 1 6.69ms $#array0 = $array_size-1; 15 1 601ms for (my $i = 0; $i < $array_size; $i++ +) { 16 $array0[$i] = $i; 17 } 18 1 21s 1 5s my $t0_delta = time() - $t0; # spent 5s making 1 call to Time::HiRes::time 19 20 1 2s 1 300ns my $t1 = time(); # spent 300ns making 1 call to Time::HiRes::time 21 1 603ms for (my $j = 0; $j < $array_size; $j++ +) { 22 $array1[$j] = $j; 23 } 24 25 1 16s 1 6s my $t1_delta = time() - $t1; # spent 6s making 1 call to Time::HiRes::time 26 27 1 26s 1 18s print scalar(@array0), " elements. + Last element value is $array0[-1]\n"; # spent 18s making 1 call to main::CORE:print 28 1 12s 1 9s printf "With preallocation: %-0.4f +seconds\n", $t0_delta; # spent 9s making 1 call to main::CORE:prtf 29 1 4s 1 2s print scalar(@array1), " elements. L +ast element value is $array1[-1]\n"; # spent 2s making 1 call to main::CORE:print 30 1 89.3ms 1 3s printf "Without preallocation: %- +0.4f seconds\n", $t1_delta; # spent 3s making 1 call to main::CORE:prtf 31 } 32 33 1 7s 1 1.30s main(); # spent 1.30s making 1 call to main::main

You can do this too: See the documentation for Devel::NYTProf.


In reply to Re^2: Best way to allocate some memory first by davido
in thread Best way to allocate some memory first by Anonymous Monk

Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":

  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or or How to display code and escape characters are good places to start.
Log In?

What's my password?
Create A New User
Domain Nodelet?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (4)
As of 2022-06-29 06:13 GMT
Find Nodes?
    Voting Booth?
    My most frequent journeys are powered by:

    Results (94 votes). Check out past polls.