comment on

The following code used twice as much memory as necessary - why? And what would you (or did I) do about it...

process($data_ref,\@set_up,$template,*OUT);
# Process
sub process {
    my @data = @{ shift()};
    my @set_up = @{ shift()};
    my $template = shift; #For unpack
    local *FH = shift;

    foreach my $record (@data) {
        --do things--}
}
[download]

The @data array was very big, and the subroutine above actually makes a copy of it. I didn't appreciate this when I first wrote it, but it was slow, and was consuming silly amounts of memory (even by Perl standards). Changing the relevant lines as follows makes it run faster, and it now uses less memory - about half as much.

    my $data_ref = shift;     #Big array

    foreach my $record (@{$data_ref}) {
        -- do things more efficiently --
                }
[download]

I am now a happy perl loving bunny, and I can read in my very big files ;-)

--
Anthony Staines

PS - What's happening here, for monks whose Perl skills are closer to my own, is that I passed the array as a reference ($data_ref) to the subroutine. There, partly from force of habit I was turning it back into an array. This, as was obvious with hindsight, doubled the memory requirements of the program. Now @data was big, 10 MB to 50MB or so, and the program was not working well. Accessing the array, through the reference only, fixed the problem - hence the @{$data_ref} bit.

In reply to Learning to *really* love references by astaines

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


The stupid question is the question not asked
	PerlMonks