Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: XML::CSV out of memory

by stonecolddevin (Parson)
on Mar 26, 2015 at 16:19 UTC ( #1121412=note: print w/replies, xml ) Need Help??


in reply to XML::CSV out of memory

In all honesty if you have a modern amount of RAM on your system, 500MB isn't that big.

That said, in the docs, it looks like you can do this:

use XML::CSV; $default_obj_xs = Text::CSV_XS->new({quote_char => '"'}); $csv_obj = XML::CSV->new({csv_xs => $default_obj_xs}); $csv_obj->{column_headings} = \@arr_of_headings; $csv_obj->{column_data} = \@arr_of_data; $csv_obj->print_xml("out.xml");

Basically, notice that you can pass a Text::CSV_XS object to XML::CSV, which would allow you to read in $n lines of the CSV file, and then pass them in as an array to XML::CSV. This would be fairly trivial to do, but if you can't get it figured out, come back and we can help you through it.

Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past

Replies are listed 'Best First'.
Re^2: XML::CSV out of memory
by monkey_boy (Priest) on Mar 26, 2015 at 16:25 UTC
    I guess we dont know how much RAM is required without seeing the input CSV data, and the derived XML, the in memory parse of the CSV will be many factors larger than the original file.
    That said, if you have access to a machine with a lot more RAM this can't hurt.


    This is not a Signature...

      I agree but, even the smallest linode and EC2 instances have a gig of RAM, and DigitalOcean is 512MB, which squeaks by. I digress, but I'm just saying, it would be a big difference it it were a 500GB file being parsed.

      Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past

Re^2: XML::CSV out of memory
by slugger415 (Monk) on Mar 26, 2015 at 16:58 UTC

    Thanks for the replies and suggestions. Yes it is strange, the system now has 12 gigs of memory. My own workstation has 16 gigs and I'm not getting the out of memory error running the same script. Anyway.

    So it's probably a dumb question, but do I even need Text::CSV for your suggestion? Couldn't the script just put $n lines of text into an array and feed those to @arr_of_data? And how would I do $csv_obj->print_xml("out.xml") at each iteration without clobbering the previous iteration? (ok, you can tell I'm still learning here - begging patience...)

    Thanks.

      do I even need Text::CSV for your suggestion? Couldn't the script just put $n lines of text into an array and feed those to @arr_of_data?

      Text::CSV will insure you are reading complete records. If you can be sure that your input files have no embedded "new lines" in the CSV records, then you could skip Text::CSV.

      And how would I do $csv_obj->print_xml("out.xml") at each iteration without clobbering the previous iteration?

      Try (untested):

      open(my $outFH, '>', "out.xml"); while (<>) { ...; $csv_obj->print_xml($outFH); }

      The example seems to call for Text::CSV_XS in this instance:

      $default_obj_xs = Text::CSV_XS->new({quote_char => '"'}); $csv_obj = XML::CSV->new({csv_xs => $default_obj_xs});

      It's really not hurting you to use it, so you might as well honestly.

      Three thousand years of beautiful tradition, from Moses to Sandy Koufax, you're god damn right I'm living in the fucking past

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1121412]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (4)
As of 2020-05-29 17:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    If programming languages were movie genres, Perl would be:















    Results (170 votes). Check out past polls.

    Notices?