Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re^2: DBD::CSV file size limitation?

by beachbum (Beadle)
on Dec 07, 2005 at 19:01 UTC ( [id://514980]=note: print w/replies, xml ) Need Help??


in reply to Re: DBD::CSV file size limitation?
in thread DBD::CSV file size limitation?

Thanks for the quick reply.
I have tried running it on 3 different machines ranging from 1.3 Ghz / 1 GB to a new dual processor box with 4 GB. The RAM is not being consumed, however at least on first box, the processor has always maxed out while reading the files.

This particular file does have 3 distinct record types in it, each with varying fields and lengths. This hasn't been an issue in the past as my query specifies WHERE RECORD_TYPE = n.

I would like to try your suggestion on feeding the data into DBD::CSV line by line, but am a little confused as to how to do that. Am I able to issue a DBI->connect() to a string instead of a file?

Replies are listed 'Best First'.
Re^3: DBD::CSV file size limitation?
by jZed (Prior) on Dec 07, 2005 at 21:12 UTC
    > This particular file does have 3 distinct record types
    > in it, each with varying fields and lengths

    That may be a problem. What record types are they? It's possible that the module assumes a given record type and then treats the other record-types as of the same kind, thereby getting confused about record boundaries and trying to create a single huge record.

[reply]

      The record types are different in that type '0' has 50 fields, type '1' has 22 fields, type '2' has 11 fields. The first 5 fields of each record are common, with field 5 being the record type (0|1|2). There is no binary data in the records.

      I am only interested in one type at a time, and have been sucessfully getting that until now with WHERE type = 1.

[reply]
[d/l]

In Section Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: note [id://514980]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others about the Monastery: (4)
As of 2024-04-25 14:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found