Re: Memory usage with DBI


We don't bite newbies here... much
	PerlMonks

Re: Memory usage with DBI

by higle (Chaplain)

on Oct 10, 2001 at 23:06 UTC ( [id://118089]=note: print w/replies, xml )

Need Help??

in reply to Memory usage with DBI

The fetchrow_array and fetchrow_arrayref methods retrieve data from the datasource one single row at a time, and thus the system that the script is running on only has to deal with one row's worth of data at a time.

The query results may be queued up in a buffer on the database side, but the database is optimized for that sort of thing.

Overall, the DBI module is quite memory efficient. I've got a similar script running on my webserver that queries about a hundred thousand rows of data, generates HTML pages using the data, and copies the whole thing over to another machine. This batch takes about 10 to 15 minutes to run, even with all the disk I/O and querying going on, and has no effect on the other processes going on with that machine.

I wouldn't worry too much about it! :)

higle

Comment on Re: Memory usage with DBI Select or Download Code

Replies are listed 'Best First'.

Re: Re: Memory usage with DBI
by seesik (Initiate) on Oct 10, 2001 at 23:21 UTC

I wouldn't worry too much about it! :)

actually, i'd worry plenty about memory usage and caching result sets if i were Kozz. as was remarked, the fetchrow_ methods do precisely that: fetch a single result record. however the fetchall_ methods cache entire result sets locally; that can get pretty massive if you're querying against credit bureau data or some 120 GB instance.

it's usually a good idea to let the database server do as much work as possible, including summarization and grouping operations. sometimes you do need all query results before you can proceed, but more often than not, you can just process the results iteratively. and to reiterate Higle's comment, the dbms is usually much more conscientious about memory optimization than you'll be on the client side.

cheers.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom