http://qs321.pair.com?node_id=118089


in reply to Memory usage with DBI

The fetchrow_array and fetchrow_arrayref methods retrieve data from the datasource one single row at a time, and thus the system that the script is running on only has to deal with one row's worth of data at a time.

The query results may be queued up in a buffer on the database side, but the database is optimized for that sort of thing.

Overall, the DBI module is quite memory efficient. I've got a similar script running on my webserver that queries about a hundred thousand rows of data, generates HTML pages using the data, and copies the whole thing over to another machine. This batch takes about 10 to 15 minutes to run, even with all the disk I/O and querying going on, and has no effect on the other processes going on with that machine.

I wouldn't worry too much about it! :)

higle

Replies are listed 'Best First'.
Re: Re: Memory usage with DBI
by seesik (Initiate) on Oct 10, 2001 at 23:21 UTC
    I wouldn't worry too much about it! :)

    actually, i'd worry plenty about memory usage and caching result sets if i were Kozz. as was remarked, the fetchrow_ methods do precisely that: fetch a single result record. however the fetchall_ methods cache entire result sets locally; that can get pretty massive if you're querying against credit bureau data or some 120 GB instance.

    it's usually a good idea to let the database server do as much work as possible, including summarization and grouping operations. sometimes you do need all query results before you can proceed, but more often than not, you can just process the results iteratively. and to reiterate Higle's comment, the dbms is usually much more conscientious about memory optimization than you'll be on the client side.

    cheers.