Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

undef quite slow?

by glwtta (Hermit)
on Aug 04, 2004 at 07:17 UTC ( [id://379914]=perlquestion: print w/replies, xml ) Need Help??

glwtta has asked for the wisdom of the Perl Monks concerning the following question:

Something of an odd problem I am having (at least seems so to me):

I have a sizeable arrayref of arrayrefs (about 20000 arrays of about 75 elements: roughly 70MB in memory) which is the result of a fetchall_arrayref call with DBI.

It takes about 2 seconds to retrieve and load the data into memory, and then about 10 seconds(!) to undef the data structure (whether explicitely or just at the end of the script). This slowdown seems to be quite consistent - an early iteration of this script was using 10-20 of such data structures, so after it finished all the actual work, it would just sit there for 5-10 minutes before exiting - quite annoying.

One thing I noticed while playing with it is that if I undef the structure and then populate it again (from the same query in fact), the second undef is then instanteneous, or close to it.

This is all on RedHat Linux with perl 5.8.4

Can anyone shed some light on what exactly is going on?

Replies are listed 'Best First'.
Re: undef quite slow?
by BrowserUk (Patriarch) on Aug 04, 2004 at 07:56 UTC

    Take a look at Re: Exiting takes a looong time (by the man that knows) and see if that is applicable.


    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
    "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
Re: undef quite slow?
by PodMaster (Abbot) on Aug 04, 2004 at 07:36 UTC
    What does perl -V report? I've heard people mention something about libc being to blame, or perl's malloc ... try recompiling with a different (newer) libc, or with perl's malloc (or without) and see if that does anything.

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

      If you just need to exit then there is a hack to skip Perl's (potentially slow) destruction phase, leaving the OS to clean up.

      use POSIX; # big data structure here POSIX::_exit(0);

      cheers

      tachyon

      perl -V output:
      Summary of my perl5 (revision 5 version 8 subversion 4) configuration: Platform: osname=linux, osvers=2.4.20-28.7bigmem, archname=i686-linux uname='linux bulldog 2.4.20-28.7bigmem #1 smp thu dec 18 11:04:21 +est 2003 i686 unknown ' config_args='-de -Dinc_version_list=none -Dprefix=/usr/local/apps/ +perl-5.8.4' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultipl +icity=undef useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fno-strict-aliasing -I/usr/local/include -D_LA +RGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm', optimize='-O2', cppflags='-fno-strict-aliasing -I/usr/local/include -I/usr/include +/gdbm' ccversion='', gccversion='2.96 20000731 (Red Hat Linux 7.3 2.96-11 +3)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -lndbm -lgdbm -ldl -lm -lcrypt -lutil -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc libc=/lib/libc-2.2.5.so, so=so, useshrplib=false, libperl=libperl. +a gnulibc_version='2.2.5' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: USE_LARGE_FILES Built under linux Compiled at Apr 27 2004 21:32:07 @INC: /usr/local/apps/perl-5.8.4/lib/5.8.4/i686-linux /usr/local/apps/perl-5.8.4/lib/5.8.4 /usr/local/apps/perl-5.8.4/lib/site_perl/5.8.4/i686-linux /usr/local/apps/perl-5.8.4/lib/site_perl/5.8.4 /usr/local/apps/perl-5.8.4/lib/site_perl .
      I don't think I can upgrade the libc on this machine. And I am not entirely sure what you mean by "with or without perl's malloc" - could you elaborate a bit on the pros and cons of that?

      I've just tried it on a very differently configured machine (perl -V output for that one is below) and it doesn't seem to have the same problem, unfortunately the above configuration is what we use everywhere.

      perl -V that works fine (this is the default perl for RedHat enterprise 3.0):

      Summary of my perl5 (revision 5.0 version 8 subversion 0) configuratio +n: Platform: osname=linux, osvers=2.4.21-1.1931.2.393.entsmp, archname=i386-lin +ux-thread-multi uname='linux por' config_args='-des -Doptimize=-O2 -g -pipe -march=i386 -mcpu=i686 - +Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red + Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux - +Dvendorprefix=/usr -Dsiteprefix=/usr -Dotherlibdirs=/usr/lib/perl5/5. +8.0 -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosui +d -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dm +an3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiono +nly -Dpager=/usr/bin/less -isr' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef' useithreads=define usemultiplicity= useperlio= d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=un uselongdouble= usemymalloc=, bincompat5005=undef Compiler: cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS + -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_S +OURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm', optimize='', cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGI +NG -fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm' ccversion='', gccversion='3.2.3 20030502 (Red Hat Linux 3.2.3-19)' +, gccosandvers='' gccversion='3.2.3 200305' intsize=o, longsize=s, ptrsize=l, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2 ivtype='long' k', ivsize=4' ivtype, nvtype='double' o_no', nvsize=, Off_t='', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='gcc' l', ldflags =' -L/usr/local/lib' ldflags_use' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil perllibs= libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libper gnulibc_version='2.3.2' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so', d_dlsymun=undef, ccdlflags='-rdynam +ic -Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE' cccdlflags='-fPIC' ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5', lddlflags='s Unicode/ +Normalize XS/A' Characteristics of this binary (from libperl): Compile-time options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_ +FILES PERL_IMPLICIT_CONTEXT Locally applied patches: MAINT18379 Built under linux Compiled at Sep 15 2003 10:03:52 @INC: /usr/lib/perl5/5.8.0/i386-linux-thread-multi /usr/lib/perl5/5.8.0 /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.0/i386-linux-thread-multi /usr/lib/perl5/5.8.0 .
        And I am not entirely sure what you mean by "with or without perl's malloc" - could you elaborate a bit on the pros and cons of that?
        Perl is usually (by default?) compiled with your libc's malloc. Perl can also be compiled to use its own malloc ( perl -V:usemymalloc ). My suggestion is advice from perlfaq3, particularly from "How can I make my Perl program take less memory?". I really don't know much about it (never use it), but perl's malloc is supposed to be faster (or better suited for perl), but may present a challenge when compiling some eXStension (at least as-hinted-by http://inline.perl.org/java/faq.html). You can find more reading if you grep perl/lib/Pod/ for malloc
        I don't think I can upgrade the libc on this machine.... I've just tried it on a very differently configured machine (perl -V output for that one is below) and it doesn't seem to have the same problem, unfortunately the above configuration is what we use everywhere.
        You'll notice that that the "working" perl is compiled with libc=/lib/libc-2.3.2.so, while the one you use everywhere is compiled with libc=/lib/libc-2.2.5.so. You should be able to compile perl with a newer libc without replacing/upgrading your systems current libc.

        MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
        I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
        ** The third rule of perl club is a statement of fact: pod is sexy.

        gnulibc_version='2.2.5'
        IIRC, that's one of the versions with the buggy malloc that was rewritten for 2.3. Try configuring perl with -Dusemymalloc.
Re: undef quite slow?
by glwtta (Hermit) on Aug 04, 2004 at 08:25 UTC
    Hmm, setting usemymalloc=y doesn't seem to have helped. Though it does seem more than likely that glibc 2.2.5 is indeed the problem.
      Hmm, setting usemymalloc=y doesn't seem to have helped
      Do you mean you reconfigured and recompiled Perl with the -Dusemymalloc option passed to ./Configure? This is something that can only be changed at build-time, not at run-time.

      Dave.

        Sorry, yes, I did rebuild perl - I am not quite naive enough to think these things can configured at runtime.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://379914]
Approved by pelagic
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others lurking in the Monastery: (8)
As of 2024-03-28 09:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found