Possible Perl Issue, plz help...

WeNdeL has asked for the wisdom of the Perl Monks concerning the following question:

Let me preface this by stating I am a junior systems admin who knows very little about Perl and tuning servers for specific applications, especially Perl apps.

We have a Netfinity 7000 M10 with the following specs:

4 Pentium II (Deschutes) 400 MHz processors each w/1024KB's of cache
2 Gigs of RAM
3 10K RPM SCSI HDD's
1 15k RPM SCSI HDD's
RedHat 9.0 kernel version 2.4.20-19.9smp

We are trying to utilize a certain PERL program that is acting as a portal (sort of) for our companies day-to-day computing needs. That is all this box is being used for. User's SSH to this box and invoke an instance of said program.

The problem is this...

This program runs a little slow... When an exact copy of this program is thrown on a plain old workstation (a box with a 450MHz cpu and maybe 128 MB's of RAM), it runs much faster than on the Netfinity. Now aside from turning off all unwanted daemons and processes, I haven't done much in the way of tuning this box.

The perl code in question lives on the 15K RPM SCSI HDD by the way...

In an attempt at defining what I mean by slow, I should state that the program consists of a a good number of PERL scripts that get exec'd as needed. The program starts when the user ssh's to said server, starts up an Xterm on their local display and then exec's an initial wrapper script. This initial script contains a menu whose options correspond to PERL scripts that when exec'd, end up replacing the initial parent script. And as expected, there are submenus of options that exhibit the same behavior.

The "slowness" in question here is when the user switches between menus. I am assumming their is some sort of IO bottleneck upon the reading/compilation of each script.

I do not believe that the network is an issue here... When I performed the intial test with my workstation versus the server, the same network was used and dramatic results were observed. I even ran the code locally on the server in question and experienced the same "slowness".

The perl install has evidently compiled with threading enabled.

# perl -V
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuratio
+n:
  Platform:
    osname=linux, osvers=2.4.20-2.48smp, archname=i386-linux-thread-mu
+lti
    uname='linux stripples.devel.redhat.com 2.4.20-2.48smp #1 smp thu 
+feb 13 11:44:55 est 2003 i686 i686 i386 gnulinux '
    config_args='-des -Doptimize=-O2 -march=i386 -mcpu=i686 -g -Dmyhos
+tname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, 
+Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendo
+rprefix=/usr -Dsiteprefix=/usr -Dotherlibdirs=/usr/lib/perl5/5.8.0 -D
+useshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_
+semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext
+=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -D
+pager=/usr/bin/less -isr'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
+ -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_S
+OURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2 -march=i386 -mcpu=i686 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGI
+NG -fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='3.2.2 20030213 (Red Hat Linux 8.0 3.2.2-
+1)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil
    perllibs=-lnsl -ldl -lm -lpthread -lc -lcrypt -lutil
    libc=/lib/libc-2.3.1.so, so=so, useshrplib=true, libperl=libperl.s
+o
    gnulibc_version='2.3.1'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynami
+c -Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl):
  Compile-time options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_
+FILES PERL_IMPLICIT_CONTEXT
  Locally applied patches:
        MAINT18379
  Built under linux
  Compiled at Feb 18 2003 22:19:53
  @INC:
    /usr/lib/perl5/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/5.8.0
    /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.0
    /usr/lib/perl5/vendor_perl
    /usr/lib/perl5/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/5.8.0
[download]

After speaking with the developers, I have assessed that they are not making use of threads nor are they using any forks to spawn new processes. They are using "exec's" and "system calls" to accomplish the tasks at hand.

Any ideas of what the problem may be? I am being told that it is a systems/server issue. I have been seeking knowledge relating to any performance enhancements I can make to help this program operate but have come up with very little if anything at all.

Care to help?

Thanks in advance...

Comment on Possible Perl Issue, plz help... Download Code

Replies are listed 'Best First'.
Re: Possible PERL Issue, plz help... by erasei (Pilgrim) on Oct 16, 2003 at 17:21 UTC
The world of software optimization is a many headed beast. Your software may not even be the cause at all. There are literally hundreds of different things that could be causing your slowness. Here are what are most obvious to me to check: What does your system look like when the scripts are being run? Running 'top' from the server, is it CPU locked, IO (disk) locked? Network locked? What version of Perl (btw: its "Perl" not "PERL") are you running on the workstation? Is it the same threaded version of 5.8? If not, that might be your issue. The threaded Perl binary might be slowing you down. Don't think so though, but check it out. How many people are running the script at one time? Obviously there are more people on the server than on your workstation. Does it do any sort of database interaction that might be slowed down by table locks? Those are a few things to check out.	[reply]
Re: Re: Possible PERL Issue, plz help... by WeNdeL (Novice) on Oct 16, 2003 at 18:22 UTC
The Perl install on the WorkStation is 5.8 as well. It makes use of threads... Well... when I load up the program and begin using it, I noticed that the CPU usage on only one of the processors jumps up to almost 100% at times. The memory usage is quite minimal. I have been testing this on a development server so not many people (if any) are utilizing it. And yes, there is some database interaction involved but the Server in question is using the same database that the WorkStation uses. /me shrugs I am still stumped... gah... this hurts! Could you explain "Disk locked" to me? How do I check this? Thanks for your help!	[reply]
Re: Re: Re: Possible PERL Issue, plz help... by l2kashe (Deacon) on Oct 16, 2003 at 18:46 UTC
The fact that your proc jumps up that much is a decent sign. It more than likely means your process is not contending with any other processes for time on the proc. What I would be interested in seeing is the output from a vmstat 5 10. One prior to initating the process and another while the processes are running. Without really looking into it too much, I want to point a finger at the I/O subsystem. I recently had similar problems with system utilities on a particular host. Come to find out the RAID controller was flaky. I couldn't see much of an issue, but it was the only thing I hadnt tested. So the disks got moved off of the hardware based controller, and ported over to software RAID. Viola the issue went away. Its not much without a more fine tuned analysis of your particular system, but its another place to look. Seeing as how the same code runs flawlessly on a different host, it is on initial assumption more than likely hardware based. Another alternative route looking at a software level would be to use the 'strace' program to watch what is actually happening during runtime on that host. Its not the easiest tool to use at a 'junior' level, but will give great insights into whats going on. use perl;	[reply]
Re: Re: Re: Re: Possible PERL Issue, plz help... by WeNdeL (Novice) on Oct 16, 2003 at 19:12 UTC
Re: Re: Re: Re: Re: Possible PERL Issue, plz help... by l2kashe (Deacon) on Oct 16, 2003 at 19:17 UTC
Some notes below your chosen depth have not been shown here
Re: Re: Re: Possible PERL Issue, plz help... by erasei (Pilgrim) on Oct 16, 2003 at 18:37 UTC
Disk locked just means that your system is waiting on a read or write to or from the hard disk. Not very likely with your current disk set up though. The one CPU being at 100% is likely your problem. Here's why: You have a quad cpu system, and a threaded version of perl, but based on your original post, your script isn't making use of the threads. That means that it is going to run serially, thus only making use of one processor. Now, in theory, each 'exec'ed script would fire off on a different processor, but depending on how its being called it might not be the case. Each user will get partial use of a different CPU. Meaning if 1 user connects he uses CPU1, another connects and she gets use of CPU2. That helps quite a bit. However, User1 can't use cpu1 and cpu2 at the same time. That hurts.	[reply]
Re: Re: Re: Re: Possible PERL Issue, plz help... by WeNdeL (Novice) on Oct 16, 2003 at 19:02 UTC
Re: Possible Perl Issue, plz help... by sauoq (Abbot) on Oct 16, 2003 at 22:13 UTC
I do not believe that the network is an issue here... When I performed the intial test with my workstation versus the server, the same network was used and dramatic results were observed. Does the application itself access the network? You mentioned a database in another post; is that database on a remote machine? It may be "the same network", but you might not be able to rule out network related issues unless you can show that both machines (fast and slow) are both configured exactly the same. Something as simple as an incorrect duplex setting could result in the symptoms you've described. I even ran the code locally on the server in question and experienced the same "slowness". If the application in question doesn't require network access itself, then you've pretty much ruled out network related problems with that test. If it does, though, the problem may be a simple misconfiguration. Don't let it be a blind spot in your investigation. You'll kick yourself if you swap out controllers and drives only to find out it was something simple. (Yes, this is the voice of experience... ;-) -sauoq "My two cents aren't worth a dime.";	[reply]
Re: Possible Perl Issue, plz help... by calin (Deacon) on Oct 16, 2003 at 18:55 UTC
We have a Netfinity 7000 M10 with the following specs: What a horrifying piece of old heavyweight junk! You'd better donate it to your local yachting club as a boat anchor (but I guess it would be too heavy even for that!) Sorry I couldn't help. I'm an IBM servicer and sometimes I have to deal with these beasts. One is ~73kg (~160lbs) empty!	[reply]
Re: Re: Possible Perl Issue, plz help... by WeNdeL (Novice) on Oct 16, 2003 at 19:23 UTC
lol... you should see what we use around here... have a good laugh on me... here is a list of "junk" that we currently utilize: Netfinity 3500's Netfinity 5500's Old Netfinity 7000's (non M10's) PC Server 325's Tons of 300GL's (for our workstations) And they make millions for the owner's... >:\ NOTE: I am not an owner	[reply]