|No such thing as a small change
Techniques to cache perl scripts into memory ?by fx (Pilgrim)
|on Dec 17, 2010 at 14:47 UTC
fx has asked for the wisdom of the Perl Monks concerning the following question:
One of my systems is a large multi-user Linux setup with several hundred users frequently accessing a selection of utility scripts I've written in Perl. These aren't big or complicated scripts but at peak times the result can be several calls per second to several scripts.
Monitoring the resources of my machine I can see this is absolutely killing the disk. CPU and RAM seem to be ok. I've also tested running this from a Solaris platform (in case this was a Linux OS specific issue) but the same is observed there.
Perl and all modules are installed on a RAID1 pair of 15K 6G SAS disks. This is as far as I can go with the money available for hardware (no SSD I'm afraid!). I've also tested a single 15K disk (to remove any RAID overhead) but no joy.
We've had a similar problems before with standard command line tools and in those cases I statically compiled them and placed the resulting binaries in a RAM disk. The OS seemed to understand what I was after and the disk was hit a lot less. Of course, they were written in C and the same isn't really possible with Perl. I've tried "pp" but it doesn't seem to work out...
I've messed around with the idea of moving my utilities to a TCP socket based "service" (to save the massive number of short lived processes being started) and having users call into that. Problem there is that my users would have to rewrite a lot of their calling code and some basic performance testing didn't really blow me away.
I'm looking for some new suggestions to investigate. Has anyone met a similar issue? I know a good OS (and I'm counting Linux in that mix) should be caching regularly used things in memory anyway but I think I've simply got too many different little things being called for that to work. Any ideas as to how to cache Perl scripts into memory and leave my poor disk alone?........
fx, Infinity is Colourless