My first big project at my first big job was an apache module that took in some search terms and some categories as input, opened some files on disk that cached some meta-data for the category, and then did a search on a search index using the user input and the meta-data.

The module worked fine, it ran fast, i had no problems in development or in qa, and we ever ran it through PMA to make sure you could respond quickly.

when we deployed it live, it worked fine for a little while and then the whole machine would seeze up and not do anything. It turns out i wasn't closing the meta-data files, And the apache "threads" (sub-processes) where long lived, and haning on to the open file handles untill they exited.

The problem didn't show up in QA because they didn't hit it with a lot of concurrent load. It didn't show up in dev (even though i specificly tried hammering it with lots of requests) because the dev server had a much shorter time to live for the thread/process pool. It didn't show up in PMA (even though we were hitting it hard, and it was configured just like production) because the client we were using to trigger load was testing lots of permutations of input was doing a lot of randomness on the search terms, but not the category ... so the processes were usually getting lucky and typically allready had the filehandle that they needed opened.

it actually turned out that we were occasionally hitting the problem in PMA, it was just blocking untill another process died and released some filehandles ... once i put in the appropriate "close" call, the average response time was cut in half because of the eliminated blocking.