My concern is not about the TCP handshake but more on the server resources.

So you want to be a friendly net citizen. Good. Open one connection, with keep-alive, and ask for one resource after the other. This is the solution with the least load on the server.

The problem starts when the server is busy with a lot of requests, it needs to allocate resources for each request and there for some of then take longer the 1 second.

Why do you insist on that one second timeout? There is no guaranteed response time in Ethernet, IP, TCP or HTTP. Even your own OS does not guarantee that your program will run again one second after issuing a system call. Sure, most times, it will run again much earlier. And like with your own OS, most times, a HTTP server somewhere on this planet answers within one or a few seconds. If you have realtime requirements, use realtime operating systems, realtime networks, and realtime protocols. Standard ethernet, most Linux, *BSD, and Windows distributions, Apache, Perl and HTTP all can't cope with real-time requirements. (Note that realtime does not mean fast response times. It means guaranteed response times under all conditions, even under maximum load. Defining what is guaranteed is a completely other story and may end with a guaranteed response time of five seconds.) If you don't have realtime requirements, forget that funny timeout.

Your server seems to be too small or too loaded. Think about the algorithms used on the server. Can you use an O(n) or an O(n log n) algorithm where you currently use an O(nē) algorithm? Think about pre-calculating data and caching on the server, for example with memcached. Can the server deliver you an all-in-one document containing everything you really want, probably unstyled and packed, instead of having you spider tons of pages?

Throw some money at the problem: If we talk about an all-in-one server with email, web, database, and loads of other jobs, spread the jobs on several dedicted servers. Use one machine for web, one machine for the database, one machine for email, one machine for each of the other jobs. Separating web server and database usually gives you the "biggest bang". Get a server with faster CPUs, perhaps multi-core CPUs, a faster harddisk subsystem, more RAM, and a faster network connection. If off-the-shelf hardware doesn't help, ask experts for high-performance computing. Don't be scared by the prices. You can easily spend more money for a single high-performance computer than for a new car.

Alexander