Super short version: client thinks the server is slow, server thinks the client is slow. Sounds like a network problem to me. You might want to try doing some network diagnostics while experiencing these slow downs. There are many ways in which things could be messed up: bad hardware, bad wires, bad configurations. And it could be at the IP layer (if the server and client are not on the same LAN), or at the ethernet layer.
Specific examples I've seen like this are: bad ethernet cables (sad but true), some sort of @#$%ed up arrangement of your switches (cycles or diamonds in your topology), ethernet interfaces renegotiating their speed/duplex, instead of just being locked down, bad routes, etc.
Anyway, just a wild stab in the dark.
------------
:Wq
Not an editor command: Wq