I built a system similar in design to Joost's. The only siginficant difference was that once the client was assigned to a worker, the "server" got out of the way and let the two communicate directly. In my case each client could have had multiple workers (implemented as virtual machines) all on the same physical machine. The server was perl, the workers were under control of a perl executive, and the clients were a Java/C combination.

HTH