Xvfb is not like vnc at all. From the Description section of the man page:

Xvfb is an X server that can run on machines with no display hardware and no physical input devices. It emulates a dumb framebuffer using virtual memory.

The primary use of this server was intended to be server testing. The fb code for any depth can be exercised with this server without the need for real hardware that supports the desired depths. The X community has found many other novel uses for Xvfb, including testing clients against unusual depths and screen configurations, doing batch processing with Xvfb as a background rendering engine, load testing, as an aid to porting the X server to a new platform, and providing an unobtrusive way to run applications that don't really need an X server but insist on having one anyway.