My point is that you have to ask for the time before every sleep. I don't see your code, so you might already be doing that. Of course, if you spawn an external process for querying the time, then that's slow, so don't do that.
I've switched to using Time::HiRes simply for portability, but it doesn't help with the latency issue at all. Starting a process is fast and predictable. The problem is that regardless of which method I use for sleeping, the latency is very long, and is unpredictable.