Perhaps it helps if you'd start with just 1 'connection'. For example, let's assume the server only listens on port 8003. Recently (AnyEvent::Handle::UDP -- can't get it to receive?) there was a question involving threaded UDP traffic, you might be able to use that.
Also, it might be a good idea to create a big state-diagram, keeping track of who (server/client) is doing what (waiting for hb, waiting for hb response, waiting for data (hexdumps), sending data) but also when responses will be ignored (as pointed out in earlier comments).
Next to that, consider what happens if you *do* (and, given that you're told to use UDP, you must expect that at some point you will) lose a packet (at any point in the state-diagram). Which data-loss will result from that, and will your server/client be able to recover from it, or will they get stuck in a state where they're waiting for something which never arrives and need to be restarted completely?