You could write a multitasking server in POE and register a session in it for each accept()ed client connection, and have those sessions post events back to the session(s) which talk(s) to AstMan, which you could also spawn dynamically based on responsiveness, latency and load.

POE::Component::Client::Asterisk::Manager may help, if you're looking to POE for a potential solution.