What OS, perl version and IO::Socket versions are you using ?
There is useful disgnostic instruction, as well as proposed solutions at http://bugs.gentoo.org/show_bug.cgi?id=58122.

"On some operating systems the Perl module IO::Socket reports obscure error
on unsuccessful connect() in non-blocking mode as: "Invalid argument" in
place of the true cause: "Connection refused". The remote service should be
checked if available and accessible (firewall rules? network problems?).

Another possible reason for "Invalid argument" reported by IO::Socket when
running within chroot jail is some network-related system file missing in a
chroot jail, e.g. /etc/protocols, /etc/services, /etc/netconfig,
/etc/resolv.conf, /etc/nsswitch.conf, /etc/hosts, /etc/host.conf
-- depending on the operating system in use. System debugging tools
(debugger, strace, truss) may help to find the cause. "