On some OSen the ulimit command is solely a shell builtin, not an external program. When you try and run it via backticks Perl is looking at your command and thinking it can cheat and run the command itself rather than invoking a shell (since it's a simple command with spaces as the only shell metacharacter). If your OS is one of those without a separate ulimit, what's really happening is that Perl tries to exec the non-existent program and fails (giving you back an undef, but you don't check for that).

The work around is to explicitly get the shell involved, either by putting some shell metacharacters in your command (qx/ulimit -n 3>&-/) or by invoking a shell explicitly (qx/bash -c "ulimit -n"/).

Update: Tweaked sample metacharacters in first example.