When you request a directory on a webserver the server gets to decide what to send you - usually index.html but not necesarily. This is straight ought of lwpcook.pod:

"If you just want to check if a document is present (i.e. the URL is valid) try to run code that looks like this:

  use LWP::Simple;

  if (head($url)) {
     # ok document exists
  }
[download]

. . ."

. . . which is the "canonical" way to make sure a url is valid.