It's stated very clearly in File::Glob.
If an error occurred, &File::Glob::GLOB_ERROR will be non-zero and $! will be set. &File::Glob::GLOB_ERROR is guaranteed to be zero if no error occurred
perlvar has more to add on $!. (Emphasis in the original.)
This means that the value of $! is meaningful only immediately after a failure. [...] A successful system or library call does not set the variable to zero.
There's even an example in File::Glob's Synopsis. (The example was adjusted to your case.)
@list = bsd_glob($pat, GLOB_ERR);
if (GLOB_ERROR) {
# an error occurred
}