... and use a regular expression to select the files you are interested in.

You can use Text::Glob to transform a file name containing wildcards into a regular expression.

Here you will find another function (_glob_to_regex) implementing a glob to regex transformation.

Don't use glob; use opendir and readdir to loop through the directory entries. Strip out file types you don't want to move (directories, system files, hidden files, etc) and use rename or File::Copy's move function. Alternatively, you can push the files you want to move in an array, move the files when the array is either filled or you run out of files (I think it's quite unlikely that the number of files mod 100 will be 0); in the former case, empty the array and iterate; in the latter, you're done.

Now, are you moving 100 files into a new directory for each 100 files or moving all the files into a single new directory?

It's all fair advice, but why wouldn't zod want to use glob?

I've no idea why zod wouldn't want to; I just offered him a way to do so without using glob.

---

Information about American English usage here and here. Floating point issues? Please read this before posting. — emc

perl -e 'while ($s=<*>) { print $s,"\n"; sleep 5 }'
[download]

As you can see a glob can read filenames one file at a time (hopefully perl really buffers the filenames in the background and doesn't read them in all at once). You might change it to something like this:

use strict;
use warnings;

my @files;
my $filecount=0;
while (my $s=<*>) {
  push @files, $s;
  if (++$filecount>=100) {
    DoTheMoveWith(@files);
    $filecount=0;
  }
}
DoTheMoveWith(@files) if (@files);
[download]

(hopefully perl really buffers the filenames in the background and doesn't read them in all at once)

No, perl reads the entire list of files returned by glob into memory at once. Run the following to test this (WARNING may run for a long time or cause system resource problems on weakish machines).

#!/usr/bin/perl

use strict;
use warnings;

mkdir "gtest" or die "screaming";
for (1..40000) {
    open my $f,">","gtest/$_" or die "gnashing";
    close $f or die "howling";
}


my $c=0;
while (my $r=glob("gtest/*")) {
    $c++;
    if ($c == 1) {
        for (1..40000) {
            unlink "gtest/$_" or die "wailing"
        }
    }
}

print "$c\n";
rmdir "gtest" or die "exhausted";
[download]

I believe this is done so that glob returns a consisten snapshot of the directory contents as they were at some point, regardless of whether the content changes while you process the results. If you want more up-to-date data, with only current files being returned, you'll have to use opendir and readdir.


All dogma is stupid.

perl reads the entire list of files returned by glob into memory at once.

This is what I originally assumed, hence my original question. So, I guess that makes the answer to my question, "No, you can't set a limit on a glob."

Thanks

Thanks. This illustrates a basic misunderstanding I had of the way glob actually works internally.

while(1){print "help, the globs are coming\n"}
[download]

Someone showed me this once here, I didn't actually check it out, but it looked impressive

...famous last words

See: bad bad glob

Given the small number of files you're reading into an array at once (40,000 names isn't a lot, really), I'd grab all of them and then use an array slice to operate on subsets of your array until finished.

Check also here, and here.

A word from the field, and sorry it took me awhile to track down my notes:

I have a script that worked fine for months, doing something like my @pdfs = glob("$path/*.pdf");. One day the script failed, and with use diagnostics; I got this:

internal error: glob failed at ...

(P) Something went wrong with the external program(s) used for glob and <*.c>. This may mean that your csh (C shell) is broken. If so, you should change all of the csh-related variables in config.sh: If you have tcsh, make the variables refer to it as if it were csh (e.g. full_csh='/usr/bin/tcsh'); otherwise, make them all empty (except that d_csh should be 'undef') so that Perl will think csh is missing. In either case, after editing config.sh, run ./Configure -S and rebuild Perl.

There were not "too many files", which I had seen before with glob. This was new... The environment is out of my hands, so I simply gave up with glob and used File::Find.