Just a suggestion, but if your intent is to spread out files across a directory tree to improve access time (e.g. you've got an older Linux using ext2 that drags on directories with over ~1000 entries) or have an NFS server that doesn't allow over n bytes worth of filenames in an inode, you might make a tree of [[:hexdigit:]]+/[[:hexdigit:]]+/realfile, where the [[:hexdigit:]]+ chunks are derived using Digest::MD5 or the like from the realfile's name.

(Just to toss that idea out as I've had to deal with both the problems I mentioned in the past . . .)

Update: The OP contacted me out-of-band with questions about how exactly to apply this to their situation. I'll answer them here just in case anyone else was interested . . .

The way I've always used this in the past is to basically use the "filename" as a key that gets run through MD5 to create the real on-disk filename. If you can regenerate the key easily (say it's a log file of the form "username-month-year") you don't really need to keep the original filenames around; otherwise you'll want to keep a list of just keys (possibly using DB_File, a real RDBMS, or a flat file) to use as a table of contents. Whichever way you go, what you want to write is a key2path( ) routine which you pass in the key and get back the real on disk path. This example does two levels deep (so you need code to make directories 00 .. ff and then 00/00..ff, 01/00..ff, ...) and uses the hashed key as the pathname (although there's no reason you couldn't use $key instead of $digest; the reason I didn't in the most recent case I used this was that it was the length of the original filename causing problems to begin with and the key was readily available externally):

sub key2path {
   my $key = shift;
   my $digest = Digest::MD5::md5_hex( $key );

   return substr( $digest, 0, 2 ) . "/" . substr( $digest, 2, 2 ) . "/
+" . $digest;
}
[download]

I never even thought of that. Great suggestion! BTW - you hit the nail on the head about the problem. NFS isn't the best when you start dealing with large file sets under one directory, so of course the solution is to spread the files out and preserve speed.

In reply to the Update

This seems to be a much simpler approach. It definitely saves on a lot of code, and takes out the confusing recursion. Thanks for offering a viable alternative that was in effect much easier to implement.

• Use == for numeric equality, not eq.
• Use the builtin mkdir, rather than system "mkdir..."
• Consider whether your recursion ever bottoms out (it doesn't seem like it does)

• use strict;
  use warnings;
  [download]

  yield several alarming messages about variables that "will not stay shared"

Yup, $depth is probably getting set on the first invocation and never getting updated un subsequent invocations. See this article

---

"The dead do not recognize context" -- Kai, Lexx

Yes, this is exactly the problem! The article you sent me to got me on the right path, but I thought it was too brief, or at least it didn't offer enough examples for me to figure out what was happen. You are right, however, the root of my problem was exactly that.

The following I thought was a very good article on the same thing, although their code is a little cryptic in some places. http://www.serverwatch.com/tutorials/article.php/10825_1128811_1

Finally The following is the re-written fileDir sub-routine that works just fine (although I think it looks a little ugly to me).



sub findDir{
    
    my ($dir, $min, $max) = @_;
    $min_depth = \$min;
    $max_depth = \$max;
    #print "findDir: $dir min: $min_depth max: $max_depth\n";
    $count_files = *count{0};
    
    find( {
        preprocess => \&preprocess,
        wanted => \&wanted,
    }, $dir);
    
    #print "count: $count_files\n";
    
    sub preprocess {
        my $depth = $File::Find::dir =~ tr[/][];
        #print "depth: $depth\n";
        return @_ if $depth < $$max_depth;
        print "depth: $depth max: $$max_depth\n";
        if ($depth == $$max_depth){
            print "greping\n";
            return grep { -df } @_;
        }
        return;
    }
    
    sub wanted {
       
        my $depth = $File::Find::dir =~ tr[/][];
        return if $depth < $$min_depth;
        
        if(!($_ =~ m/^\./)){
            $$count_files++;
        }
    }
    return $$count_files;
    
}
[download]

I thought about using the mkdir() from perl, but my problem is that if a nested directory structure doesn't exist

i.e. /foo/bar/foobar/

then the code

mkdir("/foo/bar/foobar/");

is going to fail. Therefore I used a system call with the -p option so that any depth of directory will be created without failures.
As for the recursion bottoming out, just run the code. If it runs inifinitely then it doesn't bottom out. If it exits out, then it has a base case.

Thanks for the pointers.

Check out File::Path - it has a function mkpath which solves that problem for you. Anything to avoid an unnecessary system call ;-) (Also note that this is part of the standard perl distribution so you can just use it anywhere without having to require extra modules.)

Because those subroutines are named, they're defined once, and what they use for $max_depth et al is a matter of some consternation for me. I haven't played with that sort of deep magic, myself, so I can't give you a definite diagnosis, but I think your troubles are in that area.(Even though as I think about it, it seems like it should be ok.)

Try defining the subs like

my $wanted = sub {
...
[download]

That doesn't make sense since the line of code that is being affected is the following

sub recursion{
        findDir($dir, $min, $max);
}
sub findDir{
    my ($dir, $min_depth, $max_depth) = @_;
}
[download]

granted that is a scaled down version, but for brevity's sake you get the idea. No matter what is being passed after $min and $max == 2 is not being updated in the findDir directory. It seems like the array @_ is holding on to old values, but even when I use

$dir = shift;
$max = shift;
$min = shift;
etc ...
[download]

I still get the wrong data. This has to be some property of recursion that I am missing...