Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

NET::FTP : Recursively search FTP directories (was: Net::FTP Problem)

by thawk (Novice)
on May 31, 2001 at 08:50 UTC ( [id://84458]=perlquestion: print w/replies, xml ) Need Help??

thawk has asked for the wisdom of the Perl Monks concerning the following question:

I'm working on a project that will be kind of like palavista.com or oth.net. What I'm trying to do is write a script that will log in to several FTP's at given intervals and log all of the files on the server to a MySQL DB to be searched by users. I already have the script doing basically what I want but I am having trouble making it change directories and list the contents of those directories. Keep in mind that many FTP servers go 10+ directories deep and I need to get the contents of each directory without missing directories along the way. I'm at the end of my rope. Any help would be GREATLY appreciated.

Edit 2001-06-02 by mirod: changed the title

  • Comment on NET::FTP : Recursively search FTP directories (was: Net::FTP Problem)

Replies are listed 'Best First'.
Re: Net::FTP Problem
by grinder (Bishop) on May 31, 2001 at 14:10 UTC

    It's been done already. Check out fsync, the file synchroni[sz]ation utility.


    --
    g r i n d e r
Re: Net::FTP Problem
by mullr (Sexton) on May 31, 2001 at 09:35 UTC
    This sounds to me like a classic recursion problem. The trick is to have one function you call on a directory which goes throught the listing, calling itself for each directory found. This should should allow you to traverse an entire directory tree without any trouble.

    Hope this helps.

    See you, space cowboy

      In case your code is working correctly on some but not all servers, consider that you are dealing with a slow network bound system. Don't make a lot of children trying to open a bunch of directories at once. Consider it could fail if the server is busy. Check the listing you get back and make sure you aren't trying to get the same (.) directory repeatedly or recursing upwards beyond the public readable folder, etc.

      Basically if you can do exactly what your program is supposed to do, perhaps interactively telling it to go one step at a time and seeing what it sees, you should be able to debug.

Re: Net::FTP Problem
by the_slycer (Chaplain) on May 31, 2001 at 09:31 UTC
    What you will need is a good recursive File Finder.
    You might want to poke at the source of File::Find.
    Failing that, if you run a Super Search using find file you will get a whole whack of hits, some with bad recursive search methods. Some with better. Including some here. ;-)

    I think we all agree that File::Find is the best.
      I'm not sure if it shows but he's using Net::FTP, which AFAIK isn't implemented in File::Find. This can turn out to be quite tricky, since FTP's filesystem isn't as accessible as a local one.

      Greetz
      Beatnik
      ... Quidquid perl dictum sit, altum viditur.
        No, I knew that he was using Net::FTP - notice that I said to poke at the source of File::Find. Basically just needs a recursive routine, I was just pointing him to one that is used by most people that know better.
Re: Net::FTP Problem
by thpfft (Chaplain) on May 31, 2001 at 18:22 UTC
    I suggest you have a poke around the insides of mirror. It should be a relatively simple job to adapt part of it to list rather than retrieve files.

    I've just had a quick look, and i think lsparse.pl will answer a lot of questions. i was startled to find that it isn't even using strict, but it's very readable.

Re: Net::FTP Problem
by Malach (Scribe) on Jun 01, 2001 at 06:34 UTC
    #!/usr/bin/perl use Net::FTP; $ftp = Net::FTP->new("host.name"); $ftp->login("user","password"); @data=$ftp->ls("-Rla"); $ftp->quit; foreach $line (@data) { print $line."\n"; }

    From there, it's just text parsing......

    Malach
    So, this baby seal walks into a club.....

      Are these switches ("-Rla") implemented the same in all FTP servers? There's like a billion servers out there, and not all of them are totally standards compliant.

      If most of them do use these switches correctly, this is an excellent way to recurse (IMHO). If not, you may need to look a little harder.

Re: Net::FTP Problem
by perldoc (Scribe) on Jun 02, 2001 at 04:45 UTC
Re: Net::FTP Problem
by scottstef (Curate) on Jun 01, 2001 at 21:12 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://84458]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (3)
As of 2024-04-25 22:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found