Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Copying a directory and its contents wihile displaying a progress bar

by bittis (Sexton)
on Aug 04, 2008 at 11:55 UTC ( [id://702035]=perlquestion: print w/replies, xml ) Need Help??

bittis has asked for the wisdom of the Perl Monks concerning the following question:

Hi everyone,
I have a question as to how i can copy a directory and its contentsd from one location to another while displaying to a user a progress bar so that he knows how much time there is left or atleast that there is something happening.

One idea i have is using File::Find and displaying something like dots on the screen after each copy operation has taken place, but this does not give a realistic progress bar as one file might be in the size of 1 gig in size and the user might be unsure of whether or not the copying is in progress.

Any ideas? :)

Replies are listed 'Best First'.
Re: Copying a directory and its contents wihile displaying a progress bar
by dHarry (Abbot) on Aug 04, 2008 at 12:16 UTC
Re: Copying a directory and its contents wihile displaying a progress bar
by zentara (Archbishop) on Aug 04, 2008 at 14:46 UTC
    You need to be more specific in what you mean "copy a directory from one location to another". Is this a local operation? Or over the network? A local operation can be so fast, that the progress indicator will not have time to work. I usually use Midnight Commander for my local moves, and it shows a nice curses progress bar for huge files.

    You would make your life alot easier if you can tar or zip up your directory, and just move the single file.

    Since you mention File::Find, I will assume a local move. If you want to do this on a local filesystem, you would need to open the source dir and target dir, then manually open and sysread each file in small chunks, then as you write them to the target dir, make your 'progress dot'. You could do a total byte count before starting the copy, so you could get a byte percentage. The problem with doing this, is it will slow down all your file transfers..... so why bother?

    In general: when you want to show progress, you need to intercept the callback that does the copying. Like this with LWP. ( Also see Track file upload progress )

    #!/usr/bin/perl -w use strict; use LWP::UserAgent; # don't buffer the prints to make the status update $| = 1; my $ua = LWP::UserAgent->new(); my $received_size = 0; my $url = 'http://www.cpan.org/authors/id/J/JG/JGOFF/parrot-0_0_7.tgz' +; print "Fetching $url\n"; my $request_time = time; my $last_update = 0; my $response = $ua->get($url, ':content_cb' => \&callback, ':read_size_hint' => 8192, ); print "\n"; sub callback { my ($data, $response, $protocol) = @_; my $total_size = $response->header('Content-Length') || 0; $received_size += length $data; # write the $data to a filehandle or whatever should happen # with it here. Otherwise the data isn't saved my $time_now = time; # this to make the status only update once per second. return unless $time_now > $last_update or $received_size == $total_s +ize; $last_update = $time_now; print "\rReceived $received_size bytes"; printf " (%i%%)", (100/$total_size)*$received_size if $total_size; printf " %6.1f/bps", $received_size/(($time_now-$request_time)||1) if $received_size; }

    I'm not really a human, but I play one on earth Remember How Lucky You Are
      Ah i see what you mean, so just displaying dots is good enough and then just tell the user that "file: lalala.zip has been copied over" for every file that gets copied
Re: Copying a directory and its contents wihile displaying a progress bar
by pjotrik (Friar) on Aug 04, 2008 at 12:53 UTC
    Read the list of files with readdir/File::Find and their sizes by -s (Dir::List seems to be able to do the job as well), compute the ratio. Alternatively, Filesys::DiskUsage may give you the total size.
Re: Copying a directory and its contents wihile displaying a progress bar
by leocharre (Priest) on Aug 04, 2008 at 13:27 UTC

    I know you can use Smart::Comments to display a progress bar. I read it slows down the program- but likely at compilation time and should make absolutely no difference in your case- since we are already anticipating it may take a while.

    Is the interface to your software via the terminal? Is this an educated user, thus? Why wouldn't they just use use 'top' ?

    How are you getting a list of files to copy? Are you making a system call like 'cp -R ./fromdir ./todir', or are you using File::Find, and then File::Copy one by one?

    Why is it taking so long, for a gig?

    What's 'so long'? A minute, twenty.. What speed is your hard drive read/write rate- 44100 rpm ? 15000 rpm? What's your filesystem? What's your operating system?

    Would you consider benchmarking the difference between copying 1 gig via File::Copy and system cp ?

    I have some code that creates random junk on disk to benchmark md5 sum calls, might be useful.

      Hey there, my situation is this, i will be checking if the user plugged in an external usb hard drive (checking the label of the drive + some other checks) and then i will be copying one directory from there to the local hard drive. Files in that directory will vary in size from 1 mb to 1.5 gig in some instances (not always). It currently takes around 5 minutes to copy from e:\myFolder to c:\myFolder if i do it using windows explorer. I have not decided how i will copy the files as it will depend on how i can get that progress bar or an indication of how long it will take. My idea so far is to use File::Find as i can atleast give the user an indication of what is happening at the end of copying a file. What do you think?
Re: Copying a directory and its contents wihile displaying a progress bar
by Your Mother (Archbishop) on Aug 04, 2008 at 16:38 UTC

    Pseudo-code (unchecked, just thrown out there, etc - Time::Progress). Sorry it's not more complete.

    use Time::Progress; my ( $total ) = /?/ =~ qx/ du -something -or -other target_dir / my $prog = Time::Progress->new(); $prog->attr( min => 1, max => $total ); while ( file_finding() ) { $size += $size_of_current_file print $prog->report( "%70b %p\r", $size ); } $prog->stop; # report times print $prog->elapsed_str; print $p->estimate_str;

Re: Copying a directory and its contents wihile displaying a progress bar
by hiseldl (Priest) on Aug 04, 2008 at 22:01 UTC

    Assuming you can install File::Copy::Recursive, you can use the following code to wedge in a progressbar, or whatever you want.

    I added File::Find in there so that I could calculate the sum of all the file sizes, and then use that in my version of the progress bar.

    N.B. I put the sleep command in there only for demo purposes, remove it for production code ;-)

    use File::Find; use File::Copy::Recursive qw(dircopy); use strict; use vars qw($dir_from $dir_to $size_all $size_remain *mycopy); $|=1; # turn off buffering is needed in this case # save the current version of copy that is used by F::C::R *mycopy = *File::Copy::Recursive::copy; # replace the copy func so we can wedge in a progressbar *File::Copy::Recursive::copy = *mycopy_func; # 1. call the original copy func # 2. calculate remaining bytes # 3. and call the show progress func sub mycopy_func { &mycopy(@_); -f $_[0] and $size_remain -= -s $_[0]; mycopy_showprogress($size_remain); } # this is called after every file is copied sub mycopy_showprogress { my ($remaining) = @_; printf "%s of %s remaining. \r",$size_remain, $size_all; sleep 1; ##### FOR DEMO ONLY, REMOVE OTHERWISE } # setup the directories to copy $dir_from = "/tmp/from"; $dir_to = "/tmp/to"; # Calculate the sum-total of the sizes of the files in the from dir $size_all = $size_remain = 0; find(sub {-f and $size_all+=-s;},$dir_from); $size_remain=$size_all; # Do it! dircopy($dir_from, $dir_to); # some visual cleanup print "\n"; __END__

    Here's another version that does not calculate the sum, rather it just prints the name of the file it is copying.

    use File::Copy::Recursive qw(dircopy); use strict; use vars qw($dir_from $dir_to *mycopy); $dir_from = "/tmp/from"; $dir_to = "/tmp/to"; sub mycopy_func { &mycopy(@_); mycopy_showprogress(@_); } sub mycopy_showprogress { my ($remaining) = @_; printf "copying %s to %s. \r",@_; sleep 1; ##### FOR DEMO ONLY, REMOVE OTHERWISE } $|=1; *mycopy = *File::Copy::Recursive::copy; *File::Copy::Recursive::copy = *mycopy_func; dircopy($dir_from, $dir_to); print "\n"; __END__
    HTH!

    --
    hiseldl
    What time is it? It's Camel Time!

Re: Copying a directory and its contents wihile displaying a progress bar
by Illuminatus (Curate) on Aug 04, 2008 at 17:35 UTC
    A slightly verbose example shows %completion as it runs
    #! /usr/bin/perl use strict; use File::Copy; my $tot = 0; my $SIZE_IND=7; my $smallest=2000000000; my $newLocation="/tmp"; my $some_dir = "/home/drew/temp"; opendir(DIR, $some_dir) || die "can't opendir" ; my @stats; my @dots; my $some_dir; my %dirFiles; @dots = readdir(DIR); my %fileSizes; for (my $file=0;$file <= $#dots;$file++) { if (($dots[$file] eq ".") || ($dots[$file] eq "..") || ($dots[$file] =~ /^[ ]+$/) || (-d $dots[$file])) { next; } $dirFiles{$dots[$file]} = 1;; @stats = stat $dots[$file]; $fileSizes{$dots[$file]} = $stats[$SIZE_IND]; $tot += $stats[$SIZE_IND]; if ($stats[$SIZE_IND]) { $smallest = ($smallest < $stats[$SIZE_IND])?$stats[$SIZE_IND]:$sma +llest; } } print "\n\r"; my $totPercent = 0.0; $tot.=".00"; print "\n\r"; $|=1; foreach my $file (keys(%dirFiles)) { copy($file, "/tmp/$file"); $fileSizes{$file}.=".1"; my $curPercent = ($fileSizes{$file}/$tot) * 100.0; $totPercent += $curPercent; printf STDERR "%2.2f%% complete\r", $totPercent; sleep 1; }
Re: Copying a directory and its contents wihile displaying a progress bar
by Khen1950fx (Canon) on Aug 04, 2008 at 19:37 UTC
Re: Copying a directory and its contents wihile displaying a progress bar
by blazar (Canon) on Aug 14, 2008 at 17:36 UTC
    One idea i have is using File::Find and displaying something like dots on the screen after each copy operation has taken place, but this does not give a realistic progress bar as one file might be in the size of 1 gig in size and the user might be unsure of whether or not the copying is in progress.

    I personally believe that no progress bar in the whole universe has ever given realistic info, but that of the supa dupa new gen coffee machine they installed at my last employer's! (The very second the cool electric blue light of the bar would touch the end mark, the very last drop of coffee would be spilled into the "glass" and the machine would make 'ding' for you to extract it and happily drink it!) But seriously, I think that your proposed strategy -i.e. sticking to the granularity of atomic copies- is fine enough: only, at some times one would see the progress bar seemingly not proceeding if some very big file is taking too time to be copied - which is what I gather may happen in your case. For otherwise you should either hack into some file copying modules: a literal overkill! or work around with some estimate: which would be both an overkill and prone to be imprecise, like all estimates.

    The point to which I wanted to draw your attention is rather that whatever file copying strategy and whatever progress bar technique you choose amongst the many ones that were proposed to you in this rich thread, if you want your progress bar to be accurate, i.e. to reach the end mark as the copy process also terminates, then you have to gather all the files you have to copy in advance whereas if you hadn't such a requirement, you may start copying files as soon as they're found by your favourite file finding module...

    --
    If you can't understand the incipit, then please check the IPB Campaign.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://702035]
Approved by Corion
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having an uproarious good time at the Monastery: (5)
As of 2024-03-29 01:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found