Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

read files one by one in directory and redirct output to a file

by TonyNY (Beadle)
on Jul 06, 2018 at 02:20 UTC ( [id://1218012]=perlquestion: print w/replies, xml ) Need Help??

TonyNY has asked for the wisdom of the Perl Monks concerning the following question:

Hi I'm trying to read the contents of files in a directory and then redirect the output to a txt file

When I try the following code I can read the contents of one file but not all the files in the array

#$dirname = "/logs"; #opendir DIR, $dirname or die "Couldn't open dir '$dirname': $!"; #my @files = readdir(DIR); #closedir DIR; $file = "/logs/oracle.log"; #foreach my $file (@files){ #print "$file\n"; open( my $fh, '<', $file ) or die "Can't open $file: $!"; while ( my $line = <$fh> ) { print $line; } # } close $fh;

Replies are listed 'Best First'.
Re: read files one by one in directory and redirct output to a file
by huck (Prior) on Jul 06, 2018 at 04:32 UTC

    "If you're planning to filetest the return values out of a readdir, you'd better prepend the directory in question. Otherwise, because we didn't chdir there, it would have been testing the wrong file."
    http://perldoc.perl.org/functions/readdir.html

    Ditto for opening for read/write

Re: read files one by one in directory and redirct output to a file
by tybalt89 (Monsignor) on Jul 06, 2018 at 13:00 UTC

    A non-module approach (or: why do explicit loops when perl will do them for you?).

    #!/usr/bin/perl # https://perlmonks.org/?node_id=1218012 use strict; use warnings; my $dirname = 'some/test'; my $outputfile = 'some.output.name'; local @ARGV = grep -f, glob "$dirname/*"; @ARGV or die "no files in $dirname"; open my $out, '>', $outputfile or die "$! opening $outputfile"; print $out $_ while <>; close $out;

      why do explicit loops when perl will do them for you?

      Well, there's more than one way to have less explicit loops. Not all of them require playing dirty with ARGV or using globs.

      #!perl use strict; use warnings; my $dir = 'some/directory'; opendir( my $d, $dir ) or die "can't open dir $dir: $!\n"; open my $o, '>', './output' or die "can't write to file: $!\n"; print {$o} map { open my $in, '<', sprintf '%s/%s', $dir, $_; local $/; <$in> + } grep { -f sprintf '%s/%s', $dir, $_ } readdir $d; close $o or die "can't complete writing to file: $!\n"; closedir $d;

      Such a brief representation gets a little more troublesome if one wants to handle errors rather than dying, or if one wants to warn on errors for each input file.

      If one really wants to make the main program brief and high-level but have plenty of places to add checks and error handling without making the main program itself noisy, there are ways to do that too. There are still no modules necessary.

      #!perl use strict; use warnings; sub read_file ($) { my $file = shift; if ( open my $in, '<', $file ) { local $/; return <$in>; } else { warn "can't read file $file: $!\n"; return (); } } sub files_from_dir ($) { my $dir = shift; opendir( my $d, $dir ) or die "can't open dir $dir: $!\n"; my @list = readdir $d; closedir $d; return map { -f (sprintf '%s/%s', $dir, $_) ? (sprintf '%s/%s', $d +ir, $_) : () } @list; } sub write_to_file ($@) { my $output = shift; open my $o, '>', $output or die "can't write to file : $!\n"; print {$o} @_; close $o or die "can't complete writing to file: $!\n"; } write_to_file './output', map { read_file $_ } files_from_dir '/some/d +irectory';
      Exactly what I was looking for, Thanks Tybalt!
Re: read files one by one in directory and redirct output to a file
by hippo (Bishop) on Jul 06, 2018 at 12:47 UTC

    Given your loose spec, this is pretty trivial for Path::Tiny.

    #!/usr/bin/env perl use strict; use warnings; use Path::Tiny; my $out = path ('output.txt'); for my $file (path ('logs')->children) { $out->append ($file->slurp); }
Re: read files one by one in directory and redirct output to a file (updated)
by haukex (Archbishop) on Jul 06, 2018 at 11:43 UTC

    You've commented out a bunch of code, why? It's unclear what problem you're having, and unclear how which code is relevant to the question. Please take the time to read and follow How do I post a question effectively? and SSCCE - if you follow that advice, we will be able to provide help much more efficiently.

    Just a guess, but perhaps readdir is returning directories as well as just files? And as huck pointed out, the filenames returned by readdir don't include the directory name. Here, I'm using catfile from File::Spec to prefix the directory name to the filename, no_upwards from the same module to filter directory entries like . (curdir) and .. (parent dir), and -f to filter out only files. The sort is optional.

    use File::Spec::Functions qw/no_upwards catfile/; opendir my $dh, $dirname or die "$dirname: $!"; my @files = grep {-f} map {catfile $dirname, $_} sort +no_upwards readdir $dh; closedir $dh;

    Update: Fixed: "sort no_upwards" was not correct, but the -f test was hiding the issue.

      Many apologies for the confusion. Basically what I am trying to do is read the content of all the files in a specific directory and redirect the output to one text file. e.g. 3 small log files.

        Basically what I am trying to do is read the content of all the files in a specific directory and redirect the output to one text file. e.g. 3 small log files.

        See my previous post(s). I see no need to "read the content" of the files because you are not doing any processing of the input. Treat them all as binary files and slam them together as a single output file using either "copy" on Windows or "cp" on Unix. If for some reason that doesn't meet your requirements, then give a short example with a couple of dummy files showing the problem. I don't see any "red flags" for performance issues here. But in general reading files line by line in text mode will be much slower than using an O/S command to append all of the files into a single file using a binary mode copy, even considering the overhead of launching the O/S command.

Re: read files one by one in directory and redirct output to a file
by Marshall (Canon) on Jul 06, 2018 at 06:56 UTC
    I am not sure about what you want.
    copy *.log result
      copy *.log result

      That looks like Microsoft. On Unix-based systems, you would use this:

      cat *.log > result

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
        On Windows, I believe the equivalent to your Unix command would be:
        type *.log > result
        On Unix, I think copy is cp. On either O/S it is possible to concatenate a bunch of binary files together into a single result file.
        copy (cp) works with binary files. type or cat is designed to work with text files.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1218012]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others pondering the Monastery: (3)
As of 2024-04-25 23:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found