Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: Regular expressions across multiple lines

by Anonymous Monk
on Apr 24, 2016 at 17:26 UTC ( [id://1161370]=note: print w/replies, xml ) Need Help??


in reply to Regular expressions across multiple lines

Is your file a Windows file with both \r and \n in it, or a Linux file with just \n?

Try slurping the whole file and tr/// out newlines.

tr/\r\n//d for my $string = do { local $/; <inputfile> };

Replies are listed 'Best First'.
Re^2: Regular expressions across multiple lines
by Marshall (Canon) on Apr 24, 2016 at 17:38 UTC
    chomp() is multi-platform(not exactly true). It will delete <CR><NL> and <NL>, even on Windows. These line endings even if mixed will not matter. BTW: To normalize line endings to the current platform: while(<>){chomp;print;} Works on Unix or Windows.

    Updated: In my testing and actual experience, Perl programs appear to do well under either Unix or Windows, even with mixed line endings on either platform. Perl itself also apparently doesn't have any problems. I have run the above while() code many times on several Unix platforms with great success. On my Win XP machine, there is an issue with old style Mac endings (which uses just <CR>), however I never work with files like that so I hadn't seen this before and had to write a special test case using bin mode to make a file like that.

      chomp() is multi-platform. It will delete <CR><NL> and <NL>, even on Windows. These line endings even if mixed will not matter.

      Well, it may look so, but what really happens is different. See chomp:

      This safer version of chop removes any trailing string that corresponds to the current value of $/ (also known as $INPUT_RECORD_SEPARATOR in the English module).

      Note: Not a single word of the CR or LF control characters, the CR-LF pair, or NL (newline).

      The input record separator $/ is documented, it defaults to an abstract "newline" character:

      The input record separator, newline by default. This influences Perl's idea of what a "line" is. [...] See also Newlines in perlport.

      Now, "newlines". Perl has inherited them from C, by using two modes for accessing files, text mode and binary mode. In text mode, the systems native line ending, whatever that may be, is translated from or to a logical newline, also known as "\n". In binary mode, file content is not modified during read or write. C has been defined in a way that the logical newline is identical with the native line ending on unix, LF. So, there is no difference between text mode and binary mode ON unix.

      Quoting Newlines in perlport:

      In most operating systems, lines in files are terminated by newlines. Just what is used as a newline may vary from OS to OS. Unix traditionally uses \012, one type of DOSish I/O uses \015\012, Mac OS uses \015, and z/OS uses \025.

      Perl uses \n to represent the "logical" newline, where what is logical may depend on the platform in use. In MacPerl, \n always means \015. On EBCDIC platforms, \n could be \025 or \045. In DOSish perls, \n usually means \012, but when accessing a file in "text" mode, perl uses the :crlf layer that translates it to (or from) \015\012, depending on whether you're reading or writing. Unix does the same thing on ttys in canonical mode. \015\012 is commonly referred to as CRLF.

      What happens here is that Perl has reasonable defaults for text handling, so it opens files (including STDIN, STDOUT, STDERR) in text mode by default, $/ defaults to a single logical newline ("\n"), and so native newline characters are translated before chomp just removed that "\n", on any platform.

      When reading text files using a non-native line ending, things will usually go wrong:

      /tmp/demo>file *.txt linux-file.txt: ASCII text mac-file.txt: ASCII text, with CR line terminators windows-file.txt: ASCII text, with CRLF line terminators /tmp/demo>perl -MData::Dumper -E '$Data::Dumper::Useqq=1; for $fn (@AR +GV) { open $f,"<",$fn or die; @lines=<$f>; chomp @lines; say "$fn:"; +say Dumper(\@lines); }' *.txt linux-file.txt: $VAR1 = [ "A simple file generated", "on Linux with Unix", "line endings." ]; mac-file.txt: $VAR1 = [ "A simple file generated\ron Windows with Old Mac\rline endi +ngs.\r" ]; windows-file.txt: $VAR1 = [ "A simple file generated\r", "on Windows with Windows\r", "line endings.\r" ]; /tmp/demo>

      Of course, it depends on the system you are using:

      H:\tmp\demo>perl -MWin32::autoglob -MData::Dumper -E "$Data::Dumper::U +seqq=1; for $fn (@ARGV) { open $f,'<',$fn or die; @lines=<$f>; chomp +@lines; say qq<$fn:>; say Dumper(\@lines); }" *.txt linux-file.txt: $VAR1 = [ "A simple file generated", "on Linux with Unix", "line endings." ]; mac-file.txt: $VAR1 = [ "A simple file generated\ron Windows with Old Mac\rline endi +ngs.\r" ]; windows-file.txt: $VAR1 = [ "A simple file generated", "on Windows with Windows", "line endings." ]; H:\tmp\demo>

      So, chomp is NOT cross-platform. It can handle input from native text files on all platform out of the box. But if you have to work with ASCII files with mixed line endings (CR, LF, CR-LF, LF-CR), chomp can't work reliably. This is not chomp's fault, neither is it perl's fault.

      Alexander

      --
      Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
        This looks mainly right, but with some quibbles.

        1) Correct, the standard text line endings are:
        Unix: <LF> - Line Feed
        Windows: <CR><LF> - Carriage Return, Line Feed
        Network Socket: <CR><LF> - Carriage Return, Line Feed
        Old Mac: <CR> - Carriage Return.

        That a standard network socket (of course even on Unix) uses "Windows" line endings may be news to some.

        2) The chomp description is not 100% clear. When reading in text mode and with the default input record separator of "\n", chomp() will remove any of the line endings that "\n" could mean on any of these 3 platforms between Unix and Windows. The C function getline() will work similarly. Reading a Windows file on Unix will work fine with this text oriented C read function.

        3) Some ancient Unix functions like lp (line print) will not work with Windows line endings. Perl is fine, but lp not. In that case: while(<>){chomp;print;} will set things right. I have used this many times on Unix to convert a mixed file to <LF> endings and vice versa on Windows to convert to <CR><LF>. Although my Windows programs just don't seem to care.

        4) I don't know how these test cases were generated. There is no way to do that without being in Perl bin mode or writing a C program.

        Update: well it appears that Perl doesn't like old Mac endings on my Windows XP machine. This does work with the <LF> ending. So something like "works between Unix and Windows" may be closer to the truth (dual platform) rather than "multi-platform". On Unix, Perl has to be able to read from both hard disk files and network sockets which have different line endings.

        #!/usr/bin/perl use warnings; use strict; open OUT, '>', "unixending.txt" or die "$!"; binmode OUT; print OUT pack "C8", 0x41,0x42,0x43,0x0A,0x44,0x45,0x46,0x0D; close OUT; open IN, '<', "unixending.txt" or die "$!"; while (<IN>) { chomp; print "\"$_\"\n"; } __END__ "ABC" ## fine for Unix <LF> 0x0A "DEF ## didn't work for old MAC <CR> 0x0D "
        ++afoken you have the karma of exhaustiveness! i must bookmark as newline gory details

        L*

        There are no rules, there are no thumbs..
        Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

      Reading a 100mb file a line at a time can be slow.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1161370]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others avoiding work at the Monastery: (5)
As of 2024-03-29 07:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found