http://qs321.pair.com?node_id=11108409


in reply to Fastest split possible

Update: Added regex example.

Hello all :) I tried trimming the left side of the string. Plus incorporated the memfh example by davido. I'm curious too.

Split improves ~ 2x faster for this demonstration with Perl 5.22.x and later releases. That is the case on macOS.

use strict;
use warnings;
use Time::HiRes 'time';

my $huge_string = "aaa bbb\nccc ddd\neee fff\nggg hhh\niii jjj\nkkk ll
+l\nmmm nnn\n";

# concatenate string exponentially to 917,504 lines
$huge_string .= $huge_string for 1..17;

# memfh
{
    my $string = $huge_string;
    my $start  = time;

    open my $memfh, '<', \$string;
    my @lines = <$memfh>;
    close $memfh;

    printf "duration memfh: %0.3f seconds\n", time - $start;
    printf "%d lines\n\n", scalar(@lines);
}

# regex
{
    my $string = $huge_string;
    my $start  = time;
    my @lines;

    while ( $string =~ /([^\n]+\n)/mg ) {
        my $line = $1; # save $1 to not lose the value
        push @lines, $line;
    }

    printf "duration regex: %0.3f seconds\n", time - $start;
    printf "%d lines\n\n", scalar(@lines);
}

# split
{
    my $string = $huge_string;
    my $start  = time;
    my @lines  = split(/\n/, $string);

    printf "duration split: %0.3f seconds\n", time - $start;
    printf "%d lines\n\n", scalar(@lines);
}

# trim
{
    my $string = $huge_string;
    my $start  = time;
    my @lines;

    while ( my $line = substr($string, 0, index($string, "\n") + 1, ''
+) ) {
        push @lines, $line;
    }

    printf "duration trim : %0.3f seconds\n", time - $start;
    printf "%d lines\n\n", scalar(@lines);
}

[download]

Output - Perl 5.28.2

duration memfh: 0.384 seconds
917504 lines

duration regex: 0.387 seconds
917504 lines

duration split: 0.067 seconds
917504 lines

duration trim : 0.201 seconds
917504 lines

[download]

Another machine - Perl 5.26.1

duration memfh: 0.477 seconds
917504 lines

duration regex: 0.445 seconds
917504 lines

duration split: 0.065 seconds
917504 lines

duration trim : 0.259 seconds
917504 lines

[download]

Same machine - Perl 5.18.2

duration memfh: 0.530 seconds
917504 lines

duration regex: 0.490 seconds
917504 lines

duration split: 0.130 seconds
917504 lines

duration trim : 0.261 seconds
917504 lines

[download]

Regards, Mario

Replies are listed 'Best First'.
Re^2: Fastest split possible
by marioroy (Prior) on Nov 08, 2019 at 03:22 UTC

    Update 1: Added walk2 by pushing to stack.
    Update 2: The string length now obtained inside C.
    Update 3: Changed from for loop to while loop.

    Ah, I tried Inline::C. Walk1 runs faster than split in older Perl <= 5.20.x.

    ...

use Inline 'C' => <<'END_C';

  #include <stdlib.h>
  #include <string.h>

  SV * c_walk1(SV *sv)
  {
      STRLEN len;
      const char *s = SvPV_const(sv, len);
      const char *m = s, *strend = s + len;

      AV *ret = newAV();

      while (s < strend) {
          if (*s++ == '\n') {
              av_push(ret, newSVpvn(m, s - m));
              m = s;
          }
      }

      return newRV_noinc((SV *) ret);
  }

  void c_walk2(SV *sv)
  {
      STRLEN len;
      const char *s = SvPV_const(sv, len);
      const char *m = s, *strend = s + len;

      Inline_Stack_Vars;
      Inline_Stack_Reset;

      while (s < strend) {
          if (*s++ == '\n') {
              Inline_Stack_Push(sv_2mortal(newSVpvn(m, s - m)));
              m = s;
          }
      }

      Inline_Stack_Done;
  }

END_C

# walk1 - receive array ref
{
    my $string = $huge_string;
    my $start  = time;
    my $lines  = c_walk1($string);

    printf "duration walk1: %0.3f seconds\n", time - $start;
    printf "%d lines\n\n", scalar(@$lines);
}

# walk2 - receive list
{
    my $string = $huge_string;
    my $start  = time;
    my @lines  = c_walk2($string);

    printf "duration walk2: %0.3f seconds\n", time - $start;
    printf "%d lines\n\n", scalar(@lines);
}

    [download]

    Output from Perl 5.20.3 and Perl 5.26.1

    $ /opt/perl-5.20.3/bin/perl demo.pl

duration memfh: 0.526 seconds
917504 lines

duration regex: 0.448 seconds
917504 lines

duration split: 0.110 seconds
917504 lines

duration trim : 0.268 seconds
917504 lines

duration walk1: 0.073 seconds
917504 lines

duration walk2: 0.110 seconds
917504 lines

    [download]
    $ /opt/perl-5.26.1/bin/perl demo.pl

duration memfh: 0.479 seconds
917504 lines

duration regex: 0.442 seconds
917504 lines

duration split: 0.066 seconds
917504 lines

duration trim : 0.259 seconds
917504 lines

duration walk1: 0.070 seconds
917504 lines

duration walk2: 0.084 seconds
917504 lines

    [download]

    Regards, Mario

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom