Re: Performing a tail(1) in Perl (reading the last N lines of a file)

I'd seen Tie::File touted recently, so I decided to see how it stood up. I added the following subroutine:

sub tie_file {
  my @lines;
  tie @lines, 'Tie::File', $file or die "$!";
  map {"$_\n"} @lines[-$limit..-1];
}
[download]

Pretty much as simple as it gets. Unfortunatly, it's not quite so hot on the performance:

[chmrr@supox chmrr]$ wc -l /var/log/messages
   4442 /var/log/messages
[chmrr@supox chmrr]$ perl -w tailbm.pl -10 100 /var/log/messages
Benchmark: running f_rb_obj, f_rb_obj_u, f_rb_tie, f_rb_tie_u, file_ta
+il, grinder, lastn, lastn_getc, tie_file, each for at least 10 CPU se
+conds...
  f_rb_obj: 11 wallclock secs (10.35 usr +  0.22 sys = 10.57 CPU) @ 66
+2.54/s (n=7003)
f_rb_obj_u: 11 wallclock secs (10.54 usr +  0.12 sys = 10.66 CPU) @ 65
+0.19/s (n=6931)
  f_rb_tie: 11 wallclock secs (10.48 usr +  0.20 sys = 10.68 CPU) @ 54
+5.13/s (n=5822)
f_rb_tie_u: 12 wallclock secs (10.39 usr +  0.16 sys = 10.55 CPU) @ 53
+7.25/s (n=5668)
 file_tail: 12 wallclock secs (10.60 usr +  0.14 sys = 10.74 CPU) @ 33
+3.71/s (n=3584)
   grinder: 10 wallclock secs (10.49 usr +  0.07 sys = 10.56 CPU) @ 26
+.23/s (n=277)
     lastn: 12 wallclock secs ( 7.79 usr +  2.63 sys = 10.42 CPU) @ 26
+.97/s (n=281)
lastn_getc: 11 wallclock secs ( 8.83 usr +  1.46 sys = 10.29 CPU) @ 28
+.38/s (n=292)
  tie_file: 11 wallclock secs (10.15 usr +  0.12 sys = 10.27 CPU) @  3
+.89/s (n=40)
[download]

Well, it's good to know, at least.

Update: Heh -- I started doing this test a'fore broquaint posted; great minds think alike, eh?

Update 2: On a whim, I decided to test:

sub backticks { split /$/m, `tail -$limit $file`; }

..as well. Even given the overhead of the shell, it's still darn fast:

[chmrr@supox chmrr]$ perl -w tailbm.pl 500 100 /var/log/messages
Benchmark: timing 500 iterations of backticks, f_rb_obj, f_rb_obj_u, f
+_rb_tie, f_rb_tie_u, file_tail, grinder, lastn, lastn_getc, tie_file.
+..
 backticks:  1 wallclock secs ( 0.43 usr  0.16 sys +  0.61 cusr  0.30 
+csys =  1.50 CPU) @ 847.46/s (n=500)
  f_rb_obj:  1 wallclock secs ( 0.71 usr +  0.04 sys =  0.75 CPU) @ 66
+6.67/s (n=500)
f_rb_obj_u:  1 wallclock secs ( 0.77 usr +  0.00 sys =  0.77 CPU) @ 64
+9.35/s (n=500)
  f_rb_tie:  1 wallclock secs ( 0.89 usr +  0.02 sys =  0.91 CPU) @ 54
+9.45/s (n=500)
f_rb_tie_u:  1 wallclock secs ( 0.91 usr +  0.02 sys =  0.93 CPU) @ 53
+7.63/s (n=500)
 file_tail:  1 wallclock secs ( 1.47 usr +  0.01 sys =  1.48 CPU) @ 33
+7.84/s (n=500)
   grinder: 19 wallclock secs (18.93 usr +  0.22 sys = 19.15 CPU) @ 26
+.11/s (n=500)
     lastn: 19 wallclock secs (14.20 usr +  4.34 sys = 18.54 CPU) @ 26
+.97/s (n=500)
lastn_getc: 20 wallclock secs (15.09 usr +  2.64 sys = 17.73 CPU) @ 28
+.20/s (n=500)
  tie_file: 143 wallclock secs (126.62 usr +  1.81 sys = 128.43 CPU) @
+  3.89/s (n=500)
[download]

perl -pe '"I lo*`+$^X$\"$]!$/"=~m%(.*)%s;$_=$1;y^`+*^e v^#$&V"+@( NO CARRIER'

Comment on Re: Performing a tail(1) in Perl (reading the last N lines of a file) Select or Download Code

Replies are listed 'Best First'.
Re: Performing a tail(1) in Perl (reading the last N lines of a file) by Dominus (Parson) on Apr 26, 2002 at 03:42 UTC
Says Chmrr: Tie::File ... Pretty much as simple as it gets. Unfortunatly, it's not quite so hot on the performance: Not too surprising, because unlike most of the methods you benchmarked, `Tie::File` does not read the file backwards. It reads it forwards, starting at the beginning, and constructs some data structures along the way. For a long file like `/var/adm/messages`, this will take some time, because it grovels over the entire file. The payoff would come if you then asked it to tell you what was on line 12,345, which it would do instantly---or if you asked it to modify line 12,345, which the other modules won't do at all. -- Mark Dominus Perl Paraphernalia	[reply]
Re: Re: Performing a tail(1) in Perl (reading the last N lines of a file) by Anonymous Monk on Oct 23, 2002 at 20:06 UTC
Have you considered modifying File::Tail to defer reading the file such that if the only subscripts ever given to the tied array are negative, it reads the file backwards? Not asking for the feature; just throwing out ideas.	[reply]


The stupid question is the question not asked
	PerlMonks