Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: split and sysread()

by pfaut (Priest)
on Apr 18, 2003 at 23:48 UTC ( #251582=note: print w/replies, xml ) Need Help??


in reply to split and sysread()

Are your records terminated with newlines or are they fixed length? Your call to sysread looks very wrong. If the records are terminated by newlines, use <INFILE> to read a record. If they are fixed length, the third argument to sysread should be the record length and you shouldn't use the fourth argument.

This might make the first part work. Extending it to the second part is up to you.

while ($buffer = <INFILE>) { chomp $buffer; my ($server,$ip,$api,$calls) = (split('|',$buffer))[0,9,3,4]; $totalsentrycalls++; $count_by_sentry_server{$server}++; $count_by_ip{$ip}++; $count_by_api{$api}++; $count_by_api_exec{$api}{$totalsentrycalls}=$calls; .... }
90% of every Perl application is already written.
dragonchild

Replies are listed 'Best First'.
Re: Re: split and sysread()
by relaxed137 (Acolyte) on Apr 18, 2003 at 23:58 UTC
    They are terminated by new lines. The issue with the above is it still reads in the file one line at a time, right?. That's what I am trying to get away from.
      The issue with the above is it still reads in the file one line at a time, right?.

      Wrong! Perl reads from the file in a buffer and only goes back out to disk when the buffer is empty. What <FILEHANDLE> does is read from the buffer up to the newline. I think you are falling prey to premature optimization.

      while (<INPUT>) { my @fields = split /\|/ , $_; #Do stuff with particular field }

      Cheers - L~R

      Update: See this node by chromatic as he was setting me straight on the very same matter

        Hmmm... That's what I had before I started to use sysread... I've noticed that sysread will read a file much faster though... Maybe I can make the read buffer bigger instead to tune "while <input>" ?
        open (INFILE, "/sentry/ecisock/ecisock.rpt"); while (<INFILE>) { chomp; ($serverhostname, $execdate, $cicstranid, $sentryapi, $elapsedtime, $errorcode, $corphdr, $channelcode, $clientprodcode, $clientipaddr, $sentrytermid, $signonid, $accountnum, $branch, $campaign, $segment, $custid, $localsysid, $remotesysid, $sentrytarget, $hfb, $sentryver, $corphdrfailreason, $corphdrfailtype, $corphdrbussvc, $corphdrclientcode, $mfcicstasknum, $sentrycicstasknum, $foo1, $foo2, $cpsm)=split(/\|/,$_); ########################################### # Counts total calls ########################################### $totalsentrycalls++; (.. Etcetera ..)
      The issue with the above is it still reads in the file one line at a time, right?. That's what I am trying to get away from.

      But in order to pick out the right fields, you have to know where a line starts and stops. There's no getting around that.

      You can either use sysread() to pull stuff in by blocks, then juggle blocks to handle lines that span blocks, or you can uses Perl's line-by-line IO. Your call.

      Well, what you had would read the next 16K of data from the file and append it to $buffer each time through the loop. Was your intent to process the file 16K at a time? In that case, you would still have to remove the fourth argument to sysread as it was causing new data to be written to the end of $buffer.

      Perl will allow you to read line-at-a-time or to slurp all of the file into an array with each element containing one line from the file. If you use sysread, breaking the data into lines is up to you. You would first have to break the data into lines by splitting on '\n' and then split the record into fields by splitting on '|'. It would be highly likely that your 16K read won't end at a record boundary so you would have to add code to merge the remaining data from one read with the beginning of the next. I'm not sure whatever you came up with would be more efficient than perl's line mode buffering.

      90% of every Perl application is already written.
      dragonchild

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://251582]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (4)
As of 2021-04-10 22:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?