I don't think there's a good answer to this other than "Do it in C" but I thought I'd ask anyway. I'm reading in several hundred 20k line standard data text files, with space delimited numbers like so, though with up to 8 data columns:

# time             data1     data2
0.000000          99.537     54.54
1.000000         100.273    121.54
2.000000          98.169    121.58
3.000000         105.835     99.66
4.000000          93.013      1.85
[download]

The time spent spliting the lines is about 10% of my program run time, so I was wondering if there was an easy way to speed it up. Just reading the files from disk seems to be about half my runtime, so no dramatic improvements possible. But look at the code, maybe a restructuring would be faster? Here's what I'm doing now:

# Each data point is stored in a separate file, so they have
# to be joined first.  I figure the shell is faster at that
# than perl.  That and I wasn't excited about managing 
# n file handles at a time, though it wouldn't be too bad.
my @lines = `join file1 file2 file3 file4`;
foreach my $line (@lines) {
  my ($time, @data) = split /\s+/, $line;
  foreach my $datum (@data) {
    # unlock the secrets of the universe
  }
}
[download]

---

---

• Are you posting in the right place? Check out Where do I post X? to know for sure.
• Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:

  <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>

• Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
• Want more info? How to link or How to display code and escape characters are good places to start.