Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^9: Speeds vs functionality (utf8 csv)

by tye (Sage)
on Jul 31, 2014 at 21:12 UTC ( #1095819=note: print w/replies, xml ) Need Help??


in reply to Re^8: Speeds vs functionality
in thread Speeds vs functionality

I was never considering single-byte anything. Writing code in Perl means that I don't have to (unlike writing code in XS). Yes, I actually meant what I said. Yes, I realized that your example was using multi-byte single-character tokens.

The reason that single-character vs. multi-character (usually) leads to different approaches is because [^"\\]+ as part of a regex works fine for those single-character quote and escape values (respectively) but isn't even close to what you have to do if either of those is multi-character.

And you are quite wrong about:

One glance at the source code and it's obvious the author doesn't mean single character; he means single byte.

For one, the author of Text::xSV didn't have to think about multi-byte characters. Their module is written in Perl so, unless they do something moderately strange or stupid, then multi-byte characters "just work" (provided the user of the module does the little bit of extra work to ensure that Perl has/will properly decode the strings/streams being given to the module).

Looking at the code for Text::xSV in some detail, I see that 90% of the uses of the separator character would work completely fine with a separator that is even composed of more than one multi-byte character. There is one important place where the code would break for a multi-character separator (but that, indeed, continues to work for a separator that is a single multi-byte character):

my $start_field_ms = qr/\G([^"$q_sep]*)/;

Now, fixing the unfortunate hard-coding of the quote character is probably quite a simple task. And that would probably be sufficient to make the module work fine on multi-byte quote characters. Certainly much easier than trying to get multi-byte character support into a much more complex XS module.

Why not?

Because you haven't done the tiny bit of work to fix Text::xSV? Or the small amount of work to write a simple CSV parser in Perl?

No matter. I'm almost done writing my new CSV module.

- tye        

Replies are listed 'Best First'.
Re^10: Speeds vs functionality (utf8 csv)
by ikegami (Pope) on Aug 01, 2014 at 03:17 UTC

    For one, the author of Text::xSV didn't have to think about multi-byte characters.

    Technically, true, but he did have to think about proving a means of providing decoded input. I don't see any.

    As a result, the separator can only be in U+0000..U+007F for UTF-8 files (assuming the claim that it only supports one-character seperator is correct), and it can't handle UTF-16le files with character U+0Axx, etc.

      Yeah, fixing the module to allow a file handle to be given instead of just a file name is quite in line with the trivial work that I noted might be required.

      Though, I suspect that Perl provides a way for declaring a default encoding for all file handles, perhaps related to "locale" settings. So I'm not even convinced that your objection is even technically correct. (Though, if Perl does not provide such a feature, perhaps you should look into providing one, IMHO. :)

      I'm actually a bit surprised that open does not already support (according to my recent scanning of the documentation):

      open my $fh, '<:encoding(UTF-8) foo.csv'

      which would have also been a route that would have worked with the unchanged Text::xSV.

      but he did have to think about proving a means of providing decoded input

      No, the author didn't have to think about that. The author just needed to allow a file handle to be given, even if the reason for allowing such had nothing to do with the author thinking about decoded input. I very often support taking a filehandle not just a filename, and very rarely is that due to me having thought about encodings.

      - tye        

        Though, I suspect that Perl provides a way for declaring a default encoding for all file handles,

        There's use open, but it's lexically scoped.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1095819]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (7)
As of 2020-10-29 11:00 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My favourite web site is:












    Results (270 votes). Check out past polls.

    Notices?