Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

utf8 is_valid on concat

by threedaygoaty (Novice)
on Mar 13, 2008 at 06:05 UTC ( #673910=perlquestion: print w/replies, xml ) Need Help??

threedaygoaty has asked for the wisdom of the Perl Monks concerning the following question:

Hi folks, we have suffered greatly at the hands of utf8 converting a very large application to ut8dom. It seems that we are concatenating UTF8 and non-UTF8 strings and ending up in trouble. So why not globally overload the concat operator with something that checks for is_valid before doing the concat? Just for debugging? Can it be done in Perl?
Cheers,
TDG

Replies are listed 'Best First'.
Re: utf8 is_valid on concat
by Juerd (Abbot) on Mar 13, 2008 at 11:57 UTC

    Are you perhaps doing any of the following?

    • Using the :utf8 layer for input (open, binmode)
    • Using _utf8_on anywhere
    • Forgetting to decode input
    • Forgetting to encode output
    Please describe in more detail what "trouble" is, and how you get there.

    Juerd # { site => 'juerd.nl', do_not_use => 'spamtrap', perl6_server => 'feather' }

      Thanks for this, AFAIK we are using utf8 functions correctly. However, it is very likely that somewhere in 200K lines of perl a simple string concat is happening between a UTF8 string and a non-UTF8 string. My idea is to try to trap this so we can find it. The only way I can think of doing is to overload the concat operator. I'm also keen to know if it is possible. We tried this but it does not work...
      use strict; package UNIVERSAL; use overload "." => \&concat; sub concat { my ($a, $b) = @_; return join "XXX", $a, $b; } package Foo; my $thing = "cat" . "dog"; print "\n\n\n$thing\n\n\n";
      This produces catdog
Re: utf8 is_valid on concat
by haoess (Curate) on Mar 13, 2008 at 09:14 UTC

    Could you please provide a piece of code that shows your described behaviour (the "trouble"), along with your perl version?

    Update: And what is ut8dom?

    -- Frank

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://673910]
Approved by Thilosophy
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (4)
As of 2021-11-29 13:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?