Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?

Regex For Removing Emoji

by Beaker (Beadle)
on Nov 12, 2016 at 13:12 UTC ( #1175786=perlquestion: print w/replies, xml ) Need Help??

Beaker has asked for the wisdom of the Perl Monks concerning the following question:

Someone has just started throwing emoji into text in my site, For the most part I don't mind but I want to strip it from article and thread titles. Either to change them to plain text completely, or use a regex to strip the emoji specifically. After looking into this and trying out a bunch of regex I found on stack overflow, it seems there's many types of emoji and I couldn't get any of these to work.
$text =~ s/[\x00-\x1F\x80-\xFF]+\s*//g; $text =~ s/\xEE[\x80-\xBF][\x80-\xBF]|\xEF[\x81-\x83][\x80-\xBF]+\s*// +g; $text =~ s/([0-9#][\x{20E3}])|[\x{00ae}\x{00a9}\x{203C}\x{2047}\x{2048 +}\x{2049}\x{3030}\x{303D}\x{2139}\x{2122}\x{3297}\x{3299}][\x{FE00}-\ +x{FEFF}]?|[\x{2190}-\x{21FF}][\x{FE00}-\x{FEFF}]?|[\x{2300}-\x{23FF}] +[\x{FE00}-\x{FEFF}]?|[\x{2460}-\x{24FF}][\x{FE00}-\x{FEFF}]?|[\x{25A0 +}-\x{25FF}][\x{FE00}-\x{FEFF}]?|[\x{2600}-\x{27BF}][\x{FE00}-\x{FEFF} +]?|[\x{2900}-\x{297F}][\x{FE00}-\x{FEFF}]?|[\x{2B00}-\x{2BF0}][\x{FE0 +0}-\x{FEFF}]?|[\x{1F000}-\x{1F6FF}][\x{FE00}-\x{FEFF}]|[\x{D83E}][\x{ +DD13}-\x{DD14}]?+\s*//g; $text =~ s/([0-9|#][\x{20E3}])|[\x{00ae}\x{00a9}\x{203C}\x{2047}\x{204 +8}\x{2049}\x{3030}\x{303D}\x{2139}\x{2122}\x{3297}\x{3299}][\x{FE00}- +\x{FEFF}]?|[\x{2190}-\x{21FF}][\x{FE00}-\x{FEFF}]?|[\x{2300}-\x{23FF} +][\x{FE00}-\x{FEFF}]?|[\x{2460}-\x{24FF}][\x{FE00}-\x{FEFF}]?|[\x{25A +0}-\x{25FF}][\x{FE00}-\x{FEFF}]?|[\x{2600}-\x{27BF}][\x{FE00}-\x{FEFF +}]?|[\x{2900}-\x{297F}][\x{FE00}-\x{FEFF}]?|[\x{2B00}-\x{2BF0}][\x{FE +00}-\x{FEFF}]?|[\x{1F000}-\x{1F6FF}][\x{FE00}-\x{FEFF}]?+\s*//g

Replies are listed 'Best First'.
Re: Regex For Removing Emoji
by Corion (Patriarch) on Nov 12, 2016 at 14:12 UTC

    Also see Text::Unidecode and especially for sanitizing titles for URLs, Text::CleanFragment.

    Both err rather on the side of leaving things out rather than keeping things in.

    It seems your regular expressions attempt to remove whole Unicode character planes. Personally, I would explicitly allow some character planes or look at the unicode properties (maybe via Unicode::Tussle to find out whether a character is part of a script.

    Also consider what you want to do with character art: (╯°□°)╯︵ ┻━┻

      Thanks I will check out those modules you mentioned. I do a lot of manual text sanitization and manipulation so I should probably try and "third party" some of it.
Re: Regex For Removing Emoji
by LanX (Sage) on Nov 12, 2016 at 13:46 UTC
    > found on stack overflow

    Where this?

    Did you ask there? Does it mean you are cross posting now?

    >  and I couldn't get any of these to work.

    what does this mean?

    Could you provide sample input to test?

    Personally I would restrict with a positive filter (white listing)

    what does it help to block emojis and allow arbitrary Unicode to appear?

    Cheers Rolf
    (addicted to the Perl Programming Language and ☆☆☆☆ :)
    Je suis Charlie!

      Sample input 🦂 🍷✨⛓ 🔪🐷 ⏳🎣 Shows full color icons in firefox and chrome, white only in IE and some of the icons in Safari, also white only.
        Better dump the $text your application handles, before and after applying the regexes.

        I suspect the data you are processing doesn't look like it is supposed to do. (Probably Html escaped)

        Anyway, in general providing us with a complete test script showing input and output would help you and us.

        Often errors are spotted by the authors when preparing a post like this.

        Cheers Rolf
        (addicted to the Perl Programming Language and ☆☆☆☆ :)
        Je suis Charlie!

        Fwiw, for those interested the monastery transforms the provided emojis 🍷✨⛓ 🔪🐷 ⏳🎣 into 🍷✨⛓ 🔪🐷 ⏳🎣  within code blocks

      No I didn't post on stack overflow, I just found similar questions posted and regex provided by response (in other languages).

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1175786]
Approved by marto
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (5)
As of 2022-10-04 10:18 GMT
Find Nodes?
    Voting Booth?
    My preferred way to holiday/vacation is:

    Results (16 votes). Check out past polls.