Emoji with multiple code units not detected

First: apologies if I provide insufficient information or use wrong terminology. This is my first GitHub issue ever, so please be kind

Demo code works fine for me, including emojis. However, the demo emoji are described by a single code unit. Emojis with more than one, e.g. “red heart” (2764 FE0F) are not detected, despite being in the lexicon.

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
sentences = ["Catch utf-8 emoji such as such as 💘 and 💋 and 😁",  # emojis handled
             "Not bad at all",  # Capitalized negation
             "Me and Fay are 4 years old today ❤️ (ft Grumio)…"
             ]
analyzer = SentimentIntensityAnalyzer()
for sentence in sentences:
    vs = analyzer.polarity_scores(sentence)
    print("{:-<65} {}".format(sentence, str(vs)))`

returns

Catch utf-8 emoji such as such as 💘 and 💋 and 😁------------------ {‘neg’: 0.0, ‘neu’: 0.615, ‘pos’: 0.385, ‘compound’: 0.875} Not bad at all--------------------------------------------------- {‘neg’: 0.0, ‘neu’: 0.513, ‘pos’: 0.487, ‘compound’: 0.431} Me and Fay are 4 years old today ❤️ (ft Grumio)… {‘neg’: 0.0, ‘neu’: 1.0, ‘pos’: 0.0, ‘compound’: 0.0}

[foost]

Thank you for elucidating what is happening here with emoji interpretation. We are running tests and surprised that :red heart: etc. returned neutral scores consistently. I will try your fix for now, but agree that it is merely a substitute until the logic is more substantially addressed.

Thanks for getting back about this. After a hiatus in research, I plan to get back to sentiment analysis soon, so I’ll watch this space and see whether there is anything I can contribute.

[foost]

Found the problem, not sure about a fix. I understand now that the sentiment scores are derived from the emoji description words, like normal text, in the sentiment_valence function. So, in case of ☹️ (“frowning face”) the word found in the lexicon is “frowning”, and in case of ❤️ (“red heart”) it’s “heart”. I tested it extensively, the former works (so the problem does not occur with all emojis with multiple code points), the latter does not. The problem only occurs when the lexicon word is the last word of the emoji description, because the loop (see my previous comment) only looks at the first code point to find the description, but then adds the second code point to the last word of that description. This changes the last character of that unigram, making the sentiment_valence look-up miss it. The change is barely visible in control print outs (e.g. the letter “t” becomes a tiny bit smaller), that’s why it took me so long to figure out what’s going on. How to fix it? Dealing properly with emojis with multiple code points would need some serious changes in the loop. I have chosen for a quick and dirty fix: Since the major culprit for sentiment-relevant emojis is “FE0F”, I changed the “else:” statement into “elif ord(chr) != 65039:” to completely ignore it. Seems to work.

            if chr in self.emojis:
                # get the textual description
                description = self.emojis[chr]
                if not prev_space:
                    text_no_emoji += ' '
                text_no_emoji += description
                prev_space = False
            elif ord(chr) != 65039:
                text_no_emoji += chr
                prev_space = chr == ' '```