Tune the Sentiment Model
After completing this unit, you’ll be able to:
- Describe sentiment tuning.
- Access your sentiment model.
- Tune your sentiment model.
Sentiment Model in Social Studio
Have you ever wondered how the public feels about your brand? Sentiment analysis, the process of categorizing words in posts, can help you understand public sentiment. Social Studio has a scoring algorithm that processes posts to determine the overall sentiment. Sentiment can be tricky, however. Brand-specific considerations can make the default dictionary words and weights inaccurate. This is where sentiment tuning can help. Sentiment tuning is the process of adjusting the default dictionary in Social Studio to better fit your brand. Social Studio comes with an adjustable sentiment model to help you more accurately monitor and report on post sentiment.
What Is Sentiment Analysis?
Before learning how to tune your sentiment model, it is helpful to first understand how sentiment analysis works at a high level.
- As content comes in from the social web, Social Studio collects a tremendous amount of data, analyzing 3,000+ posts per second!
- The content is preprocessed through spam filters, language detection, and normalization.
- The sentiment engine evaluates the content and assigns a sentiment value.
So, How Does Sentiment Analysis Work?
“The party was really great... until the electricity went out. :( No music, but we enjoyed the rest of the night by candlelight.”
No and the frowning emoji :( are assigned negative values, while the words really and great are assigned positive values. The sentiment engine calculates all the scores to determine an overall positive score and therefore a positive sentiment.
Each word is assigned a weight, depending upon the part of speech or type of word. The score then determines how positive, neutral, or negative the sentiment is considered.
Let’s take a closer look at the evaluation process.
- Tokenizer—A white space tokenizer takes the first look at the content and breaks it down, creating a token each time it encounters a white space.
- Part of speech—Next, the sentence is dissected to identify how each word fits grammatically. This identification determines if words are sentiment bearing.
- Dictionary—Social Studio uses its default sentiment dictionary that includes a predefined list of words. Those words are assigned weights, ranging from -1 to +1. As Social Studio ingests a sentence, it looks for matching words, analyzes the average sentiment for those matches, and assigns a sentiment classification: positive, neutral, or negative.
- Intensifiers and negators—Social Studio also considers intensifiers and negators. Intensifiers, such as the word really, can weight a score even higher or lower. Negators, such as not, change whether a word is considered positive or negative. For example, “That meal was really not bad at all.” includes both an intensifier and negator that positively affects the sentiment weight of the word bad.
Sentiment Analysis Isn’t Always Correct
Consider this example. The word sick is scored as negative because it most often refers to poor health. However, there are examples of sick used as a slang term to positively describe an outfit or experience (Einstein’s sneakers were sick!). Another example is the word cancer, which is often scored negatively. But in context of a medical breakthrough, cancer is more appropriately scored as neutral or even positive.
Depending upon the context of your social posts, it’s important to review how the sentiment model is weighing keywords like these. Over time you can adjust this model to provide more accurate sentiment based on incoming posts.
Access and Tune Your Sentiment Model
To access and tune your sentiment model, you need to be a Social Studio super user or admin. If you’re not an administrator, that’s OK just know that you won’t see this page in your Social Studio application. Under the Sentiment Model section of Admin, you can view all feedback, starred, or non-starred posts.
Feedback items are created when a Social Studio user evaluates a post in Engage and Analyze and manually adjusts the sentiment. In the sentiment model, the list of feedback items shows the original sentiment weight, the post, and highlighted words indicating the phrases that are currently used in the sentiment model dictionary. You can also view the submitter’s feedback notes for greater insight into adjusting the sentiment model.
As an admin, you can do a few things to tune your sentiment model.
(1) Star posts to use as the gold standard in the sentiment model.
As you review feedback items, look for trends—posts that you see often and are important to your business. If a feedback item meets both of those criteria, give it a gold star to add it to the sample set you use to customize your sentiment model. Starred posts are representative of the type of content relevant to your business and are important to score correctly. Accumulate at least 200 gold-starred items for a language before tuning the sentiment.
(2) Review highlighted words.
Notice the highlighted words in the list of feedback items. Your sentiment model dictionary currently includes these words. Use this information to identify words to add or correct in your sentiment model.
(3) Manage the model.
When you’re comfortable with your gold starred posts, click Manage Model to see the current model with the percentage of starred posts in agreement. Your model includes precision and recall scores that are calculated for positive, neutral, and negative sentiments. You’re generally looking for an overall agreement with starred posts of around 79%.
Making the Most of Sentiment
As you’re working with your sentiment model, consider these tips for more accuracy.
Sentiment Is Difficult to Measure
As we all know, humans don’t agree 100% of the time. If everyone taking this module, for example, reviewed 100 tweets and scored them for sentiment, we would only agree around 79% of the time. Keep that in mind when you’re scoring sentiment—don’t expect 100%; it doesn’t exist! Also keep in mind that Social Studio is analyzing sarcasm, irony, and multiple meanings—especially difficult with social media content.
Sentiment Is Only One Measure of Your Brand
Think about sentiment as one of many measures. You need multiple metrics to present the whole picture. Sentiment is a good metric but it’s not the only metric. For example, when you measure your physical fitness, your body weight is one of many factors to consider.
More Data Leads to Greater Accuracy
The larger the quantity of data that you’re analyzing, the more confident you can feel that you’re picking up the trend. Be patient with this process and only adjust the model about every 6–12 months. If you adjust the sentiment model too often, it can cause your sentiment model to wander and affect the overall scoring of sentiment.
You’re All Set!
You learned what Social Studio is and how to navigate around it. You understand how to manage workspaces and users. You learned about topic profiles and got familiar with the sentiment model. Give yourself a pat on the back!