How does it work?

When you connect your Social Account to us, you allow us access to parts of your social graph, including the ability to analyze your comments and posts. This analysis uses machine learning and a technique called ‘Sentiment Analysis’ to uncover the overall sense of your post or comment.

The more accurate data into the analysis engine, the more likely we are to get good results out. We are initially linking to your Facebook comments and posts, but have designs on analyzing other networks that you consent to share with us (Twitter, Google+, LinkedIn, etc.).

When a text block is submitted for analysis, we receive a polarity rating of 0, 2 or 4, where 0 implies negative sentiment, 2 is neutral, and 4 implies positive sentiment. For most of our analysis, we examine upwards of 50 posts or tweets per subject, and this normally gives us a good baseline to draw inferences. To counteract the fact that a lot of text is neutral, we disregard 90% of neutral statements.

Once we have sufficient data to analyze, we assign it a certain weighting (that is, how much it contributes to the overall score), convert it to a percentage from 0 to 100%. On this scale, 0% is back-the-entire-coal-truck up naughty and 100% is Santa Claus-himself niceness.

Can you tell me more about Santa’s List of Celebrities?

While the Santa-lzyer was designed in large part for individuals to assess how naughty they have been (and thus influence the Santa in their life), we thought it would be enlightening to see how those in the public eye perform to act as comparison points to the rest of us.

Where things differ slightly for the celebrities analyzed, is that we were using our beta Twitter version to analyze their public statements vs. Facebook post data for everyone else. We have been analyzing Tweets since approximately October 15th and the results are a snapshot in time. For our analysis of the US Presidential Candidates, we did even greater analysis which is described below.

What was interesting to us is that global sentiment increased and people became much nicer in the buildup and throughout the devastation of Hurricane Sandy, where those being analyzed sent out well wishes and positive thoughts to those affected.

Why were these celebrities chosen?

There was not much rhyme or reason for most of these people being chosen, with the obvious exception being the Presidential Candidates. We thought that the people selected provided a nice baseline for comparison.

How did Christmas.com Analyze the Naughty or Nice Factor for Brands?

We initially intended to use our Santa-lyzer technology to assess individuals, but a few sample tests found us querying several of the brands found around the office. Unlike our analysis of our Christmas.com members (which uses social stream data from individual Facebook accounts), our list of celebrities (which used public tweets), some famous politicians (which used speeches, Tweets, and debate transcripts), for Brands we used public mentions of the brands within the Twitter firehose.

If a comment made about a brand had positive sentiment, then we gave a check in that column. If a comment made about a brand had negative sentiment, then we indicated it as such. 90% of the ‘neutral’ comments were disregarded. Thus, if people were saying generally positive things towards a brand (e.g., “I love my Mercedes Benz”) they received positive sentiment. And as we describe below, we believe that positivity relates to niceness.

Can you analyze my company brand?

For Santa’s naughty or nice list, we have used the Sentiment Analysis API provided by Sentiment140. Type in a brand into their search engine and you can get similar results. Of course, to find out if YOU are naughty or nice, you will have to connect via Facebook.

Obama vs. Romney  - Who is naughtier?

Given the intense public interest in the US Presidential Election (even outside the United States), we thought we would perform a more in-depth analysis of the two primary contenders: President Obama and former-Governor Mitt Romney.
This analysis included the following:

• Analysis of the speeches given by the candidates at their party’s national convention
• Analysis of the transcripts from each of the three Presidential debates
• Analysis of the candidates Twitter feeds (even though the words typed were often by staffers)

For the debates and the speeches, we analyzed the transcripts as provided by NPR and treated each paragraph as an independent text block. Each debate, speech, and Twitter stream contributed 20% to the Candidate’s total score.

Obama vs. Romney – The Results

The following table shows the raw polarity from our analysis:

So you are saying that Romney is Naughtier than Obama?

No… We are saying that 18 of 20 Santa’s Elves believe that former-Governor Romney is naughtier than President Obama based on the text that we analyzed.
However, we are also saying that both candidates are naughtier than 29 of the other 32 celebrities that were analyzed (though the data sources were different)

Can you provide an example of nice vs. naughty sentiment?

Sure.
The following text excerpts were determined to contain Positive sentiment:

"I believe that the free enterprise system is the greatest engine of prosperity the world's ever known. I believe in self-reliance and individual initiative and risk-takers being rewarded. But I also believe that everybody should have a fair shot and everybody should do their fair share and everybody should play by the same rules, because that's how our economy is grown. That's how we built the world's greatest middle class." President Obama, 2nd Presidential Debate

"And Republicans and Democrats both love America. But we need to have leadership -- leadership in Washington that will actually bring people together and get the job done and could not care less if -- if it's a Republican or a Democrat. I've done it before. I'll do it again.” Mitt Romney, 1st Presidential Debate

The following text excerpts were determined to contain Negative sentiment.

"But for too many Americans, these good days are harder to come by. How many days have you woken up feeling that something really special was happening in America?" Mitt Romney, Speech to Republican National Convention

"And Governor Romney wants to take us back to those policies: a foreign policy that's wrong and reckless; economic policies that won't create jobs, won't reduce our deficit, but will make sure that folks at the very top don't have to play by the same rules that you do.” President Obama, 3rd Presidential Debate

A few more examples would include:

POSITIVE - @EvaLongoria (who scored the highest) “Hey east coast! How ya holding up? Watching the news and it looks scary! Be safe and god bless!”

NEGATIVE - @Nasty_Santa (a persona we created who scored the lowest) “The only thing worse than 2 cookies on christmas eve, is no cookies on christmas eve.”

Any other interesting observations you can share?

Hmmm. One thing that we found interesting is that President Obama’s sentiment has been steadily climbing since his low point at the Democratic Convention. However, we also analyzed the "Yes We Can" speech that then candidate Obama gave in 2008 which has been recognized by many as being "positive". That speech scored a polarity of 3.60 or 90% on our Santa-lyzer.

We also found that those celebrities who scored highly rarely say anything negative. This holds true the maxim of, "If you don’t have anything nice to say, don’t say anything at all."

What is ‘Sentiment Analysis’?

According to Wikipedia, "Sentiment analysis or opinion mining refers to the application of natural language processing, computational linguistics, and text analytics to identify and extract subjective information in source materials.

Generally speaking, sentiment analysis aims to determine the attitude of a speaker or a writer with respect to some topic or the overall contextual polarity of a document. The attitude may be his or her judgment or evaluation affective state (that is to say, the emotional state of the author when writing), or the intended emotional communication (that is to say, the emotional effect the author wishes to have on the reader)."

The sentiment engine that we use is provided by Sentiment140 via their public API.

Does positivity equate to ‘niceness’?

Scientists have learned that a person’s perceptions are primarily formed by first impressions and by feelings, how we feel about one another. First impressions are driven by both verbal and non-verbal communication, and thus, if a person communicates positive statements, that same person is deemed to produce a ‘nicer’ first impression.

Is this Christmas hooey or is there science behind this?

Science rules.  In a 2012 paper submitted by L. Dang-Huan and S. Steiglitz (see: Impact and Diffusion of Sentiment in Political Communication – An Empirical Analysis of Political Weblogs), they concluded…

"Results of the negative binomial regression (see Table 2) support our hypothesis as the coefficient of the quadratic term entry_sentiment2 is positive and highly statistically significant at one-percent level (b = 0.21, p < 0.01). In other words, clearly articulated sentiment (either positive or negative) in blog entries positively affects their likelihood to receive feedback. This result also implies that blog entries which are rather sentiment-neutral (i.e., neither positive nor negative references) or of mixed sentiment nature (i.e., similar quantities of positive and negative references which would cancel out each other according to the sentiment polarity measure as defined in equation) tend to trigger less feedback."

So in other words while it is fun to find out if people are ‘naughty’ or ‘nice’, the ability to predict or increase audience engagement is kind of cool.

Am I really that naughty?

Once again according to Wikipedia, “the accuracy of a sentiment analysis system is, in principle, how well it agrees with human judgments. This is usually measured by precision and recall. However, human raters typically agree about 70% of the time. Thus, a 70% accurate program is doing as well as humans, even though such accuracy may not sound impressive. If a program were "right" 100% of the time, humans would still disagree with it about 30% of the time, since they disagree that much about any answer. More sophisticated measures can be applied, but evaluation of sentiment analysis systems remains a complex matter. For sentiment analysis tasks returning a scale rather than a binary judgment, correlation is a better measure than precision because it takes into account how close the predicted value is to the target value.”

How can I increase my score?

The Santa-lyzer reads your Facebook status updates, so the easiest way to improve your score is to "say positive things" in your communications and updates. If you do so, your score will slowly creep up to a higher level. You can also do positive things, like donate to a charity or sponsor a cause (See these good causes we've added to Christmas.com). Come back any time you want and try again after a few status updates and you may see some movement.

About Us and the Santa-lyzer

The Santa-lyzer is for entertainment purposes only. While it is based on science, mashing up sentiment analysis with your social graph, you should not fear being put on Santa’s real Naughty or Nice list if you do come up with a ‘naughty’ ranking. After all, he knows if you’ve been bad or good, so be good for goodness sake.

The Santa-lyzer was initially built by a group of hackers, marketers, and designers that met at the Facebook World Hack Day <Vancouver> who thought it would be kind of fun to create a Naughty or Nice list using a social stream and science. After all, why can’t Santa have access to the latest tools and science to let him do his thing?

The tool itself has been supported by Christmas.com and Left of the Dot Media. You can read more about us here.
Initial contributors include:

Alan Bailward, Left of the Dot
Deesa Smirnov, Leetr
Rachel Cheng, Invoke Media
Joe Deobald, Front Row Marketing
John Lyotier, Left of the Dot

Contact

Questions about the app? Are you with the media?

Send an email to support@christmas.com. You can also contact Christmas.com by clicking here.