Feeds

Sarah Palin's words get data mined

Whoah, there's data here? Business analysis gets to work on VP transcripts

Top 5 reasons to deploy VMware with Tegile

The exercise was performed using Microsoft’s BI tools by a Microsoft employee at Redmond and the results were kindly made available to The Register. The transcripts are freely available and many of you have access to analysis tools, so why not have a go?

The transcripts were passed through a process that not only counts the number of times that a word is used but also assigns a ‘tf-idf’ weighting (term frequency–inverse document frequency) which gives some indication of the importance of the words in the document.

The (totally and absolutely not scientific) results are fascinating. Top of the list of the early Palin transcript is the term “good guy” with “bad guy” in 26th position. In the later debate she doesn’t use these terms at all. In the interests of fairness it is worth pointing out that Joe Biden (the opposition Vice Presidential candidate) uses the term “bad guy” once in the debate, so perhaps he needs some coaching too.

If we try to look for warm fuzzy patriotic words and compare their position in the early interview and then in the later debate, we find:

Word Interview Debate
Good guy 1 Never appears
Alaska 13 40
Freedom 15 80
Democratic value 16 Never appears
Face 22 170
Bad Guy 26 Never appears

Now suppose we look for words that might be considered to be more presidential – words which give the impression of a potential world leader:

Word Interview Debate
Afghanistan 125 2
People 40 4
Economy Never appears 8
Iraq 29 9
Job 144 12
Tax Never appears 13
War 68 17
Government 166 21
Nation Never appears 25

So have we proved our premise? Given that we have already hammered home the lack of science here, we leave it to you, gentle reader, to decide if there is enough ‘evidence’ here.

Of course, if we were being fair, we would (as the original analysis did) take a look at the comparable Joe Biden transcripts. But we aren’t trying to be fair or unfair. We aren’t trying to score any particular political points; we are trying to show that BI techniques can be applied to any data, not just business data. Which brings us to you.

  • Do you feel that your pointy-haired boss magically changed his or her position on some policy?
  • Do you have access to the minutes of the meetings?
  • And access to BI tools?

If you can answer “yes” to these three questions, then what are you waiting for? Hurry to the BI bonanza and get mining.

However, although BI is broadly applicable, there are data sets to which it would be entirely inappropriate to apply these techniques - for example, the work of fine, upstanding journalists such as those employed at Vulture Central. For reasons that are too technical to go into here, this data is not amenable to analysis of this kind. Which is a shame, because we know that we are always consistent. ®

Secure remote control for conventional and virtual desktops

More from The Register

next story
Facebook's Zuckerberg in EBOLA VIRUS FIGHT: Billionaire battles bug
US Centers for Disease Control and Prevention contacted as site supremo coughs up
Space exploration is just so lame. NEW APPS are mankind's future
We feel obliged to point out the headline statement is total, utter cobblers
Win a year’s supply of chocolate (no tech knowledge required)
Over £200 worth of the good stuff up for grabs
Down-under record: Australian gets $140k for pussy
'Tiffany' closes deal - 'it's more common to offer your wife', says agent
Internet finally ready to replace answering machine cassette tape
It's a simple message and I'm leaving out the whistles and bells
Swiss wildlife park serves up furry residents to visitors
'It's ecological' says spokesman, now how would you like your Bambi done?
The iPAD launch BEFORE it happened: SPECULATIVE GUFF ahead of actual event
Nerve-shattering run-up to the pre-planned known event
STONER SHEEP get the MUNCHIES after feasting on £4k worth of cannabis plants
Baaaaaa! Fanny's Farm's woolly flock is high, maaaaaan
FedEx helps deliver THOUSANDS of spam messages DIRECT to its Blighty customers
Don't worry Wilson, I'll do all the paddling. You just hang on
Red Bull does NOT give you wings, $13.5m lawsuit says so
Website letting consumers claim $10 cash back crashes after stampede
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Win a year’s supply of chocolate
There is no techie angle to this competition so we're not going to pretend there is, but everyone loves chocolate so who cares.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.