Feeds

The parable of the beer and diapers

Never let the facts get in the way of a good story

Beginner's guide to SSL certificates

Database myths and legends (Part 3) BI (Business Intelligence) is about extracting information from data and data mining is an important part of that process. Data mining is a process that looks for patterns in data, so in a sense it is like querying the data. The crucial differences between simply querying the data and data mining can be summed up as intent and scale.

When humans query data we start with an idea, such as: "I think that we sell more DVDs to males than to females." And then we run a query to test the idea and the answer either confirms or disproves our hypothesis. A data mining algorithm doesn't have ideas. It has no intention of testing ideas for the simple reason that it doesn't have any.

What it does have is huge processing power at its disposal, so it simply tests a very large number of possible correlations. This can be done by firing a very large number of queries at the database, but that approach can be very slow.

The algorithms used nowadays often use more sophisticated approaches; they can, for example, create a multi-dimensional data structure and then examine it looking for patterns, and/or outliers. When they find something of interest, they flag it for attention.

It is easier to illustrate the difference between querying and data mining with a good example and, already firmly enshrined in BI mythology, is the "beer and diapers" story.

It goes (with minor variations) like this:

Some time ago, Wal-Mart decided to combine the data from its loyalty card system with that from its point of sale systems. The former provided Wal-Mart with demographic data about its customers, the latter told it where, when and what those customers bought. Once combined, the data was mined extensively and many correlations appeared. Some of these were obvious; people who buy gin are also likely to buy tonic. They often also buy lemons. However, one correlation stood out like a sore thumb because it was so unexpected.

On Friday afternoons, young American males who buy diapers (nappies) also have a predisposition to buy beer. No one had predicted that result, so no one would ever have even asked the question in the first place. Hence, this is an excellent example of the difference between data mining and querying.

The story goes on that, once the correlation was uncovered, it was easy to back extrapolate from the effect to the cause.

  • Young American males frequently indulge in ritualised carousing behaviour with friends of Friday nights.
  • Carousing usually involves the consumption of beer.
  • Most young American males only buy diapers after they have fathered offspring.
  • Offspring acquisition is a known carousing inhibitor.

So the proud new father is walking around the store on Friday afternoon. He knows there is no way that he is going to get out of the house to join his mates at the bar. However, there is nothing to stop him from drinking beer at home. All he needs is to be reminded of that fact. After seeing the results of the data mining, Wal-Mart moved the beer next to the diapers and beer sales went up.

Like all good myths, the beer and diapers story does have its origins in fact; but sadly, almost all of the detail (and specifically the detail that makes it a great BI story) is probably fabrication. We are all indebted to Daniel Power for uncovering the origins of the story. He provides all of the detail here.

In short, he traced the story back much further than I would have believed, way back to 1992. At that time, Thomas Blischok was the manager of a group at Teradata. His group looked at point-of-sale data from Osco Drug stores - 1.2 million baskets worth in all. It isn't clear what tools the team was using. They are described as "state-of-the-art, query generation tools", which were undoubtedly leading edge in 1992.

Those queries revealed that, between 5pm and 7pm, customers tended to co-purchase beer and diapers. No correlation with age or gender was established; although it isn't clear whether these questions were ever asked.

In addition, the store chain does not appear to have exploited the information by moving the products around. So we have a correlation between beer, diapers and time, but no correlations with age, gender or day. Worst of all, it is very much open to question whether the techniques used would qualify as data mining in its modern sense.

So, where does all that leave our beer and diapers story as an example of data mining? Well, I for one am prepared to accept that it didn't really happen in the way my grandfather described it to me as I sat upon his knee. Sigh.

However, that doesn't mean that we have to consign the story to Room 101 (for those readers outside the UK, also a popular TV comedy). The story is so popular because it is a good illustration of the difference between querying and data mining. The facts don't change that at all.

The image of a forlorn young man trudging round a supermarket on Friday night; his day suddenly brightened by the sight of a stack of beer sitting incongruously next to a pile of diapers is somehow wonderfully compelling. And none of us would have asked that question in the first place.

There is no reason we cannot continue to use it as an illustrative story; as long as we are aware that it is simply an allegory or fable. In fact, given that it is a fable about a set of data, we could call it a table fable. ®

Beginner's guide to SSL certificates

More from The Register

next story
That dreaded syncing feeling: Will Microsoft EVER fix OneDrive?
Microsoft's long history of broken Windows sync
Mozilla, EFF, Cisco back free-as-in-FREE-BEER SSL cert authority
Let’s Encrypt to give HTTPS-everywhere a boost in 2015
SLURP! Flick your TONGUE around our LOLLIPOP – Google
Android 5 is coming – IF you're lucky enough to have the right gadget
Nokia's N1 fondleslab's HIDDEN BRILLIANCE: The 'Z Launcher'
Sugarcoating Android's Lollipop makes tab easier to swallow
Bug fixes! Get your APPLE BUG FIXES! iOS and OS X updates right here!
Yosemite fixes Wi-Fi hiccup, older iOS devices get performance boost
Facebook, working on Facebook at Work, works on Facebook. At Work
You don't want your cat or drunk pics at the office
Soz, web devs: Google snatches its Wallet off the table
Killing off web service in 3 months... but app-happy bonkers are fine
Meet Windows 10's new UI for OneDrive – also known as File Explorer
New preview build continues Redmond's retreat to the desktop
Microsoft: Your Linux Docker containers are now OURS to command
New tool lets admins wrangle Linux apps from Windows
prev story

Whitepapers

Choosing cloud Backup services
Demystify how you can address your data protection needs in your small- to medium-sized business and select the best online backup service to meet your needs.
Getting started with customer-focused identity management
Learn why identity is a fundamental requirement to digital growth, and how without it there is no way to identify and engage customers in a meaningful way.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Simplify SSL certificate management across the enterprise
Simple steps to take control of SSL across the enterprise, and recommendations for a management platform for full visibility and single-point of control for these Certificates.
Saudi Petroleum chooses Tegile storage solution
A storage solution that addresses company growth and performance for business-critical applications of caseware archive and search along with other key operational systems.