Feeds

Is Gartner's Magic Quadrant really magic?

The disappearing act

3 Big data security analytics techniques

Database myths and legends (Part 8) In this series we're looking at the myths and legends of the database world; some are true, some false. The myth under the spotlight today is whether Gartner's Magic Quadrant really is magic.

(True, Gartner's Magic Quadrant (MQ) isn't just applied to databases, but its recent application to the area of BI platforms, for me at least, threw into sharp relief the essence of the MQ.)

Magic?

Many formal definitions of magic refer to "supernatural influences", others, for example Merriam-Webster's, includes references to "the art of producing illusions by sleight of hand". It is unlikely that Gartner means either of these; I assume the word is used to mean "great", "wonderful" and/or "fantastic". So, is the MQ really magic?

The Magic Quadrant

I don't need to tell you that the MQ plots the intersection point of two values; on the X axis "Completeness of Vision" and on the Y "Ability to Execute". Nor do I need to tell you that the graph is then divided into four quadrants - Leaders, Challengers, Visionaries and Niche Players. If you work in IT, you are doubtless familiar with the MQ; you know it of old. As it turns out, very old. So old that even Jenni Lehman, group vice president of research operations at Gartner didn't know exactly. She guessed at 15 years – about 1992.

This presumably explains why it is monochrome, two dimensional and static. It was developed in the days even before Death-by-PowerPoint. We are talking about a time when most people didn't have computers so the MQ appeared on paper and foils (ask a passing Grandparent for details).

This makes it a child of its time - but computing has since grown up. Not only have PCs evolved into superb multi-media platforms, we have also learnt a huge amount about how to present complex information to people. It is still a challenge, but people like Edward Tufte have done ground-breaking work and this has been brought into the computer world by pioneers like Hans Rosling. If you really want to see why I think this is important, check out Hans' talk at TED. This work has revolutionised the way we think about and present data and companies like Microsoft, Spotfire and many others are adopting these ideas into commercial products.

Yet Gartner is still giving us a monochrome, two dimensional, static quadrant. You do have to ask why, so I did.

I talked to Jenni Lehman who said "the value of the Gartner MQ is that it is well-understood throughout the IT world. The MQ provides an easy to understand market overview representing complex data gained through rigorous and independent research. The tool provides an excellent starting point for interactions between clients and analysts as they discuss the market from the context of the client environment".

From Gartner's perspective, I have to agree. The MQ clearly does have value for its clients. I think the MQ provides huge brand recognition for the company. Given all of that, you can see why change is low on the agenda. But, as a modern consumer of MQs, I have a different perspective. Every time one appears in a presentation I find myself groaning (in earlier times inaudibly; as time passes, more and more loudly).

There are two fundamental reasons.

Firstly, I know that we can do much better than this. For example, given the graphical representation in Figure 1, Company E looks good with D and F close behind.

Shows a basic form of Gartner's Magic Quadrant.

Suppose I tell you that size represents, say, sales strategy and colour, Business model (blue good, red bad), so we get Figure 2:

Shows an enhanced form of Gartner's Magic Quadrant.

Does your opinion change at all? Now suppose we add, in Figure 3, lines that show changes in the recent past:

Shows a more enhanced form of Gartner's Magic Quadrant.

We could even animate the blobs so they move across the screen, the size and colour changing as they moved.

I'm certainly not suggesting that this is an optimal solution (I chose the parameters at random). What I am suggesting is that it is possible to convey a more complete picture in a single graphic without overloading the human brain.

The second major problem I have with the MQ is that, while it does simplify complex data into a digestible form, along the way it erodes the detail in a way I find alarming. To see why, you have to look at how the MQ is produced.

Gartner first defines and describes the market and selects the relevant companies. Then it uses 15 criteria (ranging from product/service to geographical strategy) to evaluate each company.

Incidentally, there is very good evidence that Gartner performs this work very diligently and with great competence. I have absolutely no doubt of the worth of Gartner's work: my concern is with the MQ, not the fundamental research work that the company performs.

In preparing this article, I talked to several companies who have been assessed by Gartner and all reported the same: Gartner does an excellent job of collecting the raw data. For example, when I asked Seán Jackson, marketing manager for Kognitio, he said: "Gartner certainly did due diligence with regards to interviewing our clients. It was very thorough."

So, Gartner collects a huge amount of valuable information; in analytical terms, it collects data over 15 different dimensions. But to produce the MQ it then proceeds to amalgamate seven of the evaluation criteria into one dimension (ability to execute) and the remaining eight into another (completeness of vision). fifteen dimensions of data are collapsed into two.

This is where the problem lies and is best illustrated by an example. Two of the initial criteria Gartner uses are Sales Strategy and Business Model. Suppose that a company develops a world class sales strategy. As it turns out, by error, this strategy fatally compromises its business model. This was clearly a poor choice (but let's face it, one that has been made by several computer companies) and you would probably wish to avoid investing in the company's products.

Let's imagine that Gartner spots the flaw and scores the business model appropriately, but also scores the sales strategy highly, since it is world class. The problem is that when these scores are aggregated with six others into completeness of vision the detail can easily be lost.

I'm not suggesting that such a crude example is likely in practice. But even the process of aggregating data with identical meaning (such as sales figures for a given product) is known to obscure detail. Aggregating non-identical data has a much greater potential to obscure vital information.

Summary

Time has not been kind to the MQ and the magic is long gone.

Does this mean that I think that Gartner is somehow discredited, its analysis flawed? No, that's part of the problem. I have great respect for the value that Gartner's analysis brings to its clients. It is the MQ that has had its day and now is the time for the final magic – the disappearing act.

Top three mobile application threats

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
Inside the Hekaton: SQL Server 2014's database engine deconstructed
Nadella's database sqares the circle of cheap memory vs speed
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
Oh no, Joe: WinPhone users already griping over 8.1 mega-update
Hang on. Which bit of Developer Preview don't you understand?
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
Pre-Update versions of new Windows version will no longer support patches
IRS boss on XP migration: 'Classic fix the airplane while you're flying it attempt'
Plus: Condoleezza Rice at Dropbox 'maybe she can find ... weapons of mass destruction'
prev story

Whitepapers

Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.